cdh 中 spark 配置 lzo

时间:2021-08-11 19:07:17

1,在cdh的spark页面 中添加如下配置:

spark-conf/spark-env.sh 的 Spark 服务高级配置代码段(安全阀):

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native
export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*


2.yarn 添加配置;

yarn.app.mapreduce.am.admin.user.env 追加 :/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/native

yarn.application.classpath追加/opt/cloudera/parcels/HADOOP_LZO/lib/hadoop/lib/*