SparkSQL 解析 Snappy

  • 版本 Spark 2.1

一、配置

  • $HADOOP_HOME/lib/native 库的支持(配置好环境变量, 退出终端重启服务)
  2. # 环境配置
  3. export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
  4. export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
  6. export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native
  7. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native:/usr/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib
  9. export SPARK_HOME=/usr/local/spark
  10. export SPARK_CONF_DIR=$SPARK_HOME/conf
  11. export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:$HADOOP_HOME/lib/native
  12. export SPARK_CLASSPATH=$SPARK_CLASSPATH
  13. export PATH=$SPARK_HOME/bin:$PATH
  16. # 启动配置
  17. spark-sql --jars file://$HADOOP_HOME/lib/snappy-java-1.0.4.1.jar,file:///etc/hive/auxlib/json-serde-1.3.7-jar-with-dependencies.jar \
  18. --name spark-sql-server \
  19. --master yarn \
  20. --deploy-mode client \
  21. --driver-cores 2 \
  22. --driver-memory 4g \
  23. --executor-cores 2 \
  24. --executor-memory 4g \
  25. --num-executors 2 \
  26. --conf spark.eventLog.enabled=false \
  27. --conf spark.eventLog.dir=hdfs://dw1:8020/tmp/spark \
  28. --conf spark.serializer=org.apache.spark.serializer.KryoSerializer \
  29. --conf spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec \
  30. --conf net.topology.script.file.name=/etc/hadoop/conf.cloudera.yarn/topology.py \
  31. --conf spark.sql.parquet.compression.codec=snappy
  34. # 参数解读
  35. -- 压缩编码 spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec;
  37. -- 解码器 spark.sql.parquet.compression.codec=snappy;
  40. # 测试
  41. SELECT common FROM ods.ods_browser_click WHERE p_dt='2017-05-07' AND p_hours='00' LIMIT 1;

二、问题

在 spark.master=yarn 模式下支持不好, 主要问题表现在子节点无法获取到 $HADOOP_HOME/lib/native 类库

spark-sql \ —name spark-sql-server \ —master yarn \ —deploy-mode client \ —driver-cores 2 \ —driver-memory 4g \ —executor-cores 2 \ —executor-memory 4g \ —num-executors 2 \ —conf spark.eventLog.enabled=false \ —conf spark.eventLog.dir=hdfs://dw1:8020/tmp/spark \ —conf spark.serializer=org.apache.spark.serializer.KryoSerializer \ —conf spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec \ —conf net.topology.script.file.name=/etc/hadoop/conf.cloudera.yarn/topology.py \ —conf spark.sql.parquet.compression.codec=snappy