一、Snappy

1. snzip 基于 Snappy 的压缩/解压工具

  • 使用的版本 1.0.4
  • 支持框架的格式化
    • framing-format
    • old framing-format
    • hadoop-snappy format (Hadoop Snappy 文件格式的压缩)
    • raw format
    • snappy-java
    • snappy-in-java
  • snzip 项目地址

下载、安装使用方法, 具体见 github 文档

  2. 1. 安装
  3.   tar xvfz snzip-1.0.4.tar.gz
  4.   cd snzip-1.0.4
  5.   ./configure --prefix=/usr/local/snappy
  6.   make
  7.   make install
  10. 2. 加载到系统环境
  11.   vim ~/.bashrc
  12.   # snzip
  13.   export SNZIP_HOME=/usr/local/snappy
  14.   export PATH=${SNZIP_HOME}/bin:$PATH
  16.   source ~/.bashrc
  19. 3. snzip -help
  21.   general options:
  22.    -c       输出到标准输出,保持原始文件不变
  23.    -d       解压缩
  24.    -k       不删除原文件
  25.    -t name  压缩框架文件格式
  26.    -h       give this help
  28.   raw_format option:
  29.    -s size  size of input data when compressing.
  30.             The default value is the file size i f available
  32.   tuning options(调优参数):
  33.    -b num   internal block size in bytes
  34.    -B num   internal block size. \'num\'-th power of two.
  35.    -R num   size of read buffer in bytes
  36.    -W num   size of write buffer in bytes
  37.    -T       trace for debug
  39.   supported formats(压缩框架格式选择):
  40.     NAME            SUFFIX  URL
  41.     ----            ------  ---
  42.     framing2        sz      https://github.com/google/snappy/blob/master/framing_format.txt
  43.     hadoop-snappy   snappy  https://code.google.com/p/hadoop-snappy/
  44.     iwa             iwa     https://github.com/obriensp/iWorkFileFormat/blob/master/Docs/index.md#snappy-compression
  45.     framing         sz      https://github.com/google/snappy/blob/0755c815197dacc77d8971ae917c86d7aa96bf8e/framing_format.txt
  46.     snzip           snz     https://github.com/kubo/snzip
  47.     snappy-java     snappy  https://github.com/xerial/snappy-java
  48.     snappy-in-java  snappy  https://github.com/dain/snappy
  49.     comment-43      snappy  http://code.google.com/p/snappy/issues/detail?id=34#c43
  52. 4. 压缩 hadoop 框架支持的格式化
  54.   snzip -t -k hadoop-snappy -k file_name  压缩
  56.   snzip -d compressed_file.snappy 解压

2. python 压缩/解压接口(不兼容 HDFS 原生的 Snappy)

  2. 1. 安装
  4. 依赖包
  5.   ubuntu:
  6.     sudo apt-get install libsnappy-dev
  8.   Centos:
  9.     sudo yum install libsnappy-devel
  11.   Brew:
  12.     brew install snappy
  14. 安装
  15.   pip install python-snappy
  17.   python -m snappy --help
  19. 2. 压缩/解压文件
  20.   python -m snappy -c uncompressed_file compressed_file.snappy
  21.   python -m snappy -d compressed_file.snappy uncompressed_file
  24. 3. 压缩/解压 Stream
  25.   cat uncompressed_data | python -m snappy -c > compressed_data.snappy
  26.   cat compressed_data.snappy | python -m snappy -d > uncompressed_data

3. java 压缩/解压接口

  1. 1. 注意事项, 如果在 Mac 环境中使用请把 jar 包解压
  3.   复制 libsnappyjava.jnilib -> libsnappyjava.dylib
  4.   cp org/xerial/snappy/native/Mac/x86_64/libsnappyjava.jnilib org/xerial/snappy/native/Mac/x86_64/libsnappyjava.dylib
  6.   重启打包
  7.   jar cf snappy-java-1.0.4.1.jar org
  10. 2. pom.xml 配置加载本地包
  11. <dependency>
  12.       <groupId>org.xerial.snappy</groupId>
  13.       <artifactId>snappy-java</artifactId>
  14.       <version>1.0.4.1</version>
  15.       <scope>system</scope>
  16.       <systemPath>${basedir}/lib/snappy-java-1.0.4.1.jar</systemPath>
  17.   </dependency>
  18. </dependencies>

二、SPARK 配置

  1. # Spark 配置 Snappy
  2. export JAVA_LIBRARY_PATH=$JAVA_LIBRARY_PATH:$HADOOP_HOME/lib/native
  3. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native
  4. export SPARK_LIBRARY_PATH=$SPARK_LIBRARY_PATH:$HADOOP_HOME/lib/native
  6. export SPARK_CLASSPATH=$SPARK_CLASSPATH:$HADOOP_HOME/lib/snappy-java-1.0.4.1.jar
  8. spark-sql --jars file:///etc/hive/auxlib/json-serde-1.3.7-jar-with-dependencies.jar,file:///usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar