1. 软件需求

(1) C/C++编译器

hyperscan使用C开发，且需要C99和C11支持，目前支持的编译器有

  1. GCC, v4.8.1 or higher
  2. Clang, v3.4 or higher (with libstdc++ or libc++)
  3. Intel C++ Compiler v15 or higher

(2) 第三方依赖库

  1. 依赖项       版本         说明
  2. -------------------------------------------------
  3. CMake       >=2.8.11     
  4. Ragel       6.9  
  5. Python      2.7  
  6. Boost       >=1.57      仅需要regex
  7. Pcap        >=0.8       Optional: 仅用于示例程序

注1：boost不需要编译安装，要下载源码包，解压后执行类似

  1. # ln -s /root/zhn/software/boost_1_58_0/boost ../hyperscan-5.0.0/include/boost

的命令创建符号链接，注意boost路径需要是绝对路径，否则可能找不到

注2：pcap库会依赖flex和bison

2. 编译过程

  1. # cd hyperscan-5.0.0
  2. # mkdir build
  3. # cd build
  4. # cmake .. -DBUILD_STATIC_AND_SHARED=on -DCMAKE_BUILD_TYPE=Release
  5. # make -j 20
  6. # ll lib
  7. total 22452
  8. -rw-r--r--. 1 root root   174930 May  8 16:22 libcorpusomatic.a
  9. -rw-r--r--. 1 root root    13598 May  8 15:50 libcrosscompileutil.a
  10. -rw-r--r--. 1 root root    12624 May  8 15:50 libdatabaseutil.a
  11. -rw-r--r--. 1 root root    38770 May  8 15:56 libexpressionutil.a
  12. -rw-r--r--. 1 root root 17232358 May  8 16:25 libhs.a
  13. -rw-r--r--. 1 root root  5502018 May  8 16:17 libhs_runtime.a
  14. # make install
  15. Install the project...
  16. -- Install configuration: "RELWITHDEBINFO"
  17. -- Installing: /usr/local/lib64/pkgconfig/libhs.pc
  18. -- Installing: /usr/local/include/hs/hs.h
  19. -- Installing: /usr/local/include/hs/hs_common.h
  20. -- Installing: /usr/local/include/hs/hs_compile.h
  21. -- Installing: /usr/local/include/hs/hs_runtime.h
  22. -- Installing: /usr/local/lib64/libhs_runtime.a
  23. -- Installing: /usr/local/lib64/libhs.a
  24. -- Installing: /usr/local/share/doc/hyperscan/examples/simplegrep.c
  25. -- Installing: /usr/local/share/doc/hyperscan/examples/pcapscan.cc
  26. -- Installing: /usr/local/share/doc/hyperscan/examples/patbench.cc
  27. -- Installing: /usr/local/share/doc/hyperscan/examples/README.md

3. 出现问题

(1) 汇编程序太老，无法汇编gcc发出的指令

  1. /tmp/ccmMtaRQ.s:12011: Error: no such instruction: `shrx %r8d,%eax,%eax'
  2. /tmp/ccmMtaRQ.s:12064: Error: no such instruction: `shlx %rax,%rsi,%rax'
  3. /tmp/ccmMtaRQ.s:12134: Error: no such instruction: `shlx %rsi,%rax,%rax'
  4. ...
  6. 当前指令集：
  7. # gcc -march=native -Q --help=target | grep march

需要安装binutils

  1. # wget https://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.gz
  2. # tar -zxvf binutils-2.27.tar.gz 
  3. # cd binutils-2.27
  4. # ./configure --prefix=/usr
  5. # make 
  6. # make install

4. demo程序

(1) simplegrep

  1. # pwd
  2. /root/zhn/software/hyperscan-5.0.0/examples
  4. # ../build/bin/simplegrep int simplegrep.c 
  5. Scanning 8051 bytes with Hyperscan
  6. Match for pattern "int" at offset 1774
  7. Match for pattern "int" at offset 2244
  8. Match for pattern "int" at offset 2707
  9. Match for pattern "int" at offset 2730
  10. Match for pattern "int" at offset 2756
  11. Match for pattern "int" at offset 2845
  12. Match for pattern "int" at offset 2875
  13. Match for pattern "int" at offset 3152
  14. Match for pattern "int" at offset 3228
  15. Match for pattern "int" at offset 3524
  16. Match for pattern "int" at offset 3732
  17. Match for pattern "int" at offset 3889
  18. Match for pattern "int" at offset 4101
  19. Match for pattern "int" at offset 4249
  20. Match for pattern "int" at offset 4347
  21. Match for pattern "int" at offset 4528
  22. Match for pattern "int" at offset 4873
  23. Match for pattern "int" at offset 5048
  24. Match for pattern "int" at offset 5086
  25. Match for pattern "int" at offset 5095
  26. Match for pattern "int" at offset 5153
  27. Match for pattern "int" at offset 5869
  28. Match for pattern "int" at offset 6090
  29. Match for pattern "int" at offset 6121
  30. Match for pattern "int" at offset 7099
  31. Match for pattern "int" at offset 7146
  32. Match for pattern "int" at offset 7322
  33. Match for pattern "int" at offset 7484
  34. Match for pattern "int" at offset 7661

(2) pcapscan

准备一个规则文件matchFile和pcap文件baidu_image.pcap

matchFile内容

  1. 123:/baidu/
  2. 456:/[f|F]ile/

执行

  1. # ./pcapscan matchFile baidu_image.pcap 
  2. Pattern file: matchFile
  3. Compiling Hyperscan databases with 2 patterns.
  4. Hyperscan streaming mode database compiled in 0.00116827 seconds.
  5. Hyperscan block mode database compiled in 0.000731925 seconds.
  6. PCAP input file: baidu_image.pcap
  7. 449 packets in 130 streams, totalling 364977 bytes.
  8. Average packet length: 812 bytes.
  9. Average stream length: 2807 bytes.
  11. Streaming mode Hyperscan database size    : 4904 bytes.
  12. Block mode Hyperscan database size        : 4904 bytes.
  13. Streaming mode Hyperscan stream state size: 22 bytes (per stream).
  15. Streaming mode:
  17.   Total matches: 266
  18.   Match rate:    0.7463 matches/kilobyte
  19.   Throughput (with stream overhead): 12860.81 megabits/sec
  20.   Throughput (no stream overhead):   14072.15 megabits/sec
  22. Block mode:
  24.   Total matches: 264
  25.   Match rate:    0.7407 matches/kilobyte
  26.   Throughput:    17829.85 megabits/sec
  29. WARNING: Input PCAP file is less than 2MB in size.
  30. This test may have been too short to calculate accurate results.

(3) patbench

  1. # ./patbench matchFile baidu_image.pcap 
  2. Base signatures: matchFile    PCAP input file: baidu_image.pcap    Repeat count: 1    Mode: streaming
  3. Scan time 0.008 sec, Scanned 364977 bytes, Throughput 381.597 Mbps, Matches 266
  4. Number of signatures: 2
  5. Base performance: 12379.131 Megabits/s
  6. Cutting signatures cumulatively for 1 generations
  7. Generation 0 .               Performance: 23840.098 Megabits/s (1.926x) after cutting:
  8. 456:/[f|F]ile/

5. 参考文献