joy涉及的源代码

  1. p2f.c
  2. config.c
  3. osdetect.c
  4. anon.c
  5. pkt_proc.c
  6. nfv9.c
  7. classify.c
  8. radix_trie.c
  9. hdr_dsc.c
  10. procwatch.c 
  11. addr_attr.c 
  12. addr.c 
  13. wht.c 
  14. str_match.c 
  15. acsm.c 
  16. example.c 
  17. updater.c 
  18. ipfix.c 
  19. salt.c 
  20. parson.c 
  21. fingerprint.c 
  22. ppi.c 
  23. utils.c 
  24. payload.c 
  25. proto_identify.c 
  26. fp_tls.c 
  27. extractor.c
  28. dhcp.c 
  29. dns.c 
  30. http.c 
  31. tls.c
  32. ssh.c 
  33. ike.c 
  34. joy.c

主流程

  1. main
  2.     finish_initial_setup
  3.         proto_identify_init    // proto_identify.c
  4.     // mode = offline
  5.     process_single_input_file
  6.         process_pcap_file
  7.             joy_libpcap_process_packet    // joy_api.c
  8.                 process_packet    // pkt_proc.c
  9.                     process_tcp
  10.                     process_udp
  11.                     process_icmp
  12.                     process_ip

proto_identify_init 会创建 tcp/udp 关键字字典树, 后缀用于检测上层协议.

process_packet 是报文处理的核心函数, 其中会对报文进行处理, 主要是解析协议层, 并调用上层协议的处理函数.

  1. process_tcp
  2.     flow_key_get_record    // p2f.c, 查表获取流记录
  3.     retrans_detected
  4.     flow_record_process_packet_length_and_time_ack
  5.     proto_identify_tcp // proto_identify.c, 识别tcp上层协议, 只检测前两个payload>0的包
  6.         search_keyword_dict
  7.     update_all_features // update header description

proto_identy_tcp 使用关键字识别来确定(某些)上层协议, 如果命中, 则返回对应的协议号和 client/server 方向.

  1. /*
  2.  * Estimate the TCP application protocol
  3.  * Optimization: stop after first 2 packets that have non-zero payload
  4.  */
  5. if ((!record->app) && record->op <= 2) {
  6.     const struct pi_container *pi = proto_identify_tcp(payload, size_payload);
  7.     if (pi != NULL) {
  8.         record->app = pi->app;
  9.         record->dir = pi->dir;
  10.     }
  11. }

update_all_features 这里用了一个宏定义技巧, 就是在多个 feature 上都调用 update_feature, 这里的 feature 就是协议插件, 比如 http, dns 等.

  1. #define tcp_feature_list salt, ppi, fpx
  2. #define payload_feature_list wht, example, dns, ssh, tls, dhcp, dhcpv6, http, ike, payload
  3. #define feature_list payload_feature_list, tcp_feature_list
  5. /** The macro update_feature(f) processes a single packet and updates
  6.  * the feature context
  7.  */
  8. #define update_feature(f) \
  9.     if (f##_filter(record) && (glb_config->report_##f)) { \
  10.         if (record->f == NULL) f##_init(&record->f); \
  11.         f##_update(record->f, header, payload, size_payload, glb_config->report_##f); \
  12.     }
  14. /** The macro update_all_features(list) invokes update_feature() for each
  15.  * feature in list
  16.  */
  17. #define update_all_features(feature_list) MAP(update_feature, feature_list)

updaet_feature 也是个宏, 展开后, 先判断是否是当前协议/有没有开启协议插件, 如果是的话再调用相应协议的 update 函数, 比如 http 展开后:

  1. if (http_filter(record) && (glb_config->report_http)) {
  2.     if (record->http == NULL)
  3.         http_init(&record->http);
  4.     http_update(record->http, header, payload, size_payload, glb_config->report_http);
  5. }

其中 http 过滤条件 (http_filter) 是:

  1. #define http_filter(record) \
  2.     ((record->key.prot == 6) && \
  3.      (record->app == 80 || (record->key.sp == 80 || record->key.dp == 80)) \
  4.     )

而 http_update 会解析/处理 HTTP 报文.

协议处理

TCP

DNS

识别规则:
udp, server->client, {WILDCARD, WILDCARD, 0x01, 0x00, 0x00, 0x01, 0x00, 0x00}

HTTP

识别规则:

  • GET
    tcp, client->server, {0x47, 0x45, 0x54, 0x20}
  • POST
    tcp, client->server, {0x50, 0x4f, 0x53, 0x54, 0x20}
  • OPTIONS
    tcp, client->server, {0x4f, 0x50, 0x54, 0x49, 0x4f, 0x4e, 0x53, 0x20}
  • HEAD
    tcp, client->server, {0x48, 0x45, 0x41, 0x44, 0x20}
  • PUT
    tcp, client->server, {0x50, 0x55, 0x54, 0x20}
  • DELETE
    tcp, client->server, {0x44, 0x45, 0x4c, 0x45, 0x54, 0x45, 0x20}
  • TRACE
    tcp, client->server, {0x54, 0x52, 0x41, 0x43, 0x45, 0x20}
  • CONNECT
    tcp, client->server, {0x43, 0x4f, 0x4e, 0x4e, 0x45, 0x43, 0x54, 0x20}
  • HTTP 1.1
    tcp, server->client, {0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20}

TLS

识别规则:

  • Client Hello 1.0
    tcp, client->server, {0x16, 0x03, 0x01, WILDCARD, WILDCARD, 0x01}
  • Server hello 1.0
    tcp, server->client, {0x16, 0x03, 0x01, WILDCARD, WILDCARD, 0x02}
  • Client Hello 1.1
    tcp, client->server, {0x16, 0x03, 0x02, WILDCARD, WILDCARD, 0x01}
  • Server hello 1.1
    tcp, server->client, {0x16, 0x03, 0x02, WILDCARD, WILDCARD, 0x02}
  • Client Hello 1.2
    tcp, client->server, {0x16, 0x03, 0x03, WILDCARD, WILDCARD, 0x01}
  • Server Hello 1.2
    tcp, server->client, {0x16, 0x03, 0x03, WILDCARD, WILDCARD, 0x02}

数据结构/算法

流记录

关键字字典树

参考