20221005-Cilium Host Routing Enabled with BPF Mode[VxLAN] - 《Cilium Docs》

ENV:
kubeadm init --kubernetes-version=v1.20.5 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --skip-phases=addon/kube-proxy --ignore-preflight-errors=Swap
[root@dev3 ~]# kubectl get nodes -o wide 
NAME   STATUS   ROLES                  AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
dev1   Ready    control-plane,master   30d   v1.20.5   192.168.2.31   <none>        CentOS Linux 7 (Core)   5.15.8-1.el7.elrepo.x86_64   docker://20.10.12
dev2   Ready    <none>                 30d   v1.20.5   192.168.2.32   <none>        CentOS Linux 7 (Core)   5.15.8-1.el7.elrepo.x86_64   docker://20.10.12
dev3   Ready    <none>                 30d   v1.20.5   192.168.2.33   <none>        CentOS Linux 7 (Core)   5.15.8-1.el7.elrepo.x86_64   docker://20.10.12
[root@dev3 ~]# 
1.add the repo:
helm repo add cilium https://helm.cilium.io/
2.generate the installation template:
helm template cilium cilium/cilium --version 1.10.6 \
    --namespace kube-system \
    --set kubeProxyReplacement=strict \
    --set k8sServiceHost=192.168.2.31 \
    --set k8sServicePort=6443 > 1.10.6_bpf.yaml
3.我们加以下参数enable log level to debug:
debug-verbose: "datapath"
debug: "true"
monitor-aggregation: "none"
4.根据log，这里设置一下参数：
[root@dev1 kubernetes]# kubectl -nkube-system logs -f cilium-7h7jl| grep -i falling
level=info msg="BPF host routing requires enable-bpf-masquerade. Falling back to legacy host routing (enable-host-legacy-routing=true)." subsys=daemon
enable-bpf-masquerade: "true"
5.检查运行的模式：
root@dev2:/home/cilium# cilium status
KVStore:                Ok   Disabled
Kubernetes:             Ok   1.20 (v1.20.5) [linux/amd64]
Kubernetes APIs:        ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1beta1::EndpointSlice", "networking.k8s.io/v1::NetworkPolicy"]
KubeProxyReplacement:   Strict   [ens33 192.168.2.32 (Direct Routing)]
Cilium:                 Ok   1.10.6 (v1.10.6-17d3d15)
NodeMonitor:            Listening for events on 128 CPUs with 64x4096 of shared memory
Cilium health daemon:   Ok   
IPAM:                   IPv4: 3/254 allocated from 10.244.1.0/24, 
BandwidthManager:       Disabled
Host Routing:           BPF
Masquerading:           BPF   [ens33]   10.244.1.0/24 [IPv4: Enabled, IPv6: Disabled]
Controller Status:      25/25 healthy
Proxy Status:           OK, ip 10.244.1.145, 0 redirects active on ports 10000-20000
Hubble:                 Ok   Current/Max Flows: 4095/4095 (100.00%), Flows/s: 24.78   Metrics: Disabled
Encryption:             Disabled
Cluster health:         3/3 reachable   (2022-01-18T13:53:29Z)
root@dev2:/home/cilium#

1.同节点Pod通信： ```properties [root@dev1 kubernetes]# kubectl get pods -o wide | grep dev1 cni-client-7ccd98bdb8-b98hw 1/1 Running 0 72s 10.244.0.168 dev1 cni-same-7d599bc5d8-qn285 1/1 Running 0 13s 10.244.0.246 dev1 [root@dev1 kubernetes]# [注意此时我们使用的host routing enabled的case：] 内核是大于5.10.所以我们便可以使用 bpf_redirect_peer() 和 bpf_redirect_neigh() helpers函数了。就能大大的增加通信的性能了。 [root@dev1 kubernetes]# tc filter show dev lxc3379243e18e1 ingress filter protocol all pref 1 bpf chain 0 filter protocol all pref 1 bpf chain 0 handle 0x1 bpf_lxc.o:[from-container] direct-action not_in_hw tag 6428620aa9985da3 此时我们知道此时同节点使用bpf_redirect_peer() 直接把数据包redirect到Pod-B的eth0网卡，而不是到Pod-B的lxc网卡。这单和我们上边的没有打开host routing是有很大的区别的。我们抓包看看： [root@dev1 kubernetes]# kubectl exec -it cni-client-7ccd98bdb8-b98hw — ping -c 2 10.244.0.246 PING 10.244.0.246 (10.244.0.246): 56 data bytes 64 bytes from 10.244.0.246: seq=0 ttl=63 time=0.163 ms 64 bytes from 10.244.0.246: seq=1 ttl=63 time=0.166 ms

—- 10.244.0.246 ping statistics —- 2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 0.163/0.164/0.166 ms [root@dev1 kubernetes]# [root@dev1 kubernetes]# tcpdump -pne -i lxc3379243e18e1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lxc3379243e18e1, link-type EN10MB (Ethernet), capture size 262144 bytes 22:05:21.227235 1a:10:ce:52:30:a8 > 1e:96:0f:38:9c:12, ethertype IPv4 (0x0800), length 98: 10.244.0.246 > 10.244.0.168: ICMP echo reply, id 8960, seq 0, length 64 22:05:22.228535 1a:10:ce:52:30:a8 > 1e:96:0f:38:9c:12, ethertype IPv4 (0x0800), length 98: 10.244.0.246 > 10.244.0.168: ICMP echo reply, id 8960, seq 1, length 64 此时我们注意上边的数据包，看到只有ICMP的replay，却没有找到ICMP的request消息，这是为什么呢？这是因为：我们的数据包是在TC这个layer.而tcpdump的是在TC层以前，所以我们能抓到一条消息。而经过lxc以后就被bpf_redirect_peer()到了Pod-B的eth0网卡，所以我们在Pod-B的lxc网卡此时应该抓不到ICMP的request的。因为这个request的ICMP 是被Pod-A直接丢到Pod-B的eth0网卡了。而我们为什么又能抓到一个replay呢。那就是tcpdump的抓包在TC层之前。

那么Pod-A的lxc上应该是有ICMP 的Request了。我们验证一下： [root@dev1 ~]# kubectl exec -it cni-client-7ccd98bdb8-b98hw — ethtool -S eth0 NIC statistics: peer_ifindex: 20 rx_queue_0_xdp_packets: 0 rx_queue_0_xdp_bytes: 0 rx_queue_0_drops: 0 rx_queue_0_xdp_redirect: 0 rx_queue_0_xdp_drops: 0 rx_queue_0_xdp_tx: 0 rx_queue_0_xdp_tx_errors: 0 tx_queue_0_xdp_xmit: 0 tx_queue_0_xdp_xmit_errors: 0 [root@dev1 ~]# tcpdump -pne -i lxc9e147bf7e5a5 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on lxc9e147bf7e5a5, link-type EN10MB (Ethernet), capture size 262144 bytes 22:12:32.652557 8a:68:cf:97:7b:1b > a6:94:05:be:ed:24, ethertype IPv4 (0x0800), length 98: 10.244.0.168 > 10.244.0.246: ICMP echo request, id 12288, seq 0, length 64 22:12:33.653660 8a:68:cf:97:7b:1b > a6:94:05:be:ed:24, ethertype IPv4 (0x0800), length 98: 10.244.0.168 > 10.244.0.246: ICMP echo request, id 12288, seq 1, length 64 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel [root@dev1 ~]#

此时HOST NS上的iptables也没有处理 [root@dev1 ~]# tailf /var/log/messages | grep -i icmp ^C [root@dev1 ~]#


- [x] **2.不同节点Pod通信：**
```properties
env:
[root@dev1 ~]# kubectl get pod -o wide 
NAME                          READY   STATUS    RESTARTS   AGE   IP             NODE   NOMINATED NODE   READINESS GATES
cni-client-7ccd98bdb8-b98hw   1/1     Running   1          14h   10.244.0.151   dev1   <none>           <none>
cni-server-7559c58f9-fjxkx    1/1     Running   0          14h   10.244.2.72    dev3   <none>           <none>

同样通过iptables TRACE来看：
iptables -t raw -I PREROUTING -p icmp -j TRACE
iptables -t raw -I PREROUTING -p udp --dport=8472 -j TRACE

同样我们可以得到是在VXLAN的封装完成以后，需要经过HOST NS的iptables 处理。我们是在POD-A对应的HOST上的iptables 抓取的信息。我们同时也需要在Pod-B对应的HOST上抓包：
我们先提供一个图：
[root@dev3 ~]# kubectl get nodes -o wide 
NAME   STATUS   ROLES                  AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
dev1   Ready    control-plane,master   30d   v1.20.5   192.168.2.31   <none>        CentOS Linux 7 (Core)   5.15.8-1.el7.elrepo.x86_64   docker://20.10.12
dev2   Ready    <none>                 30d   v1.20.5   192.168.2.32   <none>        CentOS Linux 7 (Core)   5.15.8-1.el7.elrepo.x86_64   docker://20.10.12
dev3   Ready    <none>                 30d   v1.20.5   192.168.2.33   <none>        CentOS Linux 7 (Core)   5.15.8-1.el7.elrepo.x86_64   docker://20.10.12
[root@dev3 ~]# 
这里我们的kernel是大于5.10的：
[这里我们需要注意一点：在pod出去的时候需要两边HOST都需要TRACE一下。注意这里是Native Routing的Mode，和我们的VXLAN的Mode有些区别的]

[依照上图：我们可以看出在跨Pod通信的场景中：在到达侧一边是经过iptables处理，一边则不需要经过iptables处理]
我们在cilium中使用cilium monitor -vv 来观察：
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..86..] SrcMAC=be:91:42:23:f0:ce DstMAC=1e:5d:a8:e7:e2:ed EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..64..] Version=4 IHL=5 TOS=0 Length=84 Id=58463 Flags=DF FragOffset=0 TTL=64 Protocol=ICMPv4 Checksum=15747 SrcIP=10.244.0.151 DstIP=10.244.2.72 Options=[] Padding=[]}
ICMPv4  {Contents=[..8..] Payload=[..56..] TypeCode=EchoRequest Checksum=22668 Id=19968 Seq=0}
CPU 04: MARK 0x0 FROM 183 from-endpoint: 98 bytes (98 captured), state newidentity 22099->unknown, orig-ip 0.0.0.0
CPU 04: MARK 0x0 FROM 183 DEBUG: Conntrack lookup 1/2: src=10.244.0.151:19968 dst=10.244.2.72:0
CPU 04: MARK 0x0 FROM 183 DEBUG: Conntrack lookup 2/2: nexthdr=1 flags=1
CPU 04: MARK 0x0 FROM 183 DEBUG: CT verdict: New, revnat=0
CPU 04: MARK 0x0 FROM 183 DEBUG: Successfully mapped addr=10.244.2.72 to identity=2050
CPU 04: MARK 0x0 FROM 183 DEBUG: Conntrack create: proxy-port=0 revnat=0 src-identity=22099 lb=0.0.0.0
CPU 04: MARK 0x0 FROM 183 DEBUG: Encapsulating to node 3232236065 (0xc0a80221) from seclabel 22099
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..86..] SrcMAC=be:91:42:23:f0:ce DstMAC=1e:5d:a8:e7:e2:ed EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..64..] Version=4 IHL=5 TOS=0 Length=84 Id=58463 Flags=DF FragOffset=0 TTL=64 Protocol=ICMPv4 Checksum=15747 SrcIP=10.244.0.151 DstIP=10.244.2.72 Options=[] Padding=[]}
ICMPv4  {Contents=[..8..] Payload=[..56..] TypeCode=EchoRequest Checksum=22668 Id=19968 Seq=0}
CPU 04: MARK 0x0 FROM 183 to-overlay: 98 bytes (98 captured), state new, interface cilium_vxlanidentity 22099->unknown, orig-ip 0.0.0.0
CPU 04: MARK 0x0 FROM 472 DEBUG: Conntrack lookup 1/2: src=192.168.2.31:47563 dst=192.168.2.33:8472
CPU 04: MARK 0x0 FROM 472 DEBUG: Conntrack lookup 2/2: nexthdr=17 flags=1
CPU 04: MARK 0x0 FROM 472 DEBUG: CT verdict: New, revnat=0
CPU 04: MARK 0x0 FROM 472 DEBUG: Conntrack create: proxy-port=0 revnat=0 src-identity=0 lb=0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=00:0c:29:9f:c8:16 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=134 Id=19050 Flags= FragOffset=0 TTL=64 Protocol=UDP Checksum=43628 SrcIP=192.168.2.31 DstIP=192.168.2.33 Options=[] Padding=[]}
UDP     {Contents=[..8..] Payload=[..90..] SrcPort=47563 DstPort=8472(otv) Length=114 Checksum=0}
  Packet has been truncated
CPU 04: MARK 0x0 FROM 472 to-network: 148 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=00:0c:29:9f:c8:16 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=1880 Id=27124 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=17435 SrcIP=192.168.2.31 DstIP=192.168.2.33 Options=[] Padding=[]}
TCP     {Contents=[..32..] Payload=[..66..] SrcPort=6443(sun-sr-https) DstPort=46990 Seq=698525411 Ack=2037994976 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=501 Checksum=36059 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:3008050224/643412509 0xb34b34302659b21d)] Padding=[]}
  Packet has been truncated
CPU 04: MARK 0x7a29f113 FROM 472 to-network: 1894 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:9f:c8:16 DstMAC=00:0c:29:34:a3:73 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=134 Id=16999 Flags= FragOffset=0 TTL=64 Protocol=UDP Checksum=45679 SrcIP=192.168.2.33 DstIP=192.168.2.31 Options=[] Padding=[]}
UDP     {Contents=[..8..] Payload=[..90..] SrcPort=48280 DstPort=8472(otv) Length=114 Checksum=0}
  Packet has been truncated
CPU 03: MARK 0x0 FROM 472 from-network: 148 bytes (128 captured), state new, interface ens33, orig-ip 0.0.0.0
CPU 03: MARK 0x0 FROM 472 DEBUG: Successfully mapped addr=192.168.2.33 to identity=6
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..86..] SrcMAC=02:a1:5c:50:a7:8a DstMAC=8a:ed:7c:a6:7a:ff EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..64..] Version=4 IHL=5 TOS=0 Length=84 Id=17278 Flags= FragOffset=0 TTL=64 Protocol=ICMPv4 Checksum=7781 SrcIP=10.244.2.72 DstIP=10.244.0.151 Options=[] Padding=[]}
ICMPv4  {Contents=[..8..] Payload=[..56..] TypeCode=EchoReply Checksum=24716 Id=19968 Seq=0}
CPU 03: MARK 0x0 FROM 0 from-overlay: 98 bytes (98 captured), state new, interface cilium_vxlan, orig-ip 0.0.0.0
CPU 03: MARK 0x0 FROM 0 DEBUG: Tunnel decap: id=2050 flowlabel=0
CPU 03: MARK 0x0 FROM 0 DEBUG: Attempting local delivery for container id 183 from seclabel 2050
CPU 03: MARK 0x0 FROM 183 DEBUG: Conntrack lookup 1/2: src=10.244.2.72:0 dst=10.244.0.151:19968
CPU 03: MARK 0x0 FROM 183 DEBUG: Conntrack lookup 2/2: nexthdr=1 flags=0
CPU 03: MARK 0x0 FROM 183 DEBUG: CT entry found lifetime=6227, revnat=0
CPU 03: MARK 0x0 FROM 183 DEBUG: CT verdict: Reply, revnat=0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..86..] SrcMAC=1e:5d:a8:e7:e2:ed DstMAC=be:91:42:23:f0:ce EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..64..] Version=4 IHL=5 TOS=0 Length=84 Id=17278 Flags= FragOffset=0 TTL=63 Protocol=ICMPv4 Checksum=8037 SrcIP=10.244.2.72 DstIP=10.244.0.151 Options=[] Padding=[]}
ICMPv4  {Contents=[..8..] Payload=[..56..] TypeCode=EchoReply Checksum=24716 Id=19968 Seq=0}
CPU 03: MARK 0x0 FROM 183 to-endpoint: 98 bytes (98 captured), state reply, interface lxce5049896181bidentity 2050->22099, orig-ip 10.244.2.72, to endpoint 183
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=00:0c:29:9f:c8:16 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=2603 Id=27126 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=16710 SrcIP=192.168.2.31 DstIP=192.168.2.33 Options=[] Padding=[]}
TCP     {Contents=[..32..] Payload=[..66..] SrcPort=6443(sun-sr-https) DstPort=46990 Seq=698527239 Ack=2037994976 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=501 Checksum=36782 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:3008050225/643412509 0xb34b34312659b21d)] Padding=[]}
  Packet has been truncated
CPU 03: MARK 0x7a29f113 FROM 472 to-network: 2617 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=00:0c:29:9f:c8:16 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=3086 Id=27128 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=16225 SrcIP=192.168.2.31 DstIP=192.168.2.33 Options=[] Padding=[]}
TCP     {Contents=[..32..] Payload=[..66..] SrcPort=6443(sun-sr-https) DstPort=46990 Seq=698529790 Ack=2037994976 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=501 Checksum=37265 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:3008050226/643412509 0xb34b34322659b21d)] Padding=[]}
  Packet has been truncated
CPU 03: MARK 0x7a29f113 FROM 472 to-network: 3100 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=00:0c:29:9f:c8:16 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=2462 Id=27131 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=16846 SrcIP=192.168.2.31 DstIP=192.168.2.33 Options=[] Padding=[]}
TCP     {Contents=[..32..] Payload=[..66..] SrcPort=6443(sun-sr-https) DstPort=46990 Seq=698532824 Ack=2037994976 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=501 Checksum=36641 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:3008050227/643412509 0xb34b34332659b21d)] Padding=[]}
  Packet has been truncated
CPU 03: MARK 0x7a29f113 FROM 472 to-network: 2476 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=b0:22:7a:db:86:d2 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=16 Length=376 Id=27651 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=18427 SrcIP=192.168.2.31 DstIP=192.168.2.2 Options=[] Padding=[]}
TCP     {Contents=[..20..] Payload=[..78..] SrcPort=22(ssh) DstPort=5077 Seq=1267197329 Ack=3665293808 DataOffset=5 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=501 Checksum=34524 Urgent=0 Options=[] Padding=[]}
  Packet has been truncated
CPU 04: MARK 0xfcf49389 FROM 472 to-network: 390 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..118..] SrcMAC=00:0c:29:34:a3:73 DstMAC=00:0c:29:9f:c8:16 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..98..] Version=4 IHL=5 TOS=0 Length=1776 Id=27133 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=17530 SrcIP=192.168.2.31 DstIP=192.168.2.33 Options=[] Padding=[]}
TCP     {Contents=[..32..] Payload=[..66..] SrcPort=6443(sun-sr-https) DstPort=46990 Seq=698535234 Ack=2037994976 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=501 Checksum=35955 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:3008050228/643412509 0xb34b34342659b21d)] Padding=[]}
  Packet has been truncated
CPU 03: MARK 0x7a29f113 FROM 472 to-network: 1790 bytes (128 captured), state new, orig-ip 0.0.0.0
------------------------------------------------------------------------------
Ethernet        {Contents=[..14..] Payload=[..54..] SrcMAC=00:0c:29:9f:c8:16 DstMAC=00:0c:29:34:a3:73 EthernetType=IPv4 Length=0}
IPv4    {Contents=[..20..] Payload=[..32..] Version=4 IHL=5 TOS=0 Length=52 Id=16697 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=29690 SrcIP=192.168.2.33 DstIP=192.168.2.31 Options=[] Padding=[]}
TCP     {Contents=[..32..] Payload=[] SrcPort=46990 DstPort=6443(sun-sr-https) Seq=2037994976 Ack=698536958 DataOffset=8 FIN=false SYN=false RST=false PSH=false ACK=true URG=false ECE=false CWR=false NS=false Window=148 Checksum=37611 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:643412541/3008050198 0x2659b23db34b3416)] Padding=[]}
CPU 03: MARK 0x0 FROM 472 from-network: 66 bytes (66 captured), state new, interface ens33, orig-ip 0.0.0.0
CPU 03: MARK 0x0 FROM 472 DEBUG: Successfully mapped addr=192.168.2.33 to identity=6
-----------------------------------------------------------------------------
这里我们可以看到：

当数据出去的时候，经过两次：Conntrack lookup 。而回来的时候只经过了一次Conntrack lookup 。
其中每次都有的这一次是指的是Pod中的Conntrack lookup 。