从kubelet到runc

kubelet会创建pod的cgroup

3111是kubelet
2670是containerd
2473574 是pod pause进程
2473588 是init container的containerd-shim-runc-v2
7386 是flanneld
6569 是kube-proxy

  1. 准备需要的volume
  2. 调用containerd 启动sandbox
    1. containerd 调用cni准备网络
    2. containerd 启动sandbox pause容器
  3. 调用containerd 执行 init-container
  4. 调用containerd 启动 container ```c mount 2473512 3111 0 /usr/bin/mount -t tmpfs -o size=5750771712 tmpfs /var/lib/kubelet/pods/c9a601b2-1193-4c97-8a94-a19b235b2930/volumes/kubernetes.io~secret/webhook-cert mount 2473513 3111 0 /usr/bin/mount -t tmpfs -o size=5750771712 tmpfs /var/lib/kubelet/pods/c9a601b2-1193-4c97-8a94-a19b235b2930/volumes/kubernetes.io~projected/kube-api-access-dkddq

—— containerd调用cni loopback 2473514 2670 0 /opt/cni/bin/loopback flannel 2473520 2670 0 /opt/cni/bin/flannel bridge 2473525 2473520 0 /opt/cni/bin/bridge host-local 2473534 2473525 0 /opt/cni/bin/host-local systemd-sysctl 2473535 2473533 0 /usr/lib/systemd/systemd-sysctl —prefix=/net/ipv4/conf/veth1dd8fef6 —prefix=/net/ipv4/neigh/veth1dd8fef6 —prefix=/net/ipv6/conf/veth1dd8fef6 —prefix=/net/ipv6/neigh/veth1dd8fef6 portmap 2473539 2670 0 /opt/cni/bin/portmap

—— pod pause container启动 containerd-shim 2473546 2670 0 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd -id 9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda start containerd-shim 2473554 2473546 0 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda -address /run/containerd/containerd.sock runc 2473563 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda/log.json —log-format json —systemd-cgroup create —bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda —pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda/init.pid 9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda exe 2473571 2473563 0 /proc/self/exe init runc 2473582 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda/log.json —log-format json —systemd-cgroup start 9ec2f26f9b44cb0ab12dd4bf20f1f9358763c160765f76fba64331d07107dcda pause 2473574 2473554 0 /pause

—— init-container 执行 containerd-shim 2473588 2670 0 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd -id 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b start runc 2473596 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b/log.json —log-format json —systemd-cgroup create —bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b —pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b/init.pid 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b exe 2473603 2473596 0 /proc/self/exe init runc 2473612 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b/log.json —log-format json —systemd-cgroup start 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b sh 2473606 2473554 0 /bin/sh -c if [ “$POD_IP” != “$HOST_IP” ]; then\nmount -o remount rw /proc/sys\nsysctl -w net.core.somaxconn=65535\nsysctl -w net.ipv4.ip_loca mount 2473618 2473606 0 /bin/mount -o remount rw /proc/sys sysctl 2473619 2473606 0 /bin/sysctl -w net.core.somaxconn=65535 sysctl 2473620 2473606 0 /bin/sysctl -w net.ipv4.ip_local_port_range=1024 65535 sysctl 2473621 2473606 0 /bin/sysctl -w kernel.core_uses_pid=0 runc 2473623 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b/log.json —log-format json —systemd-cgroup kill —all 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b 9 runc 2473629 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b/log.json —log-format json —systemd-cgroup delete 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b containerd-shim 2473637 2670 0 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd -id 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b -bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b delete runc 2473646 2473637 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b/log.json —log-format json delete —force 65d72ea795f9a40dfba93dff077ce56c7c6fb44f64ee325bfba22b79500e8e9b

—— flanneld在执行一些动作 iptables 2473652 7386 0 /sbin/iptables -t filter -C FORWARD -s 172.20.16.0/20 -j ACCEPT —wait iptables 2473653 7386 0 /sbin/iptables -t filter -C FORWARD -d 172.20.16.0/20 -j ACCEPT —wait iptables 2473654 7386 0 /sbin/iptables -t nat -C POSTROUTING -s 172.20.16.0/20 -d 172.20.16.0/20 -j RETURN —wait iptables 2473655 7386 0 /sbin/iptables -t nat -C POSTROUTING -s 172.20.16.0/20 ! -d 224.0.0.0/4 -j MASQUERADE —random-fully —wait iptables 2473656 7386 0 /sbin/iptables -t nat -C POSTROUTING ! -s 172.20.16.0/20 -d 172.20.16.64/26 -j RETURN —wait iptables 2473657 7386 0 /sbin/iptables -t nat -C POSTROUTING ! -s 172.20.16.0/20 -d 172.20.16.0/20 -j MASQUERADE —random-fully —wait iptables 2473658 7386 0 /sbin/iptables -t nat -C POSTROUTING -s 172.20.16.0/20 ! -d 224.0.0.0/4 -j MASQUERADE —wait

—— kube-proxy在执行 iptables-save 2473659 6569 0 /usr/sbin/iptables-save -t filter iptables-save 2473660 6569 0 /usr/sbin/iptables-save -t nat iptables 2473661 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-MARK-DROP -t nat iptables 2473662 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-SERVICES -t nat iptables 2473663 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-POSTROUTING -t nat iptables 2473664 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-FIREWALL -t nat iptables 2473665 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-NODE-PORT -t nat iptables 2473666 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-LOAD-BALANCER -t nat iptables 2473667 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-MARK-MASQ -t nat iptables 2473668 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-FORWARD -t filter iptables 2473669 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-NODE-PORT -t filter iptables 2473670 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C OUTPUT -t nat -m comment —comment kubernetes service portals -j KUBE-SERVICES iptables 2473671 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C PREROUTING -t nat -m comment —comment kubernetes service portals -j KUBE-SERVICES iptables 2473672 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C POSTROUTING -t nat -m comment —comment kubernetes postrouting rules -j KUBE-POSTROUTING iptables 2473673 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C FORWARD -t filter -m comment —comment kubernetes forwarding rules -j KUBE-FORWARD iptables 2473674 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C INPUT -t filter -m comment —comment kubernetes health check rules -j KUBE-NODE-PORT ipset 2473675 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-SOURCE-IP hash:ip,port,ip family inet hashsize 1024 maxelem 65536 -exist ipset 2473676 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-SOURCE-CIDR hash:ip,port,net family inet hashsize 1024 maxelem 65536 -exist ipset 2473677 6569 0 /sbin/ipset create KUBE-NODE-PORT-LOCAL-TCP bitmap:port range 0-65535 -exist ipset 2473678 6569 0 /sbin/ipset create KUBE-CLUSTER-IP hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473679 6569 0 /sbin/ipset create KUBE-NODE-PORT-TCP bitmap:port range 0-65535 -exist ipset 2473680 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-FW hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473681 6569 0 /sbin/ipset create KUBE-EXTERNAL-IP hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473682 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473683 6569 0 /sbin/ipset create KUBE-NODE-PORT-LOCAL-SCTP-HASH hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473684 6569 0 /sbin/ipset create KUBE-LOOP-BACK hash:ip,port,ip family inet hashsize 1024 maxelem 65536 -exist ipset 2473685 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-LOCAL hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473686 6569 0 /sbin/ipset create KUBE-NODE-PORT-UDP bitmap:port range 0-65535 -exist ipset 2473687 6569 0 /sbin/ipset create KUBE-NODE-PORT-LOCAL-UDP bitmap:port range 0-65535 -exist ipset 2473688 6569 0 /sbin/ipset create KUBE-NODE-PORT-SCTP-HASH hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473689 6569 0 /sbin/ipset create KUBE-HEALTH-CHECK-NODE-PORT bitmap:port range 0-65535 -exist ipset 2473690 6569 0 /sbin/ipset create KUBE-EXTERNAL-IP-LOCAL hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473691 6569 0 /sbin/ipset list KUBE-NODE-PORT-SCTP-HASH ipset 2473692 6569 0 /sbin/ipset list KUBE-HEALTH-CHECK-NODE-PORT ipset 2473693 6569 0 /sbin/ipset list KUBE-EXTERNAL-IP-LOCAL ipset 2473694 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-LOCAL ipset 2473695 6569 0 /sbin/ipset list KUBE-NODE-PORT-UDP ipset 2473696 6569 0 /sbin/ipset list KUBE-NODE-PORT-LOCAL-UDP ipset 2473697 6569 0 /sbin/ipset list KUBE-CLUSTER-IP ipset 2473698 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-SOURCE-IP ipset 2473699 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-SOURCE-CIDR ipset 2473700 6569 0 /sbin/ipset list KUBE-NODE-PORT-LOCAL-TCP ipset 2473701 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-FW ipset 2473702 6569 0 /sbin/ipset list KUBE-NODE-PORT-TCP ipset 2473703 6569 0 /sbin/ipset list KUBE-LOOP-BACK ipset 2473704 6569 0 /sbin/ipset add KUBE-LOOP-BACK 172.20.16.69,tcp:8443,172.20.16.69 -exist ipset 2473705 6569 0 /sbin/ipset list KUBE-EXTERNAL-IP ipset 2473706 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER ipset 2473707 6569 0 /sbin/ipset list KUBE-NODE-PORT-LOCAL-SCTP-HASH iptables-restor 2473708 6569 0 /usr/sbin/iptables-restore -w 5 -W 100000 —noflush —counters iptables-save 2473709 6569 0 /usr/sbin/iptables-save -t filter iptables-save 2473710 6569 0 /usr/sbin/iptables-save -t nat iptables 2473711 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-MARK-DROP -t nat iptables 2473712 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-SERVICES -t nat iptables 2473713 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-POSTROUTING -t nat iptables 2473714 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-FIREWALL -t nat iptables 2473715 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-NODE-PORT -t nat iptables 2473716 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-LOAD-BALANCER -t nat iptables 2473717 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-MARK-MASQ -t nat iptables 2473718 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-FORWARD -t filter iptables 2473719 6569 0 /usr/sbin/iptables -w 5 -W 100000 -N KUBE-NODE-PORT -t filter iptables 2473720 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C OUTPUT -t nat -m comment —comment kubernetes service portals -j KUBE-SERVICES iptables 2473721 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C PREROUTING -t nat -m comment —comment kubernetes service portals -j KUBE-SERVICES iptables 2473722 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C POSTROUTING -t nat -m comment —comment kubernetes postrouting rules -j KUBE-POSTROUTING iptables 2473723 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C FORWARD -t filter -m comment —comment kubernetes forwarding rules -j KUBE-FORWARD iptables 2473724 6569 0 /usr/sbin/iptables -w 5 -W 100000 -C INPUT -t filter -m comment —comment kubernetes health check rules -j KUBE-NODE-PORT ipset 2473725 6569 0 /sbin/ipset create KUBE-CLUSTER-IP hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473726 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-SOURCE-IP hash:ip,port,ip family inet hashsize 1024 maxelem 65536 -exist ipset 2473727 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-SOURCE-CIDR hash:ip,port,net family inet hashsize 1024 maxelem 65536 -exist ipset 2473728 6569 0 /sbin/ipset create KUBE-NODE-PORT-LOCAL-TCP bitmap:port range 0-65535 -exist ipset 2473729 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-FW hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473730 6569 0 /sbin/ipset create KUBE-NODE-PORT-TCP bitmap:port range 0-65535 -exist ipset 2473731 6569 0 /sbin/ipset create KUBE-LOOP-BACK hash:ip,port,ip family inet hashsize 1024 maxelem 65536 -exist ipset 2473732 6569 0 /sbin/ipset create KUBE-EXTERNAL-IP hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473733 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473734 6569 0 /sbin/ipset create KUBE-NODE-PORT-LOCAL-SCTP-HASH hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473735 6569 0 /sbin/ipset create KUBE-EXTERNAL-IP-LOCAL hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473736 6569 0 /sbin/ipset create KUBE-LOAD-BALANCER-LOCAL hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473737 6569 0 /sbin/ipset create KUBE-NODE-PORT-UDP bitmap:port range 0-65535 -exist ipset 2473738 6569 0 /sbin/ipset create KUBE-NODE-PORT-LOCAL-UDP bitmap:port range 0-65535 -exist ipset 2473739 6569 0 /sbin/ipset create KUBE-NODE-PORT-SCTP-HASH hash:ip,port family inet hashsize 1024 maxelem 65536 -exist ipset 2473740 6569 0 /sbin/ipset create KUBE-HEALTH-CHECK-NODE-PORT bitmap:port range 0-65535 -exist ipset 2473741 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-SOURCE-IP ipset 2473742 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-SOURCE-CIDR ipset 2473743 6569 0 /sbin/ipset list KUBE-NODE-PORT-LOCAL-TCP ipset 2473744 6569 0 /sbin/ipset list KUBE-CLUSTER-IP ipset 2473745 6569 0 /sbin/ipset list KUBE-NODE-PORT-TCP ipset 2473746 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-FW ipset 2473747 6569 0 /sbin/ipset list KUBE-EXTERNAL-IP ipset 2473748 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER ipset 2473749 6569 0 list KUBE-NODE-PORT-LOCAL-SCTP-HASH ipset 2473750 6569 0 /sbin/ipset list KUBE-LOOP-BACK ipset 2473751 6569 0 /sbin/ipset add KUBE-LOOP-BACK 172.20.16.69,tcp:443,172.20.16.69 -exist ipset 2473752 6569 0 /sbin/ipset add KUBE-LOOP-BACK 172.20.16.69,tcp:80,172.20.16.69 -exist ipset 2473753 6569 0 /sbin/ipset list KUBE-LOAD-BALANCER-LOCAL ipset 2473754 6569 0 /sbin/ipset list KUBE-NODE-PORT-UDP ipset 2473755 6569 0 /sbin/ipset list KUBE-NODE-PORT-LOCAL-UDP ipset 2473756 6569 0 /sbin/ipset list KUBE-NODE-PORT-SCTP-HASH ipset 2473757 6569 0 /sbin/ipset list KUBE-HEALTH-CHECK-NODE-PORT ipset 2473758 6569 0 /sbin/ipset list KUBE-EXTERNAL-IP-LOCAL iptables-restor 2473759 6569 0 /usr/sbin/iptables-restore -w 5 -W 100000 —noflush —counters unpigz 2473760 2670 0 /usr/bin/unpigz -d -c iptables 2473764 3111 0 /usr/sbin/iptables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle ip6tables 2473765 3111 0 /usr/sbin/ip6tables -w 5 -W 100000 -S KUBE-KUBELET-CANARY -t mangle sleep 2473766 6962 0 /usr/bin/sleep 3

—— containerd在准备 unpigz 2473767 2670 0 /usr/bin/unpigz -d -c unpigz 2473771 2670 0 /usr/bin/unpigz -d -c unpigz 2473775 2670 0 /usr/bin/unpigz -d -c unpigz 2473779 2670 0 /usr/bin/unpigz -d -c unpigz 2473783 2670 0 /usr/bin/unpigz -d -c unpigz 2473787 2670 0 /usr/bin/unpigz -d -c unpigz 2473791 2670 0 /usr/bin/unpigz -d -c unpigz 2473795 2670 0 /usr/bin/unpigz -d -c unpigz 2473799 2670 0 /usr/bin/unpigz -d -c unpigz 2473803 2670 0 /usr/bin/unpigz -d -c unpigz 2473807 2670 0 /usr/bin/unpigz -d -c unpigz 2473811 2670 0 /usr/bin/unpigz -d -c unpigz 2473815 2670 0 /usr/bin/unpigz -d -c unpigz 2473819 2670 0 /usr/bin/unpigz -d -c unpigz 2473823 2670 0 /usr/bin/unpigz -d -c sleep 2473827 6962 0 /usr/bin/sleep 3 ip6tables-save 2473828 6569 0 /usr/sbin/ip6tables-save -t filter ip6tables-save 2473829 6569 0 /usr/sbin/ip6tables-save -t nat ip6tables 2473830 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-MARK-DROP -t nat ip6tables 2473831 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-SERVICES -t nat ip6tables 2473832 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-POSTROUTING -t nat ip6tables 2473833 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-FIREWALL -t nat ip6tables 2473834 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-NODE-PORT -t nat ip6tables 2473835 6569 0 ip6tables 2473836 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-MARK-MASQ -t nat ip6tables 2473837 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-FORWARD -t filter ip6tables 2473838 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -N KUBE-NODE-PORT -t filter ip6tables 2473839 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -C OUTPUT -t nat -m comment —comment kubernetes service portals -j KUBE-SERVICES ip6tables 2473840 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -C PREROUTING -t nat -m comment —comment kubernetes service portals -j KUBE-SERVICES ip6tables 2473841 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -C POSTROUTING -t nat -m comment —comment kubernetes postrouting rules -j KUBE-POSTROUTING ip6tables 2473842 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -C FORWARD -t filter -m comment —comment kubernetes forwarding rules -j KUBE-FORWARD ip6tables 2473843 6569 0 /usr/sbin/ip6tables -w 5 -W 100000 -C INPUT -t filter -m comment —comment kubernetes health check rules -j KUBE-NODE-PORT ipset 2473844 6569 0 /sbin/ipset create KUBE-6-LOOP-BACK hash:ip,port,ip family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473845 6569 0 /sbin/ipset create KUBE-6-HEALTH-CHECK-NODE-PORT bitmap:port range 0-65535 -exist ipset 2473846 6569 0 /sbin/ipset create KUBE-6-NODE-PORT-UDP bitmap:port range 0-65535 -exist ipset 2473847 6569 0 /sbin/ipset create KUBE-6-NODE-PORT-LOCAL-UDP bitmap:port range 0-65535 -exist ipset 2473848 6569 0 /sbin/ipset create KUBE-6-NODE-PORT-SCTP-HASH hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473849 6569 0 /sbin/ipset create KUBE-6-EXTERNAL-IP hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473850 6569 0 /sbin/ipset create KUBE-6-EXTERNAL-IP-LOCAL hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473851 6569 0 /sbin/ipset create KUBE-6-LOAD-BALANCER-FW hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473852 6569 0 /sbin/ipset create KUBE-6-LOAD-BALANCER-SOURCE-IP hash:ip,port,ip family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473853 6569 0 /sbin/ipset create KUBE-6-LOAD-BALANCER-SOURCE-CID hash:ip,port,net family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473854 6569 0 /sbin/ipset create KUBE-6-CLUSTER-IP hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473855 6569 0 /sbin/ipset create KUBE-6-NODE-PORT-LOCAL-SCTP-HAS hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473856 6569 0 /sbin/ipset create KUBE-6-LOAD-BALANCER hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473857 6569 0 /sbin/ipset create KUBE-6-LOAD-BALANCER-LOCAL hash:ip,port family inet6 hashsize 1024 maxelem 65536 -exist ipset 2473858 6569 0 /sbin/ipset create KUBE-6-NODE-PORT-TCP bitmap:port range 0-65535 -exist ipset 2473859 6569 0 /sbin/ipset create KUBE-6-NODE-PORT-LOCAL-TCP bitmap:port range 0-65535 -exist ipset 2473860 6569 0 /sbin/ipset list KUBE-6-LOOP-BACK ipset 2473861 6569 0 /sbin/ipset list KUBE-6-HEALTH-CHECK-NODE-PORT ipset 2473862 6569 0 /sbin/ipset list KUBE-6-LOAD-BALANCER-SOURCE-IP ipset 2473863 6569 0 /sbin/ipset list KUBE-6-LOAD-BALANCER-SOURCE-CID ipset 2473864 6569 0 /sbin/ipset list KUBE-6-NODE-PORT-UDP ipset 2473865 6569 0 /sbin/ipset list KUBE-6-NODE-PORT-LOCAL-UDP ipset 2473866 6569 0 /sbin/ipset list KUBE-6-NODE-PORT-SCTP-HASH ipset 2473867 6569 0 /sbin/ipset list KUBE-6-EXTERNAL-IP ipset 2473868 6569 0 /sbin/ipset list KUBE-6-EXTERNAL-IP-LOCAL ipset 2473869 6569 0 /sbin/ipset list KUBE-6-LOAD-BALANCER-FW unpigz 2473870 2670 0 /usr/bin/unpigz -d -c ipset 2473871 6569 0 /sbin/ipset list KUBE-6-CLUSTER-IP ipset 2473874 6569 0 /sbin/ipset list KUBE-6-NODE-PORT-LOCAL-SCTP-HAS ipset 2473876 6569 0 /sbin/ipset list KUBE-6-NODE-PORT-LOCAL-TCP ipset 2473877 6569 0 /sbin/ipset list KUBE-6-LOAD-BALANCER ipset 2473878 6569 0 /sbin/ipset list KUBE-6-LOAD-BALANCER-LOCAL ipset 2473879 6569 0 /sbin/ipset list KUBE-6-NODE-PORT-TCP ip6tables-resto 2473880 6569 0 /usr/sbin/ip6tables-restore -w 5 -W 100000 —noflush —counters unpigz 2473881 2670 0 /usr/bin/unpigz -d -c iptables 2473885 7386 0 /sbin/iptables -t filter -C FORWARD -s 172.20.16.0/20 -j ACCEPT —wait iptables 2473886 7386 0 /sbin/iptables -t filter -C FORWARD -d 172.20.16.0/20 -j ACCEPT —wait iptables 2473887 7386 0 /sbin/iptables -t nat -C POSTROUTING -s 172.20.16.0/20 -d 172.20.16.0/20 -j RETURN —wait iptables 2473888 7386 0 /sbin/iptables -t nat -C POSTROUTING -s 172.20.16.0/20 ! -d 224.0.0.0/4 -j MASQUERADE —random-fully —wait iptables 2473889 7386 0 /sbin/iptables -t nat -C POSTROUTING ! -s 172.20.16.0/20 -d 172.20.16.64/26 -j RETURN —wait iptables 2473890 7386 0 /sbin/iptables -t nat -C POSTROUTING ! -s 172.20.16.0/20 -d 172.20.16.0/20 -j MASQUERADE —random-fully —wait iptables 2473891 7386 0 /sbin/iptables -t nat -C POSTROUTING -s 172.20.16.0/20 ! -d 224.0.0.0/4 -j MASQUERADE —wait unpigz 2473892 2670 0 /usr/bin/unpigz -d -c

—— containerd启动pod内的容器 containerd-shim 2473896 2670 0 /usr/bin/containerd-shim-runc-v2 -namespace k8s.io -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd -id 1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2 start runc 2473905 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2/log.json —log-format json —systemd-cgroup create —bundle /run/containerd/io.containerd.runtime.v2.task/k8s.io/1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2 —pid-file /run/containerd/io.containerd.runtime.v2.task/k8s.io/1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2/init.pid 1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2 exe 2473912 2473905 0 /proc/self/exe init runc 2473922 2473554 0 /usr/bin/runc —root /run/containerd/runc/k8s.io —log /run/containerd/io.containerd.runtime.v2.task/k8s.io/1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2/log.json —log-format json —systemd-cgroup start 1dc73e4ac171a216269a947b9d150b33756b784f698ac570eb95e25d3640d1a2 dumb-init 2473916 2473554 0 /usr/bin/dumb-init — /nginx-ingress-controller —publish-service=kube-system/ack-ingress-nginx-v1-default-controller —election-id=ingress-controller-leader-ack-nginx —controller-class=k8s.io/ack-ingress-nginx —ingress-class=ack-nginx —configmap=kube-system/ack-ingress-nginx-v1-default-controller —validating-webhook=:8443 —validating-webhook-certificate=/usr/local/certificates/cert —validating-webhook-key=/usr/local/certificates/key —v=2 nginx-ingress-c 2473928 2473916 0 /nginx-ingress-controller —publish-service=kube-system/ack-ingress-nginx-v1-default-controller —election-id=ingress-controller-leader-ack-nginx —controller-class=k8s.io/ack-ingress-nginx —ingress-class=ack-nginx —configmap=kube-system/ack-ingress-nginx-v1-default-controller —validating-webhook=:8443 —validating-webhook-certificate=/usr/local/certificates/cert —validating-webhook-key=/usr/local/certificates/key —v=2 nginx 2473935 2473928 0 /sbin/nginx -v

  1. <a name="VkJnn"></a>
  2. # 从nerdctl到runc
  4. 1. nerdctl请求containerd
  5. 1. containerd fork启动 containerd-shim-runc-v2，containerd-shim-runc-v2会再fork启动 containerd-shim-runc-v2 
  6. 1. containerd-shim-runc-v2  fork启动runc create。
  7.    1. 这里看runc  create，是创建一个容器。”--systemd-cgroup“ 表示cgroup是systemd
  8.    1. runc启动一个  nerdctl --cgroup-manager=systemd internal oci-hook createRuntime 。这是”createRuntime“的hook，是nerdctl生成的。这个命令是通过cni来初始化网络
  9. 4. containerd-shim-runc-v2  fork启动runc start。 
  10. ```c
  11. PCOMM            PID    PPID   RET ARGS
  12. nerdctl          660596 598700   0 ./nerdctl run -it --cgroup-manager=systemd --entrypoint bash registry-vpc.cn-hangzhou.aliyuncs.com/acs/node-problem-detector:v0.8.10-e0ff7d2
  13. containerd-shim  660604 2682     0 /usr/bin/containerd-shim-runc-v2 -namespace default -address /run/containerd/containerd.sock -publish-binary /usr/bin/containerd -id 3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e start
  14. containerd-shim  660610 660604   0 /usr/bin/containerd-shim-runc-v2 -namespace default -id 3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e -address /run/containerd/containerd.sock
  15. runc             660622 660610   0 /usr/bin/runc --root /run/containerd/runc/default --log /run/containerd/io.containerd.runtime.v2.task/default/3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e/log.json --log-format json --systemd-cgroup create --bundle /run/containerd/io.containerd.runtime.v2.task/default/3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e --pid-file /run/containerd/io.containerd.runtime.v2.task/default/3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e/init.pid --console-socket /tmp/pty147798108/pty.sock 3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e
  16. exe              660629 660622   0 /proc/self/exe init
  17. nerdctl          660640 660622   0 /root/nerdctl --cgroup-manager=systemd internal oci-hook createRuntime
  18. bridge           660647 660640   0 /opt/cni/bin/bridge
  19. host-local       660656 660647   0 /opt/cni/bin/host-local
  20. iptables         660663 660647   0 /usr/sbin/iptables --version
  21. iptables         660665 660647   0 /usr/sbin/iptables -t nat -S --wait
  22. iptables         660666 660647   0 /usr/sbin/iptables -t nat -N CNI-0154f5920acb72bfcf1a9795 --wait
  23. iptables         660667 660647   0 /usr/sbin/iptables -t nat -C CNI-0154f5920acb72bfcf1a9795 -d 10.4.0.8/24 -j ACCEPT -m comment --comment name: "bridge" id: "default-3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e" --wait
  24. iptables         660668 660647   0 /usr/sbin/iptables -t nat -A CNI-0154f5920acb72bfcf1a9795 -d 10.4.0.8/24 -j ACCEPT -m comment --comment name: "bridge" id: "default-3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e" --wait
  25. iptables         660669 660647   0 /usr/sbin/iptables -t nat -C CNI-0154f5920acb72bfcf1a9795 ! -d 224.0.0.0/4 -j MASQUERADE -m comment --comment name: "bridge" id: "default-3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e" --wait
  26. iptables         660670 660647   0 /usr/sbin/iptables -t nat -A CNI-0154f5920acb72bfcf1a9795 ! -d 224.0.0.0/4 -j MASQUERADE -m comment --comment name: "bridge" id: "default-3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e" --wait
  27. iptables         660671 660647   0 /usr/sbin/iptables -t nat -C POSTROUTING -s 10.4.0.8 -j CNI-0154f5920acb72bfcf1a9795 -m comment --comment name: "bridge" id: "default-3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e" --wait
  28. iptables         660672 660647   0 /usr/sbin/iptables -t nat -A POSTROUTING -s 10.4.0.8 -j CNI-0154f5920acb72bfcf1a9795 -m comment --comment name: "bridge" id: "default-3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e" --wait
  29. portmap          660673 660640   0 /opt/cni/bin/portmap
  30. firewall         660678 660640   0 /opt/cni/bin/firewall
  31. iptables         660683 660678   0 /usr/sbin/iptables --version
  32. ip6tables        660684 660678   0 /usr/sbin/ip6tables --version
  33. iptables         660685 660678   0 /usr/sbin/iptables -t filter -S --wait
  34. iptables         660686 660678   0 /usr/sbin/iptables -t filter -S --wait
  35. iptables         660687 660678   0 /usr/sbin/iptables -t filter -C FORWARD -m comment --comment CNI firewall plugin rules -j CNI-FORWARD --wait
  36. iptables         660688 660678   0 /usr/sbin/iptables -t filter -C CNI-FORWARD -m comment --comment CNI firewall plugin admin overrides -j CNI-ADMIN --wait
  37. iptables         660689 660678   0 /usr/sbin/iptables -t filter -C CNI-FORWARD -d 10.4.0.8/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT --wait
  38. iptables         660690 660678   0 /usr/sbin/iptables -t filter -A CNI-FORWARD -d 10.4.0.8/32 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT --wait
  39. iptables         660691 660678   0 /usr/sbin/iptables -t filter -C CNI-FORWARD -s 10.4.0.8/32 -j ACCEPT --wait
  40. iptables         660692 660678   0 /usr/sbin/iptables -t filter -A CNI-FORWARD -s 10.4.0.8/32 -j ACCEPT --wait
  41. iptables         660693 660678   0 /usr/sbin/iptables --version
  42. iptables         660694 660678   0 /usr/sbin/iptables -t filter -S --wait
  43. iptables         660695 660678   0 /usr/sbin/iptables -t filter -S --wait
  44. iptables         660696 660678   0 /usr/sbin/iptables -t filter -C FORWARD -j CNI-ISOLATION-STAGE-1 -m comment --comment CNI firewall plugin rules (ingressPolicy: same-bridge) --wait
  45. iptables         660697 660678   0 /usr/sbin/iptables -t filter -C CNI-ISOLATION-STAGE-1 -i nerdctl0 ! -o nerdctl0 -j CNI-ISOLATION-STAGE-2 -m comment --comment CNI firewall plugin rules (ingressPolicy: same-bridge) --wait
  46. iptables         660698 660678   0 /usr/sbin/iptables -t filter -C CNI-ISOLATION-STAGE-1 -j RETURN -m comment --comment CNI firewall plugin rules (ingressPolicy: same-bridge) --wait
  47. iptables         660699 660678   0 /usr/sbin/iptables -t filter -C CNI-ISOLATION-STAGE-2 -o nerdctl0 -j DROP -m comment --comment CNI firewall plugin rules (ingressPolicy: same-bridge) --wait
  48. iptables         660700 660678   0 /usr/sbin/iptables -t filter -C CNI-ISOLATION-STAGE-2 -j RETURN -m comment --comment CNI firewall plugin rules (ingressPolicy: same-bridge) --wait
  49. tuning           660701 660640   0 /opt/cni/bin/tuning
  50. runc             660707 660610   0 /usr/bin/runc --root /run/containerd/runc/default --log /run/containerd/io.containerd.runtime.v2.task/default/3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e/log.json --log-format json --systemd-cgroup start 3667c3066be8f700d5f1d32d3fa7d3cd54732253539f3cda27159d78cab2cb6e
  51. bash             660633 660610   0 /usr/bin/bash