Linux 版本：5.18.1-1.el7.elrepo.x86_64
Kubelet 版本：”v1.15.11”
docker 版本：18.09.7
查看k8s系统服务
[root@k8s-master ~]# kubectl get pods -n kube-system -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATEScoredns-5d4dd4b4db-9nqqd 0/1 CrashLoopBackOff 13 42m 10.40.0.1 k8s-master <none> <none>coredns-5d4dd4b4db-bb4lz 0/1 CrashLoopBackOff 22 42m 10.46.0.1 k8s-node1 <none> <none>etcd-k8s-master 1/1 Running 410 693d 192.168.131.196 k8s-master <none> <none>kube-apiserver-k8s-master 1/1 Running 367 530d 192.168.131.196 k8s-master <none> <none>kube-controller-manager-k8s-master 1/1 Running 340 693d 192.168.131.196 k8s-master <none> <none>kube-proxy-qhbbx 1/1 Running 3 161m 192.168.131.197 k8s-node1 <none> <none>kube-proxy-rn46x 1/1 Running 34 693d 192.168.131.196 k8s-master <none> <none>kube-proxy-twxs6 1/1 Running 22 692d 192.168.131.198 k8s-node2 <none> <none>kube-scheduler-k8s-master 1/1 Running 346 693d 192.168.131.196 k8s-master <none> <none>weave-net-9h65q 2/2 Running 100 692d 192.168.131.196 k8s-master <none> <none>weave-net-pvhqx 2/2 Running 15 169m 192.168.131.197 k8s-node1 <none> <none>weave-net-xmltr 2/2 Running 66 692d 192.168.131.198 k8s-node2 <none> <none>
异常内容
[root@k8s-master ~]# kubectl describe pod coredns-5d4dd4b4db-bb4lz -n kube-system
Name: coredns-5d4dd4b4db-bb4lz
Namespace: kube-system
Priority: 2000000000
Priority Class Name: system-cluster-critical
Node: k8s-node1/192.168.131.197
Start Time: Tue, 07 Jun 2022 17:49:00 +0800
Labels: k8s-app=kube-dns
pod-template-hash=5d4dd4b4db
Annotations: <none>
Status: Running
IP: 10.46.0.1
Controlled By: ReplicaSet/coredns-5d4dd4b4db
Containers:
coredns:
Container ID: docker://89c710da1dc542bf6b7bdc6d4c3a2ea61688ce232487b4404ebaae90331bcf53
Image: k8s.gcr.io/coredns:1.3.1
Image ID: docker://sha256:eb516548c180f8a6e0235034ccee2428027896af16a509786da13022fe95fe8c
Ports: 53/UDP, 53/TCP, 9153/TCP
Host Ports: 0/UDP, 0/TCP, 0/TCP
Args:
-conf
/etc/coredns/Corefile
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/89c710da1dc542bf6b7bdc6d4c3a2ea61688ce232487b4404ebaae90331bcf53/memory.kmem.limit_in_bytes: operation not supported\"": unknown
Exit Code: 128
Started: Tue, 07 Jun 2022 18:32:06 +0800
Finished: Tue, 07 Jun 2022 18:32:06 +0800
Ready: False
Restart Count: 23
Limits:
memory: 170Mi
Requests:
cpu: 100m
memory: 70Mi
Liveness: http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
Readiness: http-get http://:8080/health delay=0s timeout=1s period=10s #success=1 #failure=3
Environment: <none>
Mounts:
/etc/coredns from config-volume (ro)
/tmp from tmp (rw)
/var/run/secrets/kubernetes.io/serviceaccount from coredns-token-ddmvk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tmp:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: coredns
Optional: false
coredns-token-ddmvk:
Type: Secret (a volume populated by a Secret)
SecretName: coredns-token-ddmvk
Optional: false
QoS Class: Burstable
Node-Selectors: beta.kubernetes.io/os=linux
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 43m default-scheduler Successfully assigned kube-system/coredns-5d4dd4b4db-bb4lz to k8s-node1
Warning FailedCreatePodContainer 43m kubelet, k8s-node1 unable to ensure pod container exists: failed to create container for [kubepods burstable pod2ebe0f2c-06b0-4654-b51b-09c31859fb81] : failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/memory.kmem.limit_in_bytes: operation not supported
Normal Pulled 42m (x4 over 43m) kubelet, k8s-node1 Container image "k8s.gcr.io/coredns:1.3.1" already present on machine
Normal Created 42m (x4 over 43m) kubelet, k8s-node1 Created container coredns
Warning Failed 42m (x4 over 43m) kubelet, k8s-node1 Error: failed to start container "coredns": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/coredns/memory.kmem.limit_in_bytes: operation not supported\"": unknown
Warning BackOff 42m (x9 over 43m) kubelet, k8s-node1 Back-off restarting failed container
Normal Pulled 40m (x4 over 41m) kubelet, k8s-node1 Container image "k8s.gcr.io/coredns:1.3.1" already present on machine
Normal Created 40m (x4 over 41m) kubelet, k8s-node1 Created container coredns
Warning Failed 40m (x4 over 41m) kubelet, k8s-node1 Error: failed to start container "coredns": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/coredns/memory.kmem.limit_in_bytes: operation not supported\"": unknown
Normal SandboxChanged 39m (x3 over 39m) kubelet, k8s-node1 Pod sandbox changed, it will be killed and re-created.
Warning BackOff 21m (x94 over 41m) kubelet, k8s-node1 Back-off restarting failed container
Warning FailedCreatePodContainer 18m kubelet, k8s-node1 unable to ensure pod container exists: failed to create container for [kubepods burstable pod2ebe0f2c-06b0-4654-b51b-09c31859fb81] : failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/memory.kmem.limit_in_bytes: operation not supported
Normal SandboxChanged 17m (x6 over 18m) kubelet, k8s-node1 Pod sandbox changed, it will be killed and re-created.
Normal Pulled 17m (x3 over 17m) kubelet, k8s-node1 Container image "k8s.gcr.io/coredns:1.3.1" already present on machine
Normal Created 17m (x3 over 17m) kubelet, k8s-node1 Created container coredns
Warning Failed 17m (x3 over 17m) kubelet, k8s-node1 Error: failed to start container "coredns": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/coredns/memory.kmem.limit_in_bytes: operation not supported\"": unknown
Warning BackOff 16m (x9 over 17m) kubelet, k8s-node1 Back-off restarting failed container
Warning FailedCreatePodContainer 12m kubelet, k8s-node1 unable to ensure pod container exists: failed to create container for [kubepods burstable pod2ebe0f2c-06b0-4654-b51b-09c31859fb81] : failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/memory.kmem.limit_in_bytes: operation not supported
Normal SandboxChanged 11m (x4 over 12m) kubelet, k8s-node1 Pod sandbox changed, it will be killed and re-created.
Normal Created 10m (x3 over 11m) kubelet, k8s-node1 Created container coredns
Warning Failed 10m (x3 over 11m) kubelet, k8s-node1 Error: failed to start container "coredns": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:297: applying cgroup configuration for process caused \"failed to write 1 to memory.kmem.limit_in_bytes: write /sys/fs/cgroup/memory/kubepods/burstable/pod2ebe0f2c-06b0-4654-b51b-09c31859fb81/coredns/memory.kmem.limit_in_bytes: operation not supported\"": unknown
Normal Pulled 9m57s (x4 over 11m) kubelet, k8s-node1 Container image "k8s.gcr.io/coredns:1.3.1" already present on machine
Warning BackOff 2m16s (x46 over 11m) kubelet, k8s-node1 Back-off restarting failed container
分析与怀疑
可能是由于Linux内核版本过高，与k8s 不兼容
采取降版本的做法。从当前的 5.18.1 降至 5.4.197 再试试。
若有收获，就点个赞吧
0 人点赞
