1. k8s describe 命令不出行 reason 具体内容（event）

k8s1.18 describe 还显示 reason 内容，而 k8s1.20 不显示了，定位问题好难。

1.1 k8s1.20 的describe

  1. [root@saaspy ~]# kubectl get nodes
  2. NAME                         STATUS   ROLES    AGE    VERSION
  3. .....                                          Ready    <none>   161d   v1.20.11-.....
  5. [root@saaspy ~]#
  6. [root@saaspy ~]#
  7. [root@saaspy ~]# kubectl get pod -n noway
  8. NAME                     READY   STATUS             RESTARTS   AGE
  9. noway-587f65cd54-767sk   1/1     Running            1          79d
  10. noway-69f6479878-49g7r   0/1     CrashLoopBackOff   106        22h
  11. [root@saaspy ~]#
  12. [root@saaspy ~]#
  13. [root@saaspy ~]# kubectl describe pod  noway-69f6479878-49g7r -n noway
  14. E0120 11:57:11.465472   32735 describe.go:653] Unable to construct reference to '&v1.Pod{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"noway-69f6479878-49g7r", GenerateName:"noway-69f6479878-", Namespace:"noway", SelfLink:"", UID:"080adb30-73f2-4eb9-9239-813fee9f07b4", ResourceVersion:"63272595", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63778167542, loc:(*time.Location)(0x2d05e60)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"app":"noway", "pod-template-hash":"69f6479878"}, Annotations:map[string]string{"kubernetes.io/psp":"ack.privileged"}, OwnerReferences:[]v1.OwnerReference{v1.OwnerReference{APIVersion:"apps/v1", Kind:"ReplicaSet", Name:"noway-69f6479878", UID:"585ab0e3-2ed7-4363-9a45-55cb052ce7d0", Controller:(*bool)(0xc0003ed6b7), BlockOwnerDeletion:(*bool)(0xc0003ed6b8)}}, Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kube-controller-manager", Operation:"Update", APIVersion:"v1", Time:(*v1.Time)(0xc0006da240), Fields:(*v1.Fields)(nil)}, v1.ManagedFieldsEntry{Manager:"kubelet", Operation:"Update", APIVersion:"v1", Time:(*v1.Time)(0xc0006da260), Fields:(*v1.Fields)(nil)}}}, Spec:v1.PodSpec{Volumes:[]v1.Volume{v1.Volume{Name:"default-token-w4m84", VolumeSource:v1.VolumeSource{HostPath:(*v1.HostPathVolumeSource)(nil), EmptyDir:(*v1.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*v1.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*v1.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*v1.GitRepoVolumeSource)(nil), Secret:(*v1.SecretVolumeSource)(0xc00049b100), NFS:(*v1.NFSVolumeSource)(nil), ISCSI:(*v1.ISCSIVolumeSource)(nil), Glusterfs:(*v1.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*v1.PersistentVolumeClaimVolumeSource)(nil), RBD:(*v1.RBDVolumeSource)(nil), FlexVolume:(*v1.FlexVolumeSource)(nil), Cinder:(*v1.CinderVolumeSource)(nil), CephFS:(*v1.CephFSVolumeSource)(nil), Flocker:(*v1.FlockerVolumeSource)(nil), DownwardAPI:(*v1.DownwardAPIVolumeSource)(nil), FC:(*v1.FCVolumeSource)(nil), AzureFile:(*v1.AzureFileVolumeSource)(nil), ConfigMap:(*v1.ConfigMapVolumeSource)(nil), VsphereVolume:(*v1.VsphereVirtualDiskVolumeSource)(nil), Quobyte:(*v1.QuobyteVolumeSource)(nil), AzureDisk:(*v1.AzureDiskVolumeSource)(nil), PhotonPersistentDisk:(*v1.PhotonPersistentDiskVolumeSource)(nil), Projected:(*v1.ProjectedVolumeSource)(nil), PortworxVolume:(*v1.PortworxVolumeSource)(nil), ScaleIO:(*v1.ScaleIOVolumeSource)(nil), StorageOS:(*v1.StorageOSVolumeSource)(nil), CSI:(*v1.CSIVolumeSource)(nil)}}}, InitContainers:[]v1.Container(nil), Containers:[]v1.Container{v1.Container{Name:"noway", Image:".....", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]v1.ContainerPort{v1.ContainerPort{Name:"", HostPort:0, ContainerPort:8000, Protocol:"TCP", HostIP:""}}, EnvFrom:[]v1.EnvFromSource(nil), Env:[]v1.EnvVar(nil), Resources:v1.ResourceRequirements{Limits:v1.ResourceList(nil), Requests:v1.ResourceList(nil)}, VolumeMounts:[]v1.VolumeMount{v1.VolumeMount{Name:"default-token-w4m84", ReadOnly:true, MountPath:"/var/run/secrets/kubernetes.io/serviceaccount", SubPath:"", MountPropagation:(*v1.MountPropagationMode)(nil), SubPathExpr:""}}, VolumeDevices:[]v1.VolumeDevice(nil), LivenessProbe:(*v1.Probe)(nil), ReadinessProbe:(*v1.Probe)(nil), Lifecycle:(*v1.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*v1.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Always", TerminationGracePeriodSeconds:(*int64)(0xc0003edc00), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"default", DeprecatedServiceAccount:"default", AutomountServiceAccountToken:(*bool)(nil), NodeName:"cn-shanghai.192.168.0.88", HostNetwork:false, HostPID:false, HostIPC:false, ShareProcessNamespace:(*bool)(nil), SecurityContext:(*v1.PodSecurityContext)(0xc00003be60), ImagePullSecrets:[]v1.LocalObjectReference{v1.LocalObjectReference{Name:"secret-name"}}, Hostname:"", Subdomain:"", Affinity:(*v1.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]v1.Toleration{v1.Toleration{Key:"node.kubernetes.io/not-ready", Operator:"Exists", Value:"", Effect:"NoExecute", TolerationSeconds:(*int64)(0xc0003edd60)}, v1.Toleration{Key:"node.kubernetes.io/unreachable", Operator:"Exists", Value:"", Effect:"NoExecute", TolerationSeconds:(*int64)(0xc0003eddd0)}}, HostAliases:[]v1.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(0xc0003ede00), DNSConfig:(*v1.PodDNSConfig)(nil), ReadinessGates:[]v1.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(0xc0003ede04), PreemptionPolicy:(*v1.PreemptionPolicy)(0xc000692700)}, Status:v1.PodStatus{Phase:"Running", Conditions:[]v1.PodCondition{v1.PodCondition{Type:"Initialized", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63778167542, loc:(*time.Location)(0x2d05e60)}}, Reason:"", Message:""}, v1.PodCondition{Type:"Ready", Status:"False", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63778167542, loc:(*time.Location)(0x2d05e60)}}, Reason:"ContainersNotReady", Message:"containers with unready status: [noway]"}, v1.PodCondition{Type:"ContainersReady", Status:"False", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63778167542, loc:(*time.Location)(0x2d05e60)}}, Reason:"ContainersNotReady", Message:"containers with unready status: [noway]"}, v1.PodCondition{Type:"PodScheduled", Status:"True", LastProbeTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:63778167542, loc:(*time.Location)(0x2d05e60)}}, Reason:"", Message:""}}, Message:"", Reason:"", NominatedNodeName:"", HostIP:"192.168.0.88", PodIP:"192.168.0.31", StartTime:(*v1.Time)(0xc0006da280), InitContainerStatuses:[]v1.ContainerStatus(nil), ContainerStatuses:[]v1.ContainerStatus{v1.ContainerStatus{Name:"noway", State:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(0xc0006da2a0), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(nil)}, LastTerminationState:v1.ContainerState{Waiting:(*v1.ContainerStateWaiting)(nil), Running:(*v1.ContainerStateRunning)(nil), Terminated:(*v1.ContainerStateTerminated)(0xc0005bfe30)}, Ready:false, RestartCount:106, Image:".....", ImageID:"docker-pullable://.....", ContainerID:"....."}}, QOSClass:"BestEffort"}}': selfLink was empty, can't make reference
  15. Name:           noway-69f6479878-49g7r
  16. Namespace:      noway
  17. Priority:       0
  18. Node:           .....
  19. Start Time:     Wed, 19 Jan 2022 13:39:02 +0800
  20. Labels:         app=noway
  21.                 pod-template-hash=69f6479878
  22. Annotations:    kubernetes.io/psp: ack.privileged
  23. Status:         Running
  24. IP:             192.168.0.31
  25. Controlled By:  ReplicaSet/noway-69f6479878
  26. Containers:
  27.   noway:
  28.     Container ID:   .....
  29.     Image:          .....
  30.     Image ID:       .....
  31.     Port:           8000/TCP
  32.     Host Port:      0/TCP
  33.     State:          Waiting
  34.       Reason:       CrashLoopBackOff
  35.     Last State:     Terminated
  36.       Reason:       Error
  37.       Exit Code:    1
  38.       Started:      Thu, 20 Jan 2022 11:54:01 +0800
  39.       Finished:     Thu, 20 Jan 2022 11:54:01 +0800
  40.     Ready:          False
  41.     Restart Count:  106
  42.     Environment:    <none>
  43.     Mounts:
  44.       /var/run/secrets/kubernetes.io/serviceaccount from default-token-w4m84 (ro)
  45. Conditions:
  46.   Type              Status
  47.   Initialized       True
  48.   Ready             False
  49.   ContainersReady   False
  50.   PodScheduled      True
  51. Volumes:
  52.   default-token-w4m84:
  53.     Type:        Secret (a volume populated by a Secret)
  54.     SecretName:  default-token-w4m84
  55.     Optional:    false
  56. QoS Class:       BestEffort
  57. Node-Selectors:  <none>
  58. Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  59.                  node.kubernetes.io/unreachable:NoExecute for 300s
  60. [root@saaspy ~]#

1.2 k8s1.18的 describe

  1. [root@saaspy .kube]# kubectl get nodes
  2. NAME     STATUS   ROLES               AGE    VERSION
  3. hb-m2    Ready    controlplane,etcd   449d   v1.18.20
  4. node1    Ready    worker              449d   v1.18.20
  5. node3    Ready    worker              449d   v1.18.20
  6. node4    Ready    worker              373d   v1.18.20
  7. nodes2   Ready    worker              313d   v1.18.10
  8. [root@saaspy .kube]#
  9. [root@saaspy .kube]# kubectl describe -n trial5     pod  trial5-test-svc-app-7bfb987c98-mf6lw
  10. Name:           trial5-test-svc-app-7bfb987c98-mf6lw
  11. Namespace:      trial5
  12. Priority:       0
  13. Node:           node4/192.168.0.214
  14. Start Time:     Mon, 17 Jan 2022 10:03:43 +0800
  15. Labels:         app=trial5-test-svc-app
  16.                 pod-template-hash=7bfb987c98
  17. Annotations:    cattle.io/timestamp: 2021-11-09T09:43:08Z
  18.                 cni.projectcalico.org/podIP: 10.42.4.46/32
  19.                 cni.projectcalico.org/podIPs: 10.42.4.46/32
  20.                 field.cattle.io/ports: [[{"containerPort":9095,"dnsName":"trial5-test-svc-app","kind":"ClusterIP","name":"9095tcp2","protocol":"TCP"}]]
  21. Status:         Running
  22. IP:             10.42.4.46
  23. Controlled By:  ReplicaSet/trial5-test-svc-app-7bfb987c98
  24. Containers:
  25.   trial5-test-svc-app:
  26.     Container ID:   docker://4e943badb3f87df640111a876eba59d3943dd7963dbfeb79920e76429daa80ea
  27.     Image:          192.168.0.210:8080/hb-test-svc/hb-test-svc:test3.3.3
  28.     Image ID:       docker-pullable://192.168.0.210:8080/hb-test-svc/hb-test-svc@sha256:4649cedeaa3c839c8abeb1bf3376d42ff1c3109fea72a6b9ce3bb1c7ad19969d
  29.     Port:           9095/TCP
  30.     Host Port:      0/TCP
  31.     State:          Waiting
  32.       Reason:       CrashLoopBackOff
  33.     Last State:     Terminated
  34.       Reason:       Error
  35.       Exit Code:    1
  36.       Started:      Thu, 20 Jan 2022 12:08:29 +0800
  37.       Finished:     Thu, 20 Jan 2022 12:08:47 +0800
  38.     Ready:          False
  39.     Restart Count:  830
  40.     Environment:
  41.       APOLLO_CLUSTER:  trial5
  42.       APOLLO_META:     http://192.168.0.213:31509
  43.       env:             test
  44.     Mounts:
  45.       /var/run/secrets/kubernetes.io/serviceaccount from default-token-p5fl8 (ro)
  46. Conditions:
  47.   Type              Status
  48.   Initialized       True
  49.   Ready             False
  50.   ContainersReady   False
  51.   PodScheduled      True
  52. Volumes:
  53.   default-token-p5fl8:
  54.     Type:        Secret (a volume populated by a Secret)
  55.     SecretName:  default-token-p5fl8
  56.     Optional:    false
  57. QoS Class:       BestEffort
  58. Node-Selectors:  <none>
  59. Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  60.                  node.kubernetes.io/unreachable:NoExecute for 300s
  61. Events:
  62.   Type     Reason   Age                       From            Message
  63.   ----     ------   ----                      ----            -------
  64.   Warning  BackOff  3m37s (x19395 over 3d2h)  kubelet, node4  Back-off restarting failed container
  65. [root@saaspy .kube]#

通过对比发现，k8s1.20 少了 Events 的描述，刚好reason 信息就在这里。哪位少侠知道为什么呢？
目前只是发现问题，还不知道如何处理。对于除了问题的组件，只好根据状态，初步推断，慢慢分析。

2. k8s 的 annotations

kubernetes.io/hostname