05.pod 健康检查

保持pod健康

使用Kubernetes 的一个主要好处是， 可以给Kubernetes 一个容器列表来由其保持容器在集群中的运行。可以通过让Kubernetes 创建pod 资源， 为其选择一个工作节点并在该节点上运行该pod 的容器来完成此操作。但是，如果其中一个容器终止，或一个pod 的所有容器都终止， 怎么办？

只要将pod 调度到某个节点， 该节点上的Kub e let 就会运行pod 的容器，从此只要该pod 存在， 就会保持运行。如果容器的主进程崩溃， Kubelet 将重启容器。如果应用程序中有一个导致它每隔一段时间就会崩溃的bu g, Kubernetes 会自动重启应用程序，所以即使应用程序本身没有做任何特殊的事， 在Kubernetes 中运行也能自动获得自我修复的能力。

即使进程没有崩溃，有时应用程序也会停止正常工作。例如，具有内存泄漏的Java 应用程序将开始抛出OutOfMemoryErrors ， 但JVM 进程会一直运行。如果有一种方法， 能让应用程序向Kubernetes发出信号， 告诉Kubernetes 它运行异常并让Kubernetes 重新启动，那就很棒了。

例如，你的应用因为无限循环或死锁而停止响应。为确保应用程序在这种情况下可以重新启动，必须从外部检查应用程序的运行状况，而不是依赖于应用的内部检测。

在Kubernetes集群当中，我们可以通过配置liveness probe和readiness probe来影响容器的生存周期。

• 使用 liveness probe 来确定你的应用程序是否正在运行，通俗点将就是是否还活着。一般来说，如果你的程序一旦崩溃了， Kubernetes 就会立刻知道这个程序已经终止了，然后就会重启这个程序。而我们的 liveness probe 的目的就是来捕获到当前应用程序还没有终止，还没有崩溃，如果出现了这些情况，那么就重启处于该状态下的容器，使应用程序在存在 bug 的情况下依然能够继续运行下去。
• 使用 readiness probe 来确定容器是否已经就绪可以接收流量过来了。这个探针通俗点讲就是说是否准备好了，现在可以开始工作了。只有当 Pod 中的容器都处于就绪状态的时候 kubelet 才会认定该 Pod 处于就绪状态，因为一个 Pod 下面可能会有多个容器。当然 Pod 如果处于非就绪状态，那么我们就会将他从我们的工作队列中移除出来，这样我们的流量就不会被路由到这个 Pod 里面来了。

这两个探针支持三种配置方式：

• exec：探针在容器内执行任意命令，并检查命令的退出状态码。如果状态码是0 ，则探测成功。所有其他状态码都被认为失败。
• httpGet：探针对容器的IP 地址（ 指定的端口和路径） 执行HTTP GET 请求。如果探测器收到响应，并且响应状态码不代表错误（换句话说，如果HTT P响应状态码是2xx 或3xx ），则认为探测成功。如果服务器返回错误响应状态码或者根本没有响应，那么探测就被认为是失败的，容器将被重新启动。
• tcpSocket：探针尝试与容器指定端口建立TCP 连接。如果连接成功建立，则探测成功。否则，容器重新启动。

liveness probe

Kubernetes 可以通过存活探针(liveness probe)检查容器是否还在运行。可以为pod 中的每个容器单独指定存活探针。如果探测失败， Kubemetes 将定期执行探针并重新启动容器。

[root@master01 ~]# kubectl explain pod.spec.containers.livenessProbe
KIND:     Pod
VERSION:  v1
RESOURCE: livenessProbe <Object>
DESCRIPTION:
     Periodic probe of container liveness. Container will be restarted if the
     probe fails. Cannot be updated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
     Probe describes a health check to be performed against a container to
     determine whether it is alive or ready to receive traffic.
FIELDS:
   exec    <Object>
     One and only one of the following should be specified. Exec specifies the
     action to take.
   failureThreshold    <integer>
     Minimum consecutive failures for the probe to be considered failed after
     having succeeded. Defaults to 3. Minimum value is 1.
   httpGet    <Object>
     HTTPGet specifies the http request to perform.
   initialDelaySeconds    <integer>
     Number of seconds after the container has started before liveness probes
     are initiated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
   periodSeconds    <integer>
     How often (in seconds) to perform the probe. Default to 10 seconds. Minimum
     value is 1.
   successThreshold    <integer>
     Minimum consecutive successes for the probe to be considered successful
     after having failed. Defaults to 1. Must be 1 for liveness and startup.
     Minimum value is 1.
   tcpSocket    <Object>
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported
   timeoutSeconds    <integer>
     Number of seconds after which the probe times out. Defaults to 1 second.
     Minimum value is 1. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes

探针还可以配置如下几个参数

• initialDelaySeconds: 第一次执行探针的时候要等待时间
• periodSeconds: 每隔多久执行一次存活探针。
• timeoutSeconds: 探测超时时间，默认1秒，最小1秒。
• successThreshold: 探测失败后，最少连续探测成功多少次才被认定为成功。默认是 1，但是如果是liveness则必须是 1，最小值是 1。
• failureThreshold: 探测成功后，最少连续探测失败多少次才被认定为失败。默认是 3，最小值是 1。

exec

[root@master01 ~]# kubectl explain pod.spec.containers.livenessProbe.exec
KIND:     Pod
VERSION:  v1

RESOURCE: exec <Object>

DESCRIPTION:
     One and only one of the following should be specified. Exec specifies the
     action to take.

     ExecAction describes a "run in container" action.

FIELDS:
   command    <[]string>
     Command is the command line to execute inside the container, the working
     directory for the command is root ('/') in the container's filesystem. The
     command is simply exec'd, it is not run inside a shell, so traditional
     shell instructions ('|', etc) won't work. To use a shell, you need to
     explicitly call out to that shell. Exit status of 0 is treated as
     live/healthy and non-zero is unhealthy.

cat > exec-health-hook.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: exec-health-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: busybox
    image: busybox:latest
    command:
    - "/bin/sh"
    - "-c"
    - "touch /tmp/health; sleep 30; rm -rf /tmp/healthy; sleep 600"
    livenessProbe:
      exec:
        command: ["cat /tmp/health"]
      initialDelaySeconds: 5
      periodSeconds: 5
EOF

kubectl apply -f exec-health-hook.yaml

pod运行一个busybox容器，执行命令，创建一个/tmp/health，sleep 30，然后删除/tmp/health，再sleep 600。exec会检查/tmp/health是否存在，如果不在可以看到容器重启了

[root@master01 ~]# kubectl get pod -w
NAME              READY   STATUS              RESTARTS   AGE
exec-health-pod   0/1     ContainerCreating   0          9s
exec-health-pod   1/1     Running             0          16s
exec-health-pod   1/1     Running             1          66s

httpGet

[root@master01 ~]# kubectl explain pod.spec.containers.livenessProbe.httpGet
KIND:     Pod
VERSION:  v1

RESOURCE: httpGet <Object>

DESCRIPTION:
     HTTPGet specifies the http request to perform.

     HTTPGetAction describes an action based on HTTP Get requests.

FIELDS:
   host    <string>
     Host name to connect to, defaults to the pod IP. You probably want to set
     "Host" in httpHeaders instead.

   httpHeaders    <[]Object>
     Custom headers to set in the request. HTTP allows repeated headers.

   path    <string>
     Path to access on the HTTP server.

   port    <string> -required-
     Name or number of the port to access on the container. Number must be in
     the range 1 to 65535. Name must be an IANA_SVC_NAME.

   scheme    <string>
     Scheme to use for connecting to the host. Defaults to HTTP.

cat > httpGet-health-hook.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: http-health-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      httpGet:
        path: /index.html
        port: 80
      initialDelaySeconds: 3
      periodSeconds: 3
EOF

kubectl apply -f httpGet-health-hook.yaml

pod运行一个ikubernetes/myapp容器，httpGet检测/index.html

如果进入容器，删除index.html，容器就会重启

[root@master01 ~]# kubectl exec -it http-health-pod -- /bin/sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls
50x.html    index.html
/usr/share/nginx/html # rm -rf index.html

[root@master01 ~]# kubectl get pod -w
NAME              READY   STATUS    RESTARTS   AGE
http-health-pod   1/1     Running   0          113s
http-health-pod   1/1     Running   1          3m28s

tcpSocket

[root@master01 ~]# kubectl explain pod.spec.containers.livenessProbe.tcpSocket
KIND:     Pod
VERSION:  v1

RESOURCE: tcpSocket <Object>

DESCRIPTION:
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported

     TCPSocketAction describes an action based on opening a socket

FIELDS:
   host    <string>
     Optional: Host name to connect to, defaults to the pod IP.

   port    <string> -required-
     Number or name of the port to access on the container. Number must be in
     the range 1 to 65535. Name must be an IANA_SVC_NAME.

cat > tcpsocket-health-hook.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: tcpsocket-health-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    livenessProbe:
      tcpSocket:
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 5
EOF

进入容器关掉nginx，看是pod否会重启

[root@master01 ~]# kubectl get pod -w
NAME                   READY   STATUS    RESTARTS   AGE
tcpsocket-health-pod   1/1     Running   0          9s
tcpsocket-health-pod   0/1     Completed   0          2m12s
tcpsocket-health-pod   1/1     Running     1          2m13s

readiness probe

readiness probe（就绪探针）来确定容器是否已经就绪可以接受流量。只有当Pod中的容器都处于就绪状态时kubelet才会认定该Pod处于就绪状态。该信号的作用是控制哪些Pod应该作为service的后端。如果Pod处于非就绪状态，那么它们将会被从service的load balancer中移除。

[root@master01 ~]# kubectl explain pod.spec.containers.readinessProbe
KIND:     Pod
VERSION:  v1

RESOURCE: readinessProbe <Object>

DESCRIPTION:
     Periodic probe of container service readiness. Container will be removed
     from service endpoints if the probe fails. Cannot be updated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes

     Probe describes a health check to be performed against a container to
     determine whether it is alive or ready to receive traffic.

FIELDS:
   exec    <Object>
     One and only one of the following should be specified. Exec specifies the
     action to take.

   failureThreshold    <integer>
     Minimum consecutive failures for the probe to be considered failed after
     having succeeded. Defaults to 3. Minimum value is 1.

   httpGet    <Object>
     HTTPGet specifies the http request to perform.

   initialDelaySeconds    <integer>
     Number of seconds after the container has started before liveness probes
     are initiated. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes

   periodSeconds    <integer>
     How often (in seconds) to perform the probe. Default to 10 seconds. Minimum
     value is 1.

   successThreshold    <integer>
     Minimum consecutive successes for the probe to be considered successful
     after having failed. Defaults to 1. Must be 1 for liveness and startup.
     Minimum value is 1.

   tcpSocket    <Object>
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported

   timeoutSeconds    <integer>
     Number of seconds after which the probe times out. Defaults to 1 second.
     Minimum value is 1. More info:
     https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes

exec

[root@master01 ~]# kubectl explain pod.spec.containers.readinessProbe.exec
KIND:     Pod
VERSION:  v1

RESOURCE: exec <Object>

DESCRIPTION:
     One and only one of the following should be specified. Exec specifies the
     action to take.

     ExecAction describes a "run in container" action.

FIELDS:
   command    <[]string>
     Command is the command line to execute inside the container, the working
     directory for the command is root ('/') in the container's filesystem. The
     command is simply exec'd, it is not run inside a shell, so traditional
     shell instructions ('|', etc) won't work. To use a shell, you need to
     explicitly call out to that shell. Exit status of 0 is treated as
     live/healthy and non-zero is unhealthy.

cat > exec-ready-hook.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: exec-ready-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: busybox
    image: busybox:latest
    command:
    - "/bin/sh"
    - "-c"
    - "touch /tmp/health; sleep 30; rm -rf /tmp/healthy; sleep 6000"
    readinessProbe:
      exec:
        command: ["cat /tmp/health"]
      initialDelaySeconds: 5
      periodSeconds: 5
EOF

kubectl apply -f exec-ready-hook.yaml

[root@master01 ~]# kubectl get pod -w
NAME             READY   STATUS    RESTARTS   AGE
exec-ready-pod   0/1     Running   0          49s

[root@master01 ~]# kubectl describe pod exec-ready-pod 
Name:         exec-ready-pod
Namespace:    default
Priority:     0
Node:         node03/192.168.33.203
Start Time:   Thu, 09 Jan 2020 14:49:58 +0800
Labels:       app=myapp
              type=pod
Annotations:  cni.projectcalico.org/podIP: 10.244.186.201/32
              cni.projectcalico.org/podIPs: 10.244.186.201/32
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"myapp","type":"pod"},"name":"exec-ready-pod","namespace":"de...
Status:       Running
IP:           10.244.186.201
IPs:
  IP:  10.244.186.201
Containers:
  busybox:
    Container ID:  docker://838c8c12df55c947a332778323ab3cb245777ccfc1f280d272256551db209443
    Image:         busybox:latest
    Image ID:      docker-pullable://busybox@sha256:6915be4043561d64e0ab0f8f098dc2ac48e077fe23f488ac24b665166898115a
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
      touch /tmp/health; sleep 30; rm -rf /tmp/healthy; sleep 6000
    State:          Running
      Started:      Thu, 09 Jan 2020 14:50:03 +0800
    Ready:          False
    Restart Count:  0
    Readiness:      exec [cat /tmp/health] delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-q49sn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-q49sn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-q49sn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m9s                 default-scheduler  Successfully assigned default/exec-ready-pod to node03
  Normal   Pulling    2m8s                 kubelet, node03    Pulling image "busybox:latest"
  Normal   Pulled     2m4s                 kubelet, node03    Successfully pulled image "busybox:latest"
  Normal   Created    2m4s                 kubelet, node03    Created container busybox
  Normal   Started    2m4s                 kubelet, node03    Started container busybox
  Warning  Unhealthy  18s (x21 over 118s)  kubelet, node03    Readiness probe failed: OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"cat /tmp/health\": stat cat /tmp/health: no such file or directory": unknown

httpGet

[root@master01 ~]# kubectl explain  pod.spec.containers.readinessProbe.httpGet
KIND:     Pod
VERSION:  v1

RESOURCE: httpGet <Object>

DESCRIPTION:
     HTTPGet specifies the http request to perform.

     HTTPGetAction describes an action based on HTTP Get requests.

FIELDS:
   host    <string>
     Host name to connect to, defaults to the pod IP. You probably want to set
     "Host" in httpHeaders instead.

   httpHeaders    <[]Object>
     Custom headers to set in the request. HTTP allows repeated headers.

   path    <string>
     Path to access on the HTTP server.

   port    <string> -required-
     Name or number of the port to access on the container. Number must be in
     the range 1 to 65535. Name must be an IANA_SVC_NAME.

   scheme    <string>
     Scheme to use for connecting to the host. Defaults to HTTP.

cat > http-ready-hook.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: http-ready-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    readinessProbe:
      httpGet:
        path: /index.html
        port: 80
      initialDelaySeconds: 3
      periodSeconds: 3
EOF

kubectl apply -f http-ready-hook.yaml

手动删除index.html文件，查看pod是否就绪

[root@master01 ~]# kubectl exec -it http-ready-pod -- /bin/sh
/ # cd /usr/share/nginx/html/
/usr/share/nginx/html # ls
50x.html    index.html
/usr/share/nginx/html # rm -rf index.html

[root@master01 ~]# kubectl get pod
NAME             READY   STATUS    RESTARTS   AGE
http-ready-pod   1/1     Running   0          8s
[root@master01 ~]# kubectl get pod 
NAME             READY   STATUS    RESTARTS   AGE
http-ready-pod   0/1     Running   0          2m12s

[root@master01 ~]# kubectl describe pod http-ready-pod 
Name:         http-ready-pod
Namespace:    default
Priority:     0
Node:         node01/192.168.33.201
Start Time:   Thu, 09 Jan 2020 14:58:09 +0800
Labels:       app=myapp
              type=pod
Annotations:  cni.projectcalico.org/podIP: 10.244.196.136/32
              cni.projectcalico.org/podIPs: 10.244.196.136/32
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"myapp","type":"pod"},"name":"http-ready-pod","namespace":"de...
Status:       Running
IP:           10.244.196.136
IPs:
  IP:  10.244.196.136
Containers:
  myapp:
    Container ID:   docker://ce71492131909eb013fe813d3b09abb47eb29bae80a10d1b262ab3678b1ee5d8
    Image:          ikubernetes/myapp:v1
    Image ID:       docker-pullable://ikubernetes/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 09 Jan 2020 14:58:11 +0800
    Ready:          False
    Restart Count:  0
    Readiness:      http-get http://:80/index.html delay=3s timeout=1s period=3s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-q49sn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-q49sn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-q49sn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  3m3s                default-scheduler  Successfully assigned default/http-ready-pod to node01
  Normal   Pulled     3m1s                kubelet, node01    Container image "ikubernetes/myapp:v1" already present on machine
  Normal   Created    3m1s                kubelet, node01    Created container myapp
  Normal   Started    3m1s                kubelet, node01    Started container myapp
  Warning  Unhealthy  12s (x22 over 75s)  kubelet, node01    Readiness probe failed: HTTP probe failed with statuscode: 404

tcpSocket

[root@master01 ~]# kubectl explain  pod.spec.containers.readinessProbe.tcpSocket
KIND:     Pod
VERSION:  v1

RESOURCE: tcpSocket <Object>

DESCRIPTION:
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported

     TCPSocketAction describes an action based on opening a socket

FIELDS:
   host    <string>
     Optional: Host name to connect to, defaults to the pod IP.

   port    <string> -required-
     Number or name of the port to access on the container. Number must be in
     the range 1 to 65535. Name must be an IANA_SVC_NAME.

cat > tcpsocket-ready-hook.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: tcpsocket-ready-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    readinessProbe:
      tcpSocket:
        port: 80
      initialDelaySeconds: 5
      periodSeconds: 5
EOF

kubectl apply -f tcpsocket-ready-hook.yaml

修改端口

[root@master01 ~]# kubectl exec -it tcpsocket-ready-pod  -- /bin/sh
/ # sed -i s/80/8080/  /etc/nginx/conf.d/default.conf 
/ # nginx -s reload
2020/01/09 07:23:57 [notice] 51#51: signal process started
/ # netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      1/nginx: master pro

[root@master01 ~]# kubectl get pod
NAME                  READY   STATUS    RESTARTS   AGE
tcpsocket-ready-pod   0/1     Running   0          20m

[root@master01 ~]# kubectl describe pod tcpsocket-ready-pod 
Name:         tcpsocket-ready-pod
Namespace:    default
Priority:     0
Node:         node01/192.168.33.201
Start Time:   Thu, 09 Jan 2020 15:03:50 +0800
Labels:       app=myapp
              type=pod
Annotations:  cni.projectcalico.org/podIP: 10.244.196.137/32
              cni.projectcalico.org/podIPs: 10.244.196.137/32
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"myapp","type":"pod"},"name":"tcpsocket-ready-pod","namespace...
Status:       Running
IP:           10.244.196.137
IPs:
  IP:  10.244.196.137
Containers:
  myapp:
    Container ID:   docker://125701341cf1f1dc3ff6ad1c98f13220edced75d57b314986fe22f08f808efd8
    Image:          ikubernetes/myapp:v1
    Image ID:       docker-pullable://ikubernetes/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 09 Jan 2020 15:03:52 +0800
    Ready:          False
    Restart Count:  0
    Readiness:      tcp-socket :80 delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-q49sn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  default-token-q49sn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-q49sn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  21m                default-scheduler  Successfully assigned default/tcpsocket-ready-pod to node01
  Normal   Pulled     21m                kubelet, node01    Container image "ikubernetes/myapp:v1" already present on machine
  Normal   Created    21m                kubelet, node01    Created container myapp
  Normal   Started    21m                kubelet, node01    Started container myapp
  Warning  Unhealthy  3s (x14 over 68s)  kubelet, node01    Readiness probe failed: dial tcp 10.244.196.137:80: connect: connection refused

pod hook

Kubernetes 为我们的容器提供了生命周期钩子，就是我们说的Pod Hook，Pod Hook 是由 kubelet 发起的，当容器中的进程启动前或者容器中的进程终止之前运行，这是包含在容器的生命周期之中。我们可以同时为 Pod 中的所有容器都配置 hook。

Kubernetes 为我们提供了两种钩子函数：

• PostStart：这个钩子在容器创建后立即执行。但是，并不能保证钩子将在容器ENTRYPOINT之前运行，因为没有参数传递给处理程序。主要用于资源部署、环境准备等。不过需要注意的是如果钩子花费太长时间以至于不能运行或者挂起， 容器将不能达到running状态。
• PreStop：这个钩子在容器终止之前立即被调用。它是阻塞的，意味着它是同步的， 所以它必须在删除容器的调用发出之前完成。主要用于优雅关闭应用程序、通知其他系统等。如果钩子在执行期间挂起， Pod阶段将停留在running状态并且永不会达到failed状态。

如果PostStart或者PreStop钩子失败， 它也会杀死容器。所以我们应该让钩子函数尽可能的轻量。当然有些情况下，长时间运行命令是合理的， 比如在停止容器之前预先保存状态。

[root@master01 ~]# kubectl explain pod.spec.containers.lifecycle
KIND:     Pod
VERSION:  v1

RESOURCE: lifecycle <Object>

DESCRIPTION:
     Actions that the management system should take in response to container
     lifecycle events. Cannot be updated.

     Lifecycle describes actions that the management system should take in
     response to container lifecycle events. For the PostStart and PreStop
     lifecycle handlers, management of the container blocks until the action is
     complete, unless the container process fails, in which case the handler is
     aborted.

FIELDS:
   postStart    <Object>
     PostStart is called immediately after a container is created. If the
     handler fails, the container is terminated and restarted according to its
     restart policy. Other management of the container blocks until the hook
     completes. More info:
     https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks

   preStop    <Object>
     PreStop is called immediately before a container is terminated due to an
     API request or management event such as liveness/startup probe failure,
     preemption, resource contention, etc. The handler is not called if the
     container crashes or exits. The reason for termination is passed to the
     handler. The Pod's termination grace period countdown begins before the
     PreStop hooked is executed. Regardless of the outcome of the handler, the
     container will eventually terminate within the Pod's termination grace
     period. Other management of the container blocks until the hook completes
     or until the termination grace period is reached. More info:
     https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks

postStart

postStart：这个钩子在容器创建后立即执行

[root@master01 ~]# kubectl explain pod.spec.containers.lifecycle.postStart
KIND:     Pod
VERSION:  v1

RESOURCE: postStart <Object>

DESCRIPTION:
     PostStart is called immediately after a container is created. If the
     handler fails, the container is terminated and restarted according to its
     restart policy. Other management of the container blocks until the hook
     completes. More info:
     https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks

     Handler defines a specific action that should be taken

FIELDS:
   exec    <Object>
     One and only one of the following should be specified. Exec specifies the
     action to take.

   httpGet    <Object>
     HTTPGet specifies the http request to perform.

   tcpSocket    <Object>
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported

cat > poststart-hook-pod.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: poststart-hook-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "echo Hello from the postStart handler > /usr/share/message"]
EOF

kubectl apply -f poststart-hook-pod.yaml

进入容器查看poststart hook是否生效

[root@master01 ~]# kubectl get pod
NAME                 READY   STATUS    RESTARTS   AGE
poststart-hook-pod   1/1     Running   0          63s
[root@master01 ~]# kubectl exec -it poststart-hook-pod -- /bin/sh
/ # cat /usr/share/message 
Hello from the postStart handler

preStop

当用户请求删除含有 pod 的资源对象时，K8S 为了让应用程序优雅关闭（即让应用程序完成正在处理的请求后，再关闭软件），K8S提供两种信息通知：

默认：K8S 通知 node 执行docker stop命令，docker 会先向容器中PID为1的进程发送系统信号SIGTERM，然后等待容器中的应用程序终止执行，如果等待时间达到设定的超时时间，或者默认超时时间（30s），会继续发送SIGKILL的系统信号强行 kill 掉进程。
使用 pod 生命周期（利用PreStop回调函数），它执行在发送终止信号之前。
默认所有的优雅退出时间都在30秒内。kubectl delete 命令支持 —grace-period=选项，这个选项允许用户用他们自己指定的值覆盖默认值。值’0’代表 强制删除 pod. 在 kubectl 1.5 及以上的版本里，执行强制删除时必须同时指定 —force —grace-period=0。

强制删除一个 pod 是从集群状态还有 etcd 里立刻删除这个 pod。 当 Pod 被强制删除时， api 服务器不会等待来自 Pod 所在节点上的 kubelet 的确认信息：pod 已经被终止。在 API 里 pod 会被立刻删除，在节点上， pods 被设置成立刻终止后，在强行杀掉前还会有一个很小的宽限期。

[root@master01 ~]# kubectl explain pod.spec.containers.lifecycle.preStop
KIND:     Pod
VERSION:  v1

RESOURCE: preStop <Object>

DESCRIPTION:
     PreStop is called immediately before a container is terminated due to an
     API request or management event such as liveness/startup probe failure,
     preemption, resource contention, etc. The handler is not called if the
     container crashes or exits. The reason for termination is passed to the
     handler. The Pod's termination grace period countdown begins before the
     PreStop hooked is executed. Regardless of the outcome of the handler, the
     container will eventually terminate within the Pod's termination grace
     period. Other management of the container blocks until the hook completes
     or until the termination grace period is reached. More info:
     https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks

     Handler defines a specific action that should be taken

FIELDS:
   exec    <Object>
     One and only one of the following should be specified. Exec specifies the
     action to take.

   httpGet    <Object>
     HTTPGet specifies the http request to perform.

   tcpSocket    <Object>
     TCPSocket specifies an action involving a TCP port. TCP hooks not yet
     supported

cat > prestop-hook-pod.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
  name: prestop-hook-pod
  namespace: default
  labels:
    app: myapp
    type: pod
spec:
  containers:
  - name: myapp
    image: ikubernetes/myapp:v1
    ports:
    - name: http
      containerPort: 80
    volumeMounts:
    - name: message
      mountPath: /data/
    lifecycle:
      preStop:
        exec:
          command: ['/bin/sh', '-c', 'echo Hello from the preStop Handler > /data/message']
  volumes:
  - name: message
    hostPath:
      path: /tmp
EOF

kubectl apply -f prestop-hook-pod.yaml

[root@master01 ~]# kubectl get pod -o wide
NAME               READY   STATUS    RESTARTS   AGE   IP               NODE     NOMINATED NODE   READINESS GATES
prestop-hook-pod   1/1     Running   0          21s   10.244.186.203   node03   <none>           <none>
[root@master01 ~]# kubectl delete pod prestop-hook-pod 
pod "prestop-hook-pod" deleted

删掉pod 查看prestop hook是否生效,因为我的prestop-hook-pod在node03上，所以去node03上查看

[root@node03 ~]# cat /tmp/message 
Hello from the preStop Handler