03-3-Pod控制器

1. Pod

1.1. Pod介绍

1.1.1. Pod简介

Pod 是 Kubernetes 的基本构建块，它是 Kubernetes 对象模型中创建或部署的最小和最简单的单元。 Pod 表示集群上正在运行的进程。Pod 封装了应用程序容器（或者在某些情况下封装多个容器）、存储资源、唯一网络 IP 以及控制容器应该如何运行的选项。 Pod 表示部署单元：Kubernetes 中应用程序的单个实例，它可能由单个容器或少量紧密耦合并共享资源的容器组成。

一个pod内部一般仅运行一个pod，也可以运行多个pod，如果存在多个pod时，其中一个为主容器，其它作为辅助容器，也被称为边车模式。同一个pod共享一个网络名称空间和外部存储卷。

03-8-K8S网络 - 图1

1.1.2. Pod生命周期

Pod的生命周期中可以经历多个阶段，在一个Pod中在主容器(Main Container)启动前可以由init container来完成一些初始化操作。初始化完毕后，init Container 退出，Main Container启动。

在主容器启动后可以执行一些特定的指令，称为启动后钩子(PostStart)，在主容器退出前也可以执行一些特殊指令完成清理工作，称为结束前钩子(PreStop)。

在主容器工作周期内，并不是刚创建就能对外提供服务，容器内部可能需要加载相关配置，因此可以使用特定命令确定容器是否就绪，称为就绪性检测(ReadinessProbe)，完成就绪性检测才能成为Ready状态。

主容器对外提供服务后，可能出现意外导致容器异常，虽然此时容器仍在运行，但是不具备对外提供业务的能力，因此需要对其做存活性探测(LivenessProbe)。

03-8-K8S网络 - 图2

1.1.3. Pod状态

  • Pending: Pod 已被 Kubernetes 系统接受，但有一个或者多个容器尚未创建。
  • Running: 该 Pod 已经绑定到了一个节点上，Pod 中所有的容器都已被创建。至少有一个容器正在运行，或者正处于启动或重启状态。
  • Succeeded: Pod 中的所有容器都被成功终止，并且不会再重启。
  • Failed: Pod 中的所有容器都已终止了，并且至少有一个容器是因为失败终止。
  • Unknown: 因为某些原因无法取得 Pod 的状态，通常是因为与 Pod 所在主机通信失败。

03-8-K8S网络 - 图3

1.2. Pod模板

1.2.1. apiversion/kind

  1. apiVersion: v1
  2. kind: Pod

1.2.2. metadata

  1. metadata
  2.     name        <string>            # 在一个名称空间内不能重复
  3.     namespace   <string>            # 指定名称空间，默认defalut
  4.     labels      <map[string]string> # 标签
  5.     annotations <map[string]string> # 注释，不能作为被筛选

1.2.3. spec

  1. spec
  2.   containers                <[]Object>  -required-      # 必选参数
  3.     name                    <string>    -required-      # 指定容器名称，不可更新
  4.     image                   <string>    -required-      # 指定镜像
  5.     imagePullPolicy         <string>                    # 指定镜像拉取方式
  6.       # Always: 始终从registory拉取镜像。如果镜像标签为latest，则默认值为Always
  7.       # Never: 仅使用本地镜像
  8.       # IfNotPresent: 本地不存在镜像时才去registory拉取。默认值
  9.     env                     <[]Object>                  # 指定环境变量
  10.       name                  <string>    -required-      # 变量名称
  11.       value                 <string>                    # 变量值
  12.       valueFrom             <Object>                    # 从文件中读取，不常用
  13.     command                 <[]string>                  # 以数组方式指定容器运行指令，替代docker的ENTRYPOINT指令
  14.     args                    <[]string>                  # 以数组方式指定容器运行参数，替代docker的CMD指令
  15.     workingDir              <string>                    # 指定工作目录，不指定则使用镜像默认值
  16.     ports                   <[]Object>                  # 指定容器暴露的端口
  17.       containerPort         <integer>   -required-      # 容器的监听端口
  18.       name                  <string>                    # 为端口取名，该名称可以在service种被引用
  19.       protocol              <string>                    # 指定协议:UDP, TCP, SCTP，默认TCP
  20.       hostIP                <string>                    # 绑定到宿主机的某个IP
  21.       hostPort              <integer>                   # 绑定到宿主机的端口
  22.     resources               <Object>                    # 资源设置
  23.       limits                <map[string]string>         # 消耗的最大资源限制，通常设置cpu和memory
  24.       requests              <map[string]string>         # 最低资源要求，在scheduler中被用到，通常设置cpu和memory
  25.     volumeMounts            <[]Object>                  # 指定存储卷挂载
  26.       name                  <string>    -required-      # 存储卷名称
  27.       mountPath             <string>    -required-      # 容器内挂载路径
  28.       subPath               <string>                    # 存储卷的子目录
  29.       readOnly              <boolean>                   # 是否为只读方式挂载
  30.     volumeDevices           <[]Object>                  # 配置块设备的挂载
  31.       devicePath            <string>    -required-      # 容器内挂载路径
  32.       name                  <string>    -required-      # pvc名称
  33.     readinessProbe          <Object>                    # 就绪性探测，确认就绪后提供服务
  34.       initialDelaySeconds   <integer>                   # 容器启动后到开始就绪性探测中间的等待秒数
  35.       periodSeconds         <integer>                   # 两次探测的间隔多少秒，默认值为10
  36.       successThreshold      <integer>                   # 连续多少次检测成功认为容器正常，默认值为1。不支持修改
  37.       failureThreshold      <integer>                   # 连续多少次检测失败认为容器异常，默认值为3
  38.       timeoutSeconds        <integer>                   # 探测请求超时时间
  39.       exec                  <Object>                    # 通过执行特定命令来探测容器健康状态
  40.         command             <[]string>                  # 执行命令，返回值为0表示健康，不自持shell模式
  41.       tcpSocket             <Object>                    # 检测TCP套接字
  42.         host                <string>                    # 指定检测地址，默认pod的IP
  43.         port                <string>  -required-        # 指定检测端口
  44.       httpGet               <Object>                    # 以HTTP请求方式检测
  45.         host                <string>                    # 指定检测地址，默认pod的IP
  46.         httpHeaders         <[]Object>                  # 设置请求头，很少会需要填写
  47.         path                <string>                    # 设置请求的location
  48.         port                <string>  -required-        # 指定检测端口
  49.         scheme              <string>                    # 指定协议，默认HTTP
  50.     livenessProbe           <Object>                    # 存活性探测，确认pod是否具备对外服务的能力,该对象中字段和readinessProbe一致
  51.     lifecycle               <Object>                    # 生命周期
  52.       postStart             <Object>                    # pod启动后钩子，执行指令或者检测失败则退出容器或者重启容器
  53.         exec                <Object>                    # 执行指令，参考readinessProbe.exec
  54.         httpGet             <Object>                    # 执行HTTP，参考readinessProbe.httpGet
  55.         tcpSocket           <Object>                    # 检测TCP套接字，参考readinessProbe.tcpSocket
  56.       preStop               <Object>                    # pod停止前钩子，停止前执行清理工作,该对象中字段和postStart一致
  57.     startupProbe            <Object>                    # 容器启动完毕的配置，该配置与readinessProbe一致，在lifecycle和Probe之前运行，失败则重启
  58.     securityContext         <Object>                    # 与容器安全相关的配置，如运行用户、特权模式等
  59.   initContainers            <[]Object>                  # 初始化容器，执行完毕会退出，用户数据迁移、文件拷贝等
  60.   volumes                   <[]Object>                  # 存储卷配置,https://www.yuque.com/duduniao/k8s/vgms23#Ptdfs
  61.   restartPolicy             <string>                    # Pod重启策略，Always, OnFailure,Never，默认Always
  62.   nodeName                  <string>                    # 调度到指定的node节点, 强制要求满足
  63.   nodeSelector              <map[string]string>         # 指定预选的node节点, 强制要求满足
  64.   affinity                  <Object>                    # 调度亲和性配置
  65.     nodeAffinity            <Object>                    # node亲和性配置
  66.       preferredDuringSchedulingIgnoredDuringExecution   <[]Object>  # 首选配置
  67.         preference          <Object>    -required-      # 亲和偏好
  68.           matchExpressions  <[]Object>                  # 表达式匹配
  69.             key             <string>    -required-      # label的key
  70.             values          <[]string>                  # label的value,当operator为Exists和DoesNotExist时为空
  71.             operator        <string>    -required-      # key和value的连接符，In,NotIn,Exists,DoesNotExist,Gt,Lt
  72.           matchFields       <[]Object>                  # 字段匹配,与matchExpressions一致
  73.         weight              <integer>   -required-      # 权重
  74.       requiredDuringSchedulingIgnoredDuringExecution      <Object>    # 强制要求的配置
  75.         nodeSelectorTerms   <[]Object>  -required-      # nodeselect配置，与preferredDuringSchedulingIgnoredDuringExecution.preference一致
  76.     podAffinity             <Object>                    # pod亲和性配置
  77.       preferredDuringSchedulingIgnoredDuringExecution   <[]Object>  # 首选配置
  78.         podAffinityTerm     <Object>    -required-      # 选择器
  79.           labelSelector     <Object>                    # pod标签选择器
  80.             matchExpressions<[]Object>                  # 表达式匹配
  81.             matchLabels     <map[string]string>         # 标签匹配
  82.           namespaces        <[]string>                  # 对方Pod的namespace,为空时表示与当前Pod同一名称空间
  83.           topologyKey       <string>    -required-      # 与对方Pod亲和的Node上具备的label名称
  84.         weight              <integer>   -required-      # 权重
  85.       requiredDuringSchedulingIgnoredDuringExecution    <[]Object>  # 强制配置，与requiredDuringSchedulingIgnoredDuringExecution.podAffinityTerm一致
  86.     podAntiAffinity         <Object>                    # Pod反亲和性配置,与podAffinity一致
  87.   tolerations               <[]Object>                  # 污点容忍配置
  88.     key                     <string>                    # 污点的Key，为空表示所有污点的Key
  89.     operator                <string>                    # key和value之间的操作符，Exists,Equal。Exists时value为空，默认值 Equal
  90.     value                   <string>                    # 污点的值
  91.     effect                  <string>                    # 污点的影响行为，空表示容忍所有的行为
  92.     tolerationSeconds       <integer>                   # 当Pod被节点驱逐时，延迟多少秒
  93.   hostname                  <string>                    # 指定pod主机名
  94.   hostIPC                   <boolean>                   # 使用宿主机的IPC名称空间，默认false
  95.   hostNetwork               <boolean>                   # 使用宿主机的网络名称空间，默认false
  96.   hostPID                   <boolean>                   # 使用宿主机的PID名称空间，默认false
  97.   serviceAccountName        <string>                    # Pod运行时的使用的serviceAccount
  98.   imagePullSecrets          <[]Object>                  # 当拉取私密仓库镜像时，需要指定的密码密钥信息
  99.     name                    <string>                    # secrets 对象名

1.2.4. k8s和image中的命令

03-8-K8S网络 - 图4

1.2.4. 就绪性探测和存活性探测

  • 就绪性探测失败不会重启pod，只是让pod不处于ready状态。存活性探测失败会触发pod重启。
  • 就绪性探测和存活性探测会持续进行下去，直到pod终止。

1.3. 案例

一般不会单独创建pod，而是通过控制器的方式创建。

1.3.1. 创建简单pod

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: pod-demo
  5.   namespace: app
  6.   labels:
  7.     app: centos7
  8.     release: stable
  9.     environment: dev
  10. spec:
  11.   containers:
  12.   - name: centos
  13.     image: harbor.od.com/public/centos:7
  14.     command:
  15.     - /bin/bash
  16.     - -c
  17.     - "sleep 3600"
  1. [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/base_resource/pods/myapp.yaml
  2. [root@hdss7-21 ~]# kubectl get pod -o wide -n app
  3. NAME       READY   STATUS    RESTARTS   AGE   IP           NODE                NOMINATED NODE   READINESS GATES
  4. pod-demo   1/1     Running   0          16s   172.7.22.2   hdss7-22.host.com   <none>           <none>
  5. [root@hdss7-21 ~]# kubectl exec pod-demo -n app -- ps uax
  6. USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
  7. root          1  0.0  0.0   4364   352 ?        Ss   04:41   0:00 sleep 3600
  8. root         11  0.0  0.0  51752  1696 ?        Rs   04:42   0:00 ps uax
  1. [root@hdss7-21 ~]# kubectl describe pod pod-demo -n app | tail
  2. Events:
  3.   Type    Reason     Age    From                        Message
  4.   ----    ------     ----   ----                        -------
  5.   Normal  Scheduled  3m46s  default-scheduler           Successfully assigned app/pod-demo to hdss7-22.host.com
  6.   Normal  Pulling    3m45s  kubelet, hdss7-22.host.com  Pulling image "harbor.od.com/public/centos:7"
  7.   Normal  Pulled     3m45s  kubelet, hdss7-22.host.com  Successfully pulled image "harbor.od.com/public/centos:7"
  8.   Normal  Created    3m45s  kubelet, hdss7-22.host.com  Created container centos
  9.   Normal  Started    3m45s  kubelet, hdss7-22.host.com  Started container centos

1.3.2. 带健康检测的pod

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4.   name: pod-01
  5.   namespace: app
  6.   labels:
  7.     app: centos7
  8.     release: stable
  9.     version: t1
  10. spec:
  11.   containers:
  12.   - name: centos
  13.     image: harbor.od.com/public/centos:7
  14.     command:
  15.     - /bin/bash
  16.     - -c
  17.     - "echo 'abc' > /tmp/health;sleep 60;rm -f /tmp/health;sleep 600"
  18.     livenessProbe:
  19.       exec:
  20.         command:
  21.         - /bin/bash
  22.         - -c
  23.         - "[ -f /tmp/health ]"

2. Deployment

2.1. 介绍

2.1.1. 简介

Pod控制器有很多种，最初的是使用 ReplicationController，即副本控制器，用于控制pod数量。随着版本升级，出现了ReplicaSet，跟ReplicationController没有本质的不同，只是名字不一样，并且ReplicaSet支持集合式的selector。ReplicaSet的核心管理对象有三种：用户期望的副本数、标签选择器、pod模板。

ReplicaSet一般不会直接使用，而是采用Deployment，Deployment是用来管理Replicaset，ReplicaSet来管理Pod。Deployment为ReplicaSet 提供了一个声明式定义(declarative)方法，用来替代以前的 ReplicationController 来方便的管理应用，比ReplicaSet的功能更加强大，且包含了ReplicaSet的功能。Deployment支持以下功能：

  • 定义Deployment来创建Pod和ReplicaSet
  • 滚动升级和回滚应用
  • 扩容和缩容
  • 暂停部署功能和手动部署

2.1.2. 部署方式

  • 蓝绿发布

如图，假设副本数是5，目标是从v1升级到v2。先部署5个v2版本的业务机器，再将SLB的流量全部切换到v2上。如果出现异常，可以快速切换到v1版本。但是实际上用的不多，因为需要消耗大量的额外机器资源。

03-8-K8S网络 - 图5

  • 滚动发布

滚动发布是逐台(批次)升级，需要占用的额外资源少。比如先升级一台，再升级一台，直到全部升级完毕。也可以每次升级10%数量的机器，逐批次升级。

03-8-K8S网络 - 图6

  • 灰度发布(金丝雀发布)

灰度发布也叫金丝雀发布，起源是，矿井工人发现，金丝雀对瓦斯气体很敏感，矿工会在下井之前，先放一只金丝雀到井中，如果金丝雀不叫了，就代表瓦斯浓度高。

灰度发布会先升级一台灰度机器，将版本升级为v2，此时先经过测试验证，确认没有问题后。从LB引入少量流量进入灰度机器，运行一段时间后，再将其它机器升级为v2版本，引入全部流量。

03-8-K8S网络 - 图7

2.1.3. Deployment升级方案

Deployment的升级方案默认是滚动升级，支持升级暂停，支持指定最大超过预期pod数量，支持指定最小低于预期pod数量。可以实现上述三种部署方案(以目标预期pod数量5个，v1版本升级到v2版本为案例)：

  • 蓝绿发布场景实现方案：新创建5个v2版本pod，等待5个v2版本Pod就绪后，下掉5个v1版本pod。
  • 灰度发布场景实现案例：新创建的第一个pod最为灰度pod，此时暂定升级，等待灰度成功后再升级v1版本Pod
  • 滚动发布：通过控制超出预期pod数量和低于预期Pod数量来控制滚动发布的节奏。

如下图，预期pod数量5个，滚动升级，最大超出副本数为2个，最大低于期望值2个的升级方式：

03-8-K8S网络 - 图8

2.2. 模板

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata
  4.     name        <string>            # 在一个名称空间不能重复
  5.     namespace   <string>            # 指定名称空间，默认defalut
  6.     labels      <map[string]string> # 标签
  7.     annotations <map[string]string> # 注释
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata
  4.     name        <string>                            # 在一个名称空间不能重复
  5.     namespace   <string>                            # 指定名称空间，默认defalut
  6.     labels      <map[string]string>                 # 标签
  7.     annotations <map[string]string>                 # 注释
  9. spec
  10.     replicas                    <integer>           # 期望副本数，默认值1
  11.     selector                    <Object>            # 标签选择器
  12.         matchExpressions        <[]Object>          # 标签选择器的一种形式,多个条件使用AND连接
  13.             key                 <string> -required- # 标签中的Key
  14.             operator            <string> -required- # 操作符，支持 In, NotIn, Exists, DoesNotExist
  15.             values              <[]string>          # value的数组集合，当操作符为In或NotIn时不能为空
  16.         matchLabels             <map[string]string> # 使用key/value的格式做筛选
  17.     strategy                    <Object>            # pod更新策略，即如何替换已有的pod
  18.         type                    <string>            # 更新类型，支持 Recreate, RollingUpdate。默认RollingUpdate
  19.         rollingUpdate           <Object>            # 滚动更新策略，仅在type为RollingUpdate时使用
  20.             maxSurge            <string>            # 最大浪涌pod数，即滚动更新时最多可多于出期望值几个pod。支持数字和百分比格式
  21.             maxUnavailable      <string>            # 最大缺失Pod数，即滚动更新时最多可少于期望值出几个pod。支持数字和百分比格式
  22.     revisionHistoryLimit        <integer>           # 历史版本记录数，默认为最大值(2^32)
  23.     template                    <Object> -required- # Pod模板，和Pod管理器yaml几乎格式一致
  24.         metadata                <Object>            # Pod的metadata
  25.         spec                    <Object>            # Pod的spec

2.3. 案例

2.3.1. 创建deployment

  1. [root@hdss7-200 deployment]# vim /data/k8s-yaml/base_resource/deployment/nginx-v1.12.yaml
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: nginx-deploy
  6.   namespace: app
  7. spec:
  8.   replicas: 5
  9.   selector:
  10.     matchLabels:
  11.       app: nginx
  12.       release: stable
  13.       tier: slb
  14.       partition: website
  15.   strategy:
  16.     rollingUpdate:
  17.       maxSurge: 1
  18.       maxUnavailable: 0
  19.   template:
  20.     metadata:
  21.       labels:
  22.         app: nginx
  23.         release: stable
  24.         tier: slb
  25.         partition: website
  26.         version: v1.12
  27.     spec:
  28.       containers:
  29.       - name: nginx-pod
  30.         image: harbor.od.com/public/nginx:v1.12
  31.         lifecycle:
  32.           postStart:
  33.             exec:
  34.               command:
  35.                 - /bin/bash
  36.                 - -c
  37.                 - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  38.         readinessProbe: 
  39.           initialDelaySeconds: 5
  40.           httpGet:
  41.             port: 80
  42.             path: /health.html
  43.         livenessProbe:
  44.           initialDelaySeconds: 10
  45.           periodSeconds: 5
  46.           httpGet:
  47.             port: 80
  48.             path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record
  3. [root@hdss7-21 ~]# kubectl get pods -n app -l partition=website  # 查看
  4. NAME                           READY   STATUS    RESTARTS   AGE
  5. nginx-deploy-5597c8b45-425ms   1/1     Running   0          5m12s
  6. nginx-deploy-5597c8b45-5p2rz   1/1     Running   0          9m34s
  7. nginx-deploy-5597c8b45-dw7hd   1/1     Running   0          9m34s
  8. nginx-deploy-5597c8b45-fg82k   1/1     Running   0          5m12s
  9. nginx-deploy-5597c8b45-sfxmg   1/1     Running   0          9m34s
  10. [root@hdss7-21 ~]# kubectl get rs -n app -l partition=website -o wide
  11. NAME                     DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES                             SELECTOR
  12. nginx-deploy-5597c8b45   8         8         8       10m   nginx-pod    harbor.od.com/public/nginx:v1.12   app=nginx,partition=website,pod-template-hash=5597c8b45,release=stable,tier=slb
  13. [root@hdss7-21 ~]# kubectl get deployment -n app -o wide
  14. NAME           READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES                             SELECTOR
  15. nginx-deploy   8/8     8            8           11m   nginx-pod    harbor.od.com/public/nginx:v1.12   app=nginx,partition=website,release=stable,tier=slb

2.3.2. 模拟蓝绿发布

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5.   namespace: app
  6. spec:
  7.   replicas: 5
  8.   selector:
  9.     matchLabels:
  10.       app: nginx
  11.       release: stable
  12.       tier: slb
  13.       partition: website
  14.   strategy:
  15.     rollingUpdate:
  16.       # 最大浪涌数量为5
  17.       maxSurge: 5
  18.       maxUnavailable: 0
  19.   template:
  20.     metadata:
  21.       labels:
  22.         app: nginx
  23.         release: stable
  24.         tier: slb
  25.         partition: website
  26.         # 修改版本信息，用于查看当前版本
  27.         version: v1.13
  28.     spec:
  29.       containers:
  30.       - name: nginx-pod
  31.         # 修改镜像
  32.         image: harbor.od.com/public/nginx:v1.13
  33.         lifecycle:
  34.           postStart:
  35.             exec:
  36.               command:
  37.                 - /bin/bash
  38.                 - -c
  39.                 - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  40.         readinessProbe: 
  41.           initialDelaySeconds: 5
  42.           httpGet:
  43.             port: 80
  44.             path: /health.html
  45.         livenessProbe:
  46.           initialDelaySeconds: 10
  47.           periodSeconds: 5
  48.           httpGet:
  49.             port: 80
  50.             path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  2. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app
  3. REVISION  CHANGE-CAUSE
  4. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  5. 2         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  6. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version # 多个ReplicaSet对应不同版本
  7. NAME                      DESIRED   CURRENT   READY   AGE     VERSION
  8. nginx-deploy-5597c8b45    0         0         0       10m     v1.12
  9. nginx-deploy-6bd88df699   5         5         5       9m31s   v1.13
  1. # 升级过程中的状态变化:
  2. [root@hdss7-21 ~]# kubectl rollout status deployment nginx-deploy -n app 
  3. Waiting for deployment "nginx-deploy" rollout to finish: 5 old replicas are pending termination...
  4. Waiting for deployment "nginx-deploy" rollout to finish: 5 old replicas are pending termination...
  5. Waiting for deployment "nginx-deploy" rollout to finish: 5 old replicas are pending termination...
  6. Waiting for deployment "nginx-deploy" rollout to finish: 4 old replicas are pending termination...
  7. Waiting for deployment "nginx-deploy" rollout to finish: 4 old replicas are pending termination...
  8. Waiting for deployment "nginx-deploy" rollout to finish: 4 old replicas are pending termination...
  9. Waiting for deployment "nginx-deploy" rollout to finish: 3 old replicas are pending termination...
  10. Waiting for deployment "nginx-deploy" rollout to finish: 3 old replicas are pending termination...
  11. Waiting for deployment "nginx-deploy" rollout to finish: 3 old replicas are pending termination...
  12. Waiting for deployment "nginx-deploy" rollout to finish: 2 old replicas are pending termination...
  13. Waiting for deployment "nginx-deploy" rollout to finish: 2 old replicas are pending termination...
  14. Waiting for deployment "nginx-deploy" rollout to finish: 2 old replicas are pending termination...
  15. Waiting for deployment "nginx-deploy" rollout to finish: 1 old replicas are pending termination...
  16. Waiting for deployment "nginx-deploy" rollout to finish: 1 old replicas are pending termination...
  17. deployment "nginx-deploy" successfully rolled out
  18. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w
  19. NAME                           READY   STATUS    RESTARTS   AGE   VERSION
  20. nginx-deploy-5597c8b45-t5plt   1/1     Running   0          19s   v1.12
  21. nginx-deploy-5597c8b45-tcq69   1/1     Running   0          19s   v1.12
  22. nginx-deploy-5597c8b45-vdjxg   1/1     Running   0          19s   v1.12
  23. nginx-deploy-5597c8b45-vqn9x   1/1     Running   0          19s   v1.12
  24. nginx-deploy-5597c8b45-zl6qr   1/1     Running   0          19s   v1.12
  25. ---- 立刻创建5个新版本pod，Pending调度中
  26. nginx-deploy-6bd88df699-242fr   0/1     Pending   0          0s    v1.13
  27. nginx-deploy-6bd88df699-242fr   0/1     Pending   0          0s    v1.13
  28. nginx-deploy-6bd88df699-8pmdg   0/1     Pending   0          0s    v1.13
  29. nginx-deploy-6bd88df699-4kj8z   0/1     Pending   0          0s    v1.13
  30. nginx-deploy-6bd88df699-n7x6n   0/1     Pending   0          0s    v1.13
  31. nginx-deploy-6bd88df699-8pmdg   0/1     Pending   0          0s    v1.13
  32. nginx-deploy-6bd88df699-4kj8z   0/1     Pending   0          0s    v1.13
  33. nginx-deploy-6bd88df699-8j85n   0/1     Pending   0          0s    v1.13
  34. nginx-deploy-6bd88df699-n7x6n   0/1     Pending   0          0s    v1.13
  35. nginx-deploy-6bd88df699-8j85n   0/1     Pending   0          0s    v1.13
  36. ---- 创建pod中
  37. nginx-deploy-6bd88df699-242fr   0/1     ContainerCreating   0          0s    v1.13
  38. nginx-deploy-6bd88df699-8pmdg   0/1     ContainerCreating   0          0s    v1.13
  39. nginx-deploy-6bd88df699-4kj8z   0/1     ContainerCreating   0          0s    v1.13
  40. nginx-deploy-6bd88df699-n7x6n   0/1     ContainerCreating   0          0s    v1.13
  41. nginx-deploy-6bd88df699-8j85n   0/1     ContainerCreating   0          0s    v1.13
  42. ---- 启动pod
  43. nginx-deploy-6bd88df699-242fr   0/1     Running             0          1s    v1.13
  44. nginx-deploy-6bd88df699-8j85n   0/1     Running             0          1s    v1.13
  45. nginx-deploy-6bd88df699-4kj8z   0/1     Running             0          1s    v1.13
  46. nginx-deploy-6bd88df699-n7x6n   0/1     Running             0          1s    v1.13
  47. nginx-deploy-6bd88df699-8pmdg   0/1     Running             0          1s    v1.13
  48. ---- Pod逐个就绪，且替换旧版本的pod
  49. nginx-deploy-6bd88df699-242fr   1/1     Running             0          6s    v1.13
  50. nginx-deploy-5597c8b45-t5plt    1/1     Terminating         0          50s   v1.12
  51. nginx-deploy-6bd88df699-8j85n   1/1     Running             0          7s    v1.13
  52. nginx-deploy-5597c8b45-vdjxg    1/1     Terminating         0          51s   v1.12
  53. nginx-deploy-5597c8b45-t5plt    0/1     Terminating         0          51s   v1.12
  54. nginx-deploy-5597c8b45-t5plt    0/1     Terminating         0          51s   v1.12
  55. nginx-deploy-6bd88df699-4kj8z   1/1     Running             0          7s    v1.13
  56. nginx-deploy-5597c8b45-zl6qr    1/1     Terminating         0          51s   v1.12
  57. nginx-deploy-5597c8b45-vdjxg    0/1     Terminating         0          52s   v1.12
  58. nginx-deploy-5597c8b45-vdjxg    0/1     Terminating         0          52s   v1.12
  59. nginx-deploy-5597c8b45-zl6qr    0/1     Terminating         0          53s   v1.12
  60. nginx-deploy-5597c8b45-t5plt    0/1     Terminating         0          54s   v1.12
  61. nginx-deploy-5597c8b45-t5plt    0/1     Terminating         0          54s   v1.12
  62. nginx-deploy-5597c8b45-zl6qr    0/1     Terminating         0          56s   v1.12
  63. nginx-deploy-5597c8b45-zl6qr    0/1     Terminating         0          56s   v1.12
  64. nginx-deploy-6bd88df699-n7x6n   1/1     Running             0          13s   v1.13
  65. nginx-deploy-5597c8b45-tcq69    1/1     Terminating         0          57s   v1.12
  66. nginx-deploy-5597c8b45-tcq69    0/1     Terminating         0          58s   v1.12
  67. nginx-deploy-5597c8b45-tcq69    0/1     Terminating         0          59s   v1.12
  68. nginx-deploy-6bd88df699-8pmdg   1/1     Running             0          15s   v1.13
  69. nginx-deploy-5597c8b45-vqn9x    1/1     Terminating         0          59s   v1.12
  70. nginx-deploy-5597c8b45-vqn9x    0/1     Terminating         0          60s   v1.12
  71. nginx-deploy-5597c8b45-vqn9x    0/1     Terminating         0          61s   v1.12
  72. nginx-deploy-5597c8b45-vqn9x    0/1     Terminating         0          61s   v1.12
  73. nginx-deploy-5597c8b45-vdjxg    0/1     Terminating         0          64s   v1.12
  74. nginx-deploy-5597c8b45-vdjxg    0/1     Terminating         0          64s   v1.12
  75. nginx-deploy-5597c8b45-tcq69    0/1     Terminating         0          64s   v1.12
  76. nginx-deploy-5597c8b45-tcq69    0/1     Terminating         0          64s   v1.12

2.3.3. 滚动发布

通过定义 maxsurge 和 maxUnavailable 来实现滚动升级的速度，滚动升级中，可以使用 kubectl rollout pause 来实现暂停。

  1. [root@hdss7-200 deployment]# vim /data/k8s-yaml/base_resource/deployment/nginx-v1.14.yaml
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: nginx-deploy
  6.   namespace: app
  7. spec:
  8.   replicas: 5
  9.   selector:
  10.     matchLabels:
  11.       app: nginx
  12.       release: stable
  13.       tier: slb
  14.       partition: website
  15.   strategy:
  16.     rollingUpdate:
  17.       # 以下两项，控制升级速度
  18.       maxSurge: 1
  19.       maxUnavailable: 0
  20.   template:
  21.     metadata:
  22.       labels:
  23.         app: nginx
  24.         release: stable
  25.         tier: slb
  26.         partition: website
  27.         # 修改版本
  28.         version: v1.14
  29.     spec:
  30.       containers:
  31.       - name: nginx-pod
  32.         # 修改镜像版本
  33.         image: harbor.od.com/public/nginx:v1.14
  34.         lifecycle:
  35.           postStart:
  36.             exec:
  37.               command:
  38.                 - /bin/bash
  39.                 - -c
  40.                 - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  41.         readinessProbe: 
  42.           initialDelaySeconds: 5
  43.           httpGet:
  44.             port: 80
  45.             path: /health.html
  46.         livenessProbe:
  47.           initialDelaySeconds: 10
  48.           periodSeconds: 5
  49.           httpGet:
  50.             port: 80
  51.             path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  2. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version  # replicaset 数量增加
  3. NAME                      DESIRED   CURRENT   READY   AGE    VERSION
  4. nginx-deploy-5597c8b45    0         0         0       155m   v1.12
  5. nginx-deploy-6bd88df699   0         0         0       154m   v1.13
  6. nginx-deploy-7c5976dcd9   5         5         5       83s    v1.14
  7. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app # 升级记录
  8. REVISION  CHANGE-CAUSE
  9. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  10. 2         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  11. 3         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  1. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w  # 逐个滚动升级
  2. NAME                            READY   STATUS    RESTARTS   AGE    VERSION
  3. nginx-deploy-6bd88df699-242fr   1/1     Running   0          152m   v1.13
  4. nginx-deploy-6bd88df699-4kj8z   1/1     Running   0          152m   v1.13
  5. nginx-deploy-6bd88df699-8j85n   1/1     Running   0          152m   v1.13
  6. nginx-deploy-6bd88df699-8pmdg   1/1     Running   0          152m   v1.13
  7. nginx-deploy-6bd88df699-n7x6n   1/1     Running   0          152m   v1.13
  8. nginx-deploy-7c5976dcd9-ttlqx   0/1     Pending   0          0s     v1.14
  9. nginx-deploy-7c5976dcd9-ttlqx   0/1     Pending   0          0s     v1.14
  10. nginx-deploy-7c5976dcd9-ttlqx   0/1     ContainerCreating   0          0s     v1.14
  11. nginx-deploy-7c5976dcd9-ttlqx   0/1     Running             0          1s     v1.14
  12. nginx-deploy-7c5976dcd9-ttlqx   1/1     Running             0          9s     v1.14
  13. nginx-deploy-6bd88df699-8pmdg   1/1     Terminating         0          153m   v1.13
  14. ......

2.3.4. 模拟灰度(金丝雀)发布

灰度发布在不同场景中实现方式不同，如果当前灰度机器仅对测试开放，可以定义一个新的deployment来配合service来实现。如果需要切入一部分随机真实用户的流量，可以将生产机器中一台机器作为灰度机器，通过灰度后再升级其它的机器。

  1. # nginx-v1.15.yaml 与 nginx-v1.14.yaml 一致，仅仅修改了镜像文件
  2. [root@hdss7-21 ~]# kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true && kubectl rollout pause deployment nginx-deploy -n app
  3. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app
  4. REVISION  CHANGE-CAUSE
  5. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  6. 2         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  7. 3         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  8. 4         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true
  9. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version  # 存在两个ReplicaSet对外提供服务
  10. NAME                      DESIRED   CURRENT   READY   AGE     VERSION
  11. nginx-deploy-5597c8b45    0         0         0       177m    v1.12
  12. nginx-deploy-6695fd9655   1         1         1       2m22s   v1.15
  13. nginx-deploy-6bd88df699   0         0         0       176m    v1.13
  14. nginx-deploy-7c5976dcd9   5         5         5       23m     v1.14
  15. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w # 新老共存
  16. NAME                            READY   STATUS    RESTARTS   AGE   VERSION
  17. nginx-deploy-6695fd9655-tcm76   1/1     Running   0          17s   v1.15
  18. nginx-deploy-7c5976dcd9-4tnv4   1/1     Running   0          21m   v1.14
  19. nginx-deploy-7c5976dcd9-bpjc2   1/1     Running   0          20m   v1.14
  20. nginx-deploy-7c5976dcd9-gv8qm   1/1     Running   0          20m   v1.14
  21. nginx-deploy-7c5976dcd9-ttlqx   1/1     Running   0          21m   v1.14
  22. nginx-deploy-7c5976dcd9-xq2qs   1/1     Running   0          21m   v1.14
  1. # 手动暂停
  2. [root@hdss7-21 ~]# kubectl rollout resume deployment nginx-deploy -n app && kubectl rollout pause deployment nginx-deploy -n app
  3. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w
  4. NAME                            READY   STATUS    RESTARTS   AGE     VERSION
  5. nginx-deploy-6695fd9655-jmb94   1/1     Running   0          19s     v1.15
  6. nginx-deploy-6695fd9655-tcm76   1/1     Running   0          6m19s   v1.15
  7. nginx-deploy-7c5976dcd9-4tnv4   1/1     Running   0          27m     v1.14
  8. nginx-deploy-7c5976dcd9-gv8qm   1/1     Running   0          26m     v1.14
  9. nginx-deploy-7c5976dcd9-ttlqx   1/1     Running   0          27m     v1.14
  10. nginx-deploy-7c5976dcd9-xq2qs   1/1     Running   0          27m     v1.14
  11. # 升级剩余所有机器
  12. [root@hdss7-21 ~]# kubectl rollout resume deployment nginx-deploy -n app

2.3.5. 版本回滚

当升级出现异常时，执行回滚即可。

  1. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app # 查看历史版本记录
  2. deployment.extensions/nginx-deploy 
  3. REVISION  CHANGE-CAUSE
  4. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  5. 2         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  6. 3         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  7. 4         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true
  8. [root@hdss7-21 ~]# kubectl rollout undo deployment nginx-deploy -n app
  9. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app # 版本3已经被版本5替代
  10. deployment.extensions/nginx-deploy 
  11. REVISION  CHANGE-CAUSE
  12. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  13. 2         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  14. 4         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true
  15. 5         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  16. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version 
  17. NAME                            READY   STATUS    RESTARTS   AGE     VERSION
  18. nginx-deploy-7c5976dcd9-2kps8   1/1     Running   0          2m20s   v1.14
  19. nginx-deploy-7c5976dcd9-bqs28   1/1     Running   0          2m6s    v1.14
  20. nginx-deploy-7c5976dcd9-jdvps   1/1     Running   0          2m13s   v1.14
  21. nginx-deploy-7c5976dcd9-vs8l4   1/1     Running   0          116s    v1.14
  22. nginx-deploy-7c5976dcd9-z99mb   1/1     Running   0          101s    v1.14
  23. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version
  24. NAME                      DESIRED   CURRENT   READY   AGE    VERSION
  25. nginx-deploy-5597c8b45    0         0         0       3h7m   v1.12
  26. nginx-deploy-6695fd9655   0         0         0       12m    v1.15
  27. nginx-deploy-6bd88df699   0         0         0       3h7m   v1.13
  28. nginx-deploy-7c5976dcd9   5         5         5       34m    v1.14

2.3.6. 常用命令

  1. kubectl rollout status deployment nginx-deploy -n app  # 查看版本升级过程
  2. kubectl rollout history deployment nginx-deploy -n app # 查看版本升级历史
  3. kubectl apply -f http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true  # 升级且记录升级命令
  4. kubectl rollout undo deployment nginx-deploy -n app    # 回滚到上个版本
  5. kubectl rollout undo deployment nginx-deploy --to-revision=3 -n app # 回滚到版本3

3. DaemonSet

3.1. DaemonSet介绍

DaemonSet 确保全部（或者一些）Node 上运行一个 Pod 的副本。当有 Node 加入集群时，也会为他们新增一个 Pod 。当有 Node 从集群移除时，这些 Pod 也会被回收。删除 DaemonSet 将会删除它创建的所有 Pod。使用 DaemonSet 的一些典型用法：

  • 运行集群存储 daemon，例如在每个 Node 上运行 glusterd、ceph。
  • 在每个 Node 上运行日志收集 daemon，例如fluentd、logstash。
  • 在每个 Node 上运行监控 daemon，例如 Prometheus Node Exporter。

3.2. 模板

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata
  4.     name        <string>            # 在一个名称空间不能重复
  5.     namespace   <string>            # 指定名称空间，默认defalut
  6.     labels      <map[string]string> # 标签
  7.     annotations <map[string]string> # 注释
  8. spec
  9.     selector                    <Object>            # 标签选择器
  10.         matchExpressions        <[]Object>          # 标签选择器的一种形式,多个条件使用AND连接
  11.             key                 <string> -required- # 标签中的Key
  12.             operator            <string> -required- # 操作符，支持 In, NotIn, Exists, DoesNotExist
  13.             values              <[]string>          # value的数组集合，当操作符为In或NotIn时不能为空
  14.         matchLabels             <map[string]string> # 使用key/value的格式做筛选
  15.     updateStrategy              <Object>            # 更新策略
  16.         type                    <string>            # 更新类型，支持 Recreate, RollingUpdate。默认RollingUpdate
  17.         rollingUpdate           <Object>            # 滚动更新策略，仅在type为RollingUpdate时使用
  18.             maxUnavailable      <string>            # 最大缺失Pod数，即滚动更新时最多可少于期望值出几个pod。支持数字和百分比格式
  19.     template                    <Object> -required- # Pod模板，和Pod管理器yaml几乎格式一致
  20.         metadata                <Object>            # Pod的metadata
  21.         spec                    <Object>            # Pod的spec

3.3. 案例

3.3.1. 创建daemonset

  1. [root@hdss7-200 base_resource]# cat /data/k8s-yaml/base_resource/daemonset/proxy-v1.12.yaml
  2. apiVersion: apps/v1
  3. kind: DaemonSet
  4. metadata:
  5.   name: proxy-daemonset
  6.   namespace: app
  7.   labels:
  8.       app: nginx
  9.       release: stable
  10.       partition: CRM
  11. spec:
  12.   selector:
  13.     matchLabels:
  14.       app: nginx
  15.       release: stable
  16.       tier: proxy
  17.       partition: CRM
  18.   updateStrategy:
  19.     rollingUpdate:
  20.       maxUnavailable: 1
  21.   template:
  22.     metadata:
  23.       labels:
  24.         app: nginx
  25.         release: stable
  26.         tier: proxy
  27.         partition: CRM
  28.         version: v1.12
  29.     spec:
  30.       containers:
  31.       - name: nginx-proxy
  32.         image: harbor.od.com/public/nginx:v1.12
  33.         ports:
  34.         - name: http
  35.           containerPort: 80
  36.           hostPort: 10080
  37.         lifecycle:
  38.           postStart:
  39.             exec:
  40.               command:
  41.                 - /bin/bash
  42.                 - -c
  43.                 - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  44.         readinessProbe: 
  45.           initialDelaySeconds: 5
  46.           httpGet:
  47.             port: 80
  48.             path: /health.html
  49.         livenessProbe:
  50.           initialDelaySeconds: 10
  51.           periodSeconds: 5
  52.           httpGet:
  53.             port: 80
  54.             path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.12.yaml --record
  2. [root@hdss7-21 ~]# kubectl get daemonset -n app
  3. NAME              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
  4. proxy-daemonset   2         2         2       2            2           <none>          56s
  1. [root@hdss7-21 ~]# kubectl get pod -n app -l tier=proxy -o wide
  2. NAME                    READY   STATUS    RESTARTS   AGE     IP            NODE                NOMINATED NODE   READINESS GATES
  3. proxy-daemonset-7stgs   1/1     Running   0          8m31s   172.7.22.9    hdss7-22.host.com   <none>           <none>
  4. proxy-daemonset-dxgdp   1/1     Running   0          8m31s   172.7.21.10   hdss7-21.host.com   <none>           <none>
  5. [root@hdss7-21 ~]# curl -s 10.4.7.22:10080/info  # 通过宿主机的端口访问
  6. 2020-01-22T13:15:58+00:00|172.7.22.9|nginx:v1.12
  7. [root@hdss7-21 ~]# curl -s 10.4.7.21:10080/info
  8. 2020-01-22T13:16:05+00:00|172.7.21.10|nginx:v1.12

3.3.2. 升级daemonset

daemonset的升级方式和deployment一致

  1. [root@hdss7-21 ~]# kubectl rollout history daemonset proxy-daemonset -n app
  2. REVISION  CHANGE-CAUSE
  3. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.12.yaml --record=true
  5. [root@hdss7-21 ~]# kubectl apply -f  http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.13.yaml --record
  7. [root@hdss7-21 ~]# kubectl rollout history daemonset proxy-daemonset -n app
  8. daemonset.extensions/proxy-daemonset 
  9. REVISION  CHANGE-CAUSE
  10. 1         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.12.yaml --record=true
  11. 2         kubectl apply --filename=http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.13.yaml --record=true
  12. [root@hdss7-21 ~]# kubectl get pod -n app -l tier=proxy -L version
  13. NAME                    READY   STATUS    RESTARTS   AGE     VERSION
  14. proxy-daemonset-7wr4f   1/1     Running   0          119s    v1.13
  15. proxy-daemonset-clhqk   1/1     Running   0          2m11s   v1.13

4. Job

4.1. Job介绍

Job也是一种很常用的控制器，在执行自动化脚本的时候就可以使用，比如使用ansible脚本job创建一个新的k8s集群，使用job创建MySQL账号密码，使用job去执行helm脚本等等。

4.2. 模板

Job的spec中没有selector!

  1. apiVersion: batch/v1
  2. kind: Job
  3. metadata
  4.     name        <string>            # 在一个名称空间不能重复
  5.     namespace   <string>            # 指定名称空间，默认defalut
  6.     labels      <map[string]string> # 标签
  7.     annotations <map[string]string> # 注释
  8. spec
  9.     backoffLimit    <integer>       # 重试次数，默认6
  10.     completions     <integer>       # 指定当前job要执行多少次pod
  11.     parallelism     <integer>       # 指定多个pod执行时的并发数
  12.     template                    <Object> -required- # Pod模板，和Pod管理器yaml几乎格式一致
  13.         metadata                <Object>            # Pod的metadata
  14.         spec                    <Object>            # Pod的spec

4.3. 案例

  1. [root@centos-7-51 ~]# cat /tmp/job.yaml 
  2. apiVersion: batch/v1
  3. kind: Job
  4. metadata:
  5.   name: test-job
  6.   namespace: default
  7. spec:
  8.   template:
  9.     metadata:
  10.       labels:
  11.         jobType: sleep
  12.     spec:
  13.       restartPolicy: Never
  14.       containers:
  15.       - name: job-sleep
  16.         image: busybox:latest
  17.         command:
  18.         - sleep
  19.         - "60"
  20. [root@centos-7-51 ~]# kubectl apply -f /tmp/job.yaml
  21. [root@centos-7-51 ~]# kubectl get pod -o wide| grep job
  22. test-job-2fccp                 0/1     Completed   0          3m21s   172.16.4.43   centos-7-55   <none>           <none>
  23. [root@centos-7-51 ~]# kubectl get job -o wide
  24. NAME       COMPLETIONS   DURATION   AGE     CONTAINERS   IMAGES           SELECTOR
  25. test-job   1/1           68s        3m26s   job-sleep    busybox:latest   controller-uid=3ba88fdd-ad1c-46c7-86b4-dc6e641ebdb4

5. CronJob

5.1. cronjob介绍

cronJob 是类似于Linux中的计划任务，用于执行周期性任务，如数据备份、周期性采集相关信息并发送邮件等。

https://kubernetes.io/zh/docs/tasks/job/automated-tasks-with-cron-jobs/

5.2. 模板

  1. apiVersion: batch/v1beta1
  2. kind: batch/v1beta1
  3. metadata        <Object>
  4.   name        <string>            # 在一个名称空间不能重复
  5.   namespace   <string>            # 指定名称空间，默认defalut
  6.   labels      <map[string]string> # 标签
  7.   annotations <map[string]string> # 注释
  8. spec                          <Object>
  9.   concurrencyPolicy           <string>              # 新任务执行时上个任务还未结束时如何处理，Allow(默认值)：允许新老任务并发执行；Forbid：忽略新任务；Replace：停止老任务，运行新任务
  10.   failedJobsHistoryLimit      <integer>             # 失败历史记录保留次数，默认3
  11.   successfulJobsHistoryLimit    <integer>             # 成功历史记录保留次数，默认1
  12.   schedule                    <string> -required-   # 计划任务的cron，与Linux的一致
  13.   startingDeadlineSeconds       <integer>             # 如果任务错过调度的时间超过秒数表示任务失败，超过100次不再调度，参考 https://kubernetes.io/zh/docs/tasks/job/automated-tasks-with-cron-jobs/#starting-deadline
  14.   suspend                       <boolean>             # https://kubernetes.io/zh/docs/tasks/job/automated-tasks-with-cron-jobs/#%E6%8C%82%E8%B5%B7
  15.   jobTemplate                   <Object> -required-   # job 的任务模板
  16.     metadata                  <Object>
  17.     spec                      <Object>

5.3. 案例

  1. apiVersion: batch/v1beta1
  2. kind: CronJob
  3. metadata:
  4.   name: hello
  5. spec:
  6.   schedule: "*/1 * * * *"
  7.   concurrencyPolicy: Forbid
  8.   successfulJobsHistoryLimit: 3
  9.   failedJobsHistoryLimit: 10
  10.   jobTemplate:
  11.     spec:
  12.       template:
  13.         spec:
  14.           containers:
  15.           - name: hello
  16.             image: busybox
  17.             imagePullPolicy: IfNotPresent
  18.             args:
  19.             - /bin/sh
  20.             - -c
  21.             - date; echo Hello from the Kubernetes cluster
  22.           restartPolicy: OnFailure
  23. [root@centos-7-52 ~]# kubectl apply -f /tmp/cronjob.yaml 
  24. [root@centos-7-52 ~]# kubectl get cronjob -o wide #查看cronjob
  25. NAME    SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE     CONTAINERS   IMAGES    SELECTOR
  26. hello   */1 * * * *   False     0        52s             6m36s   hello        busybox   <none>
  27. [root@centos-7-52 ~]# kubectl get job -o wide | grep hello # 查看执行的job历史，Job名称中的数字为时间戳
  28. hello-1607757060   1/1           1s         2m48s   hello        busybox   controller-uid=194350a4-21ca-4d01-b264-34a963e78ce7
  29. hello-1607757120   1/1           1s         108s    hello        busybox   controller-uid=e30a318c-6c0f-4d1d-b200-3dfaffe083c5
  30. hello-1607757180   1/1           1s         48s     hello        busybox   controller-uid=a00d02fc-8d7e-40b3-8e2a-25620ecab427
  31. [root@centos-7-52 ~]# date -d @1607757180
  32. Sat Dec 12 15:13:00 CST 2020

6. StatefulSet

6.1. 介绍

Deployment 是用来发布和管理无状态服务的，所有节点之间是相同的，他们挂载相同的共享存储，且Pod名称后缀也是随机生成的，异常后可以随时重启。而有状态应用程序，他们的Pod之间是有区别的，启动也是有顺序要求，而且往往需要挂载不同的存储卷。由于有状态的应用程序本身比较复杂，尤其是发生故障重启时可能需要人为干预，因此复杂的有状态程序不建议放到K8s集群中，如MySQL集群。

K8S中StatefulSet 是用来管理有状态应用的工作负载 API 对象，StatefulSet 用来管理某 Pod 集合的部署和扩缩， 并为这些 Pod 提供持久存储和持久标识符。和 Deployment 类似， StatefulSet 管理基于相同容器规约的一组 Pod。但和 Deployment 不同的是， StatefulSet 为它们的每个 Pod 维护了一个有粘性的 ID。这些 Pod 是基于相同的规约来创建的， 但是不能相互替换：无论怎么调度，每个 Pod 都有一个永久不变的 ID。如果希望使用存储卷为工作负载提供持久存储，可以使用 StatefulSet 作为解决方案的一部分。 尽管 StatefulSet 中的单个 Pod 仍可能出现故障， 但持久的 Pod 标识符使得将现有卷与替换已失败 Pod 的新 Pod 相匹配变得更加容易。

6.1.1. Statefulset使用场景

  • 稳定的、唯一的网络标识符，即每个Pod有自己独有的域名，无论重启多少次都不会变
  • 稳定的、持久的存储，即每个Pod有自己独享的PVC
  • 有序的、优雅的部署和缩放，即每个Pod会按照序号从小到大启动，按照需要从大到小被停止
  • 有序的、自动的滚动更新，即每个Pod按照需要从大到小进行重启升级

6.1.2. Statefulset注意项

  • 需要定义pvc申请模板，每个pod使用不同的pvc。可以通过storageclass分配或者提前创建好pv
  • 需要一个headless的service作为服务发现
  • 如果需要删除statefulset，推荐将副本数置为0再删除
  • 删除statefulset，并不会删除申请的pvc和pv
  • 升级过程中，如果出现问题，可能需要人为干预

6.2. 模板

  1. apiVersion: apps/v1
  2. kind: StatefulSet
  3. metadata                    <Object>
  4. spec:
  5.   replicas                  <integer>
  6.   serviceName                 <string> -required-
  7.   selector                  <Object> -required-   # Pod选择器
  8.   template                  <Object> -required-   # Pod模板
  9.   volumeClaimTemplates      <[]Object>
  10.     metadata                <Object>
  11.     spec                    <Object>
  12.       accessModes           <[]string>            # RWO:单路读写;ROX:单路只读;RWX:多路读写
  13.       resources             <Object>              # 指定当前PVC需要的系统最小资源限制
  14.         limits              <map[string]string>   # 资源限制,一般不配置
  15.         requests            <map[string]string>   # 资源限制，常用为 storage: xGi
  16.       storageClassName      <string>              # 声明使用的存储类，用于动态分配
  18.       volumeMode              <string>              # 使用卷的文件系统还是当成块设备来用，一般不配置
  19.       selector              <Object>              # 标签选择器，选择PV的标签，默认在所有PV中寻找
  20.       volumeName              <string>              # 指定PV名称，直接绑定PV
  21.   revisionHistoryLimit      <integer>             # 保留历史版本数量，默认10
  22.   podManagementPolicy         <string>              # 控制Pod启停顺序，默认OrderedReady，启动顺序，停止倒序。Parallel表示并行
  23.   updateStrategy              <Object>              # 更新策略
  24.     rollingUpdate           <Object>              # 滚动更新策略
  25.       partition             <integer>             # 指定更新到哪个分区，N表示更新序号大于等于N的Pod
  26.     type                    <string>              # 更新策略，默认滚动更新RollingUpdate.

6.3. 案例

6.3.1. 创建Statefulset资源

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: my-svc
  6.   namespace: apps
  7. spec:
  8.   selector:
  9.     app: nginx-web
  10.     role: web
  11.     state: "true"
  12.   clusterIP: None
  13.   ports:
  14.   - name: http
  15.     port: 80
  16.     targetPort: 80
  18. ---
  19. apiVersion: apps/v1
  20. kind: StatefulSet
  21. metadata:
  22.   name: nginx-web
  23.   namespace: apps
  24.   labels:
  25.     app: nginx
  26.     role: web
  27. spec:
  28.   replicas: 2
  29.   serviceName: my-svc
  30.   selector:
  31.     matchLabels:
  32.       app: nginx-web
  33.       role: web
  34.       state: "true"
  35.   template:
  36.     metadata:
  37.       labels:
  38.         app: nginx-web
  39.         role: web
  40.         state: "true"
  41.     spec:
  42.       containers:
  43.       - name: nginx-web
  44.         image: linuxduduniao/nginx:v1.0.0
  45.         ports:
  46.         - name: http
  47.           containerPort: 80
  48.         readinessProbe:
  49.           httpGet:
  50.             port: 80
  51.             path: /health
  52.         volumeMounts:
  53.         - name: nginx-web
  54.           mountPath: /usr/share/nginx/html
  55.   volumeClaimTemplates:
  56.   - metadata:
  57.       name: nginx-web
  58.       namespace: apps
  59.     spec:
  60.       accessModes: ["ReadWriteOnce"]
  61.       resources:
  62.         requests:
  63.           storage: 2Gi      
  64.       storageClassName: managed-nfs-storage # nfs classstorage,参考https://www.yuque.com/duduniao/k8s/vgms23#3W9oz
  65. [root@duduniao local-k8s-yaml]# kubectl -n apps get sts -o wide # 当前sts 状态
  66. NAME        READY   AGE   CONTAINERS   IMAGES
  67. nginx-web   2/2     19s   nginx-web    linuxduduniao/nginx:v1.0.0
  69. [root@duduniao local-k8s-yaml]# kubectl -n apps get pod -o wide # 注意Pod名称
  70. NAME          READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
  71. nginx-web-0   1/1     Running   0          86s   172.16.4.158   centos-7-55   <none>           <none>
  72. nginx-web-1   1/1     Running   0          77s   172.16.3.98    centos-7-54   <none>           <none>
  74. [root@duduniao local-k8s-yaml]# kubectl -n apps get pvc -o wide # 注意pvc名称
  75. NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE   VOLUMEMODE
  76. nginx-web-nginx-web-0   Bound    pvc-b36b90f1-a268-4815-b55b-6f7fda587193   2Gi        RWX            managed-nfs-storage   93s   Filesystem
  77. nginx-web-nginx-web-1   Bound    pvc-1461c537-e784-41fd-9e78-b1e6b212912c   2Gi        RWX            managed-nfs-storage   84s   Filesystem
  79. [root@duduniao local-k8s-yaml]# kubectl -n apps get pv -o wide # 注意pv名称
  80. NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS          REASON   AGE   VOLUMEMODE
  81. pvc-1461c537-e784-41fd-9e78-b1e6b212912c   2Gi        RWX            Delete           Bound    apps/nginx-web-nginx-web-1   managed-nfs-storage            88s   Filesystem
  82. pvc-b36b90f1-a268-4815-b55b-6f7fda587193   2Gi        RWX            Delete           Bound    apps/nginx-web-nginx-web-0   managed-nfs-storage            97s   Filesystem
  84. [root@duduniao local-k8s-yaml]# kubectl -n apps describe svc my-svc
  85. Name:              my-svc
  86. Namespace:         apps
  87. Labels:            <none>
  88. Annotations:       Selector:  app=nginx-web,role=web,state=true
  89. Type:              ClusterIP
  90. IP:                None
  91. Port:              http  80/TCP
  92. TargetPort:        80/TCP
  93. Endpoints:         172.16.3.98:80,172.16.4.158:80
  94. Session Affinity:  None
  95. Events:            <none>
  96. [root@centos-7-51 ~]# dig -t A my-svc.apps.svc.cluster.local @10.96.0.10 +short   # 无头服务特性，用于服务发现
  97. 172.16.4.216
  98. 172.16.3.105
  99. [root@centos-7-51 ~]# dig -t A nginx-web-0.my-svc.apps.svc.cluster.local @10.96.0.10 +short # statefulset 特性，可以用pod名称解析到Pod的地址
  100. 172.16.4.216
  101. [root@centos-7-51 ~]# dig -t A nginx-web-1.my-svc.apps.svc.cluster.local @10.96.0.10 +short
  102. 172.16.3.105

6.3.2. 扩缩容

在默认的策略下，statefulset启停Pod都是按照顺序执行的，新建Pod按照序号从小到大进行，删除Pod从大到小进行，通常需要在Pod配置就绪性探针，严格保证次序。

  1. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":5}}'  # 增加副本数
  2. [root@duduniao ~]# kubectl get pod -n apps -w  # 观察启动顺序，前一个就绪后，再启动下一个
  3. NAME          READY   STATUS    RESTARTS   AGE
  4. nginx-web-0   1/1     Running   2          10h
  5. nginx-web-1   1/1     Running   2          10h
  6. nginx-web-2   0/1     Pending   0          0s
  7. nginx-web-2   0/1     Pending   0          0s
  8. nginx-web-2   0/1     Pending   0          1s
  9. nginx-web-2   0/1     ContainerCreating   0          2s
  10. nginx-web-2   0/1     Running             0          3s
  11. nginx-web-2   1/1     Running             0          5s
  12. nginx-web-3   0/1     Pending             0          0s
  13. nginx-web-3   0/1     Pending             0          0s
  14. nginx-web-3   0/1     Pending             0          1s
  15. nginx-web-3   0/1     ContainerCreating   0          2s
  16. nginx-web-3   0/1     Running             0          3s
  17. nginx-web-3   1/1     Running             0          9s
  18. nginx-web-4   0/1     Pending             0          0s
  19. nginx-web-4   0/1     Pending             0          0s
  20. nginx-web-4   0/1     Pending             0          1s
  21. nginx-web-4   0/1     ContainerCreating   0          2s
  22. nginx-web-4   0/1     Running             0          3s
  23. nginx-web-4   1/1     Running             0          7s
  24. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":2}}' # 减少副本数
  25. [root@duduniao ~]# kubectl get pod -n apps -w   # 观察停止顺序，按序号倒序停止
  26. NAME          READY   STATUS    RESTARTS   AGE
  27. nginx-web-0   1/1     Running   2          10h
  28. nginx-web-1   1/1     Running   2          10h
  29. nginx-web-2   1/1     Running   0          117s
  30. nginx-web-3   1/1     Running   0          112s
  31. nginx-web-4   1/1     Running   0          103s
  32. nginx-web-4   1/1     Terminating   0          109s
  33. nginx-web-4   0/1     Terminating   0          110s
  34. nginx-web-4   0/1     Terminating   0          111s
  35. nginx-web-4   0/1     Terminating   0          111s
  36. nginx-web-4   0/1     Terminating   0          111s
  37. nginx-web-3   1/1     Terminating   0          2m
  38. nginx-web-3   0/1     Terminating   0          2m
  39. nginx-web-3   0/1     Terminating   0          2m1s
  40. nginx-web-3   0/1     Terminating   0          2m1s
  41. nginx-web-2   1/1     Terminating   0          2m7s
  42. nginx-web-2   0/1     Terminating   0          2m7s
  43. nginx-web-2   0/1     Terminating   0          2m13s
  44. nginx-web-2   0/1     Terminating   0          2m13s

6.3.3. 滚动更新

滚动更新是按照倒序方式进行，可以指定分区号，如指定分区号N，则表示更新序号大于等于N的Pod，默认分区为0，表示更新所有Pod

  1. [root@duduniao local-k8s-yaml]# kubectl -n apps set image sts nginx-web nginx-web=linuxduduniao/nginx:v1.0.1
  2. [root@duduniao ~]# kubectl get pod -n apps -w  # 注意是倒序更新所有Pod
  3. NAME          READY   STATUS    RESTARTS   AGE
  4. nginx-web-0   1/1     Running   2          10h
  5. nginx-web-1   1/1     Running   2          10h
  6. nginx-web-1   1/1     Terminating   2          10h
  7. nginx-web-1   0/1     Terminating   2          10h
  8. nginx-web-1   0/1     Terminating   2          10h
  9. nginx-web-1   0/1     Terminating   2          10h
  10. nginx-web-1   0/1     Pending       0          0s
  11. nginx-web-1   0/1     Pending       0          0s
  12. nginx-web-1   0/1     ContainerCreating   0          0s
  13. nginx-web-1   0/1     Running             0          22s
  14. nginx-web-1   1/1     Running             0          27s
  15. nginx-web-0   1/1     Terminating         2          10h
  16. nginx-web-0   0/1     Terminating         2          10h
  17. nginx-web-0   0/1     Terminating         2          10h
  18. nginx-web-0   0/1     Terminating         2          10h
  19. nginx-web-0   0/1     Pending             0          0s
  20. nginx-web-0   0/1     Pending             0          0s
  21. nginx-web-0   0/1     ContainerCreating   0          0s
  22. nginx-web-0   0/1     ErrImagePull        0          29s
  23. nginx-web-0   0/1     ImagePullBackOff    0          44s
  24. nginx-web-0   0/1     Running             0          62s
  25. nginx-web-0   1/1     Running             0          70s
  26. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":5}}' # 扩容到5个副本，方便演示
  27. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":3}}}}' # 指定更新分区为3
  28. [root@duduniao local-k8s-yaml]# kubectl -n apps set image sts nginx-web nginx-web=linuxduduniao/nginx:v1.0.2 # 开始更新
  29. [root@duduniao ~]# kubectl get pod -n apps -w  # 只更新序号4和3的Pod
  30. NAME          READY   STATUS    RESTARTS   AGE
  31. nginx-web-0   1/1     Running   0          6m31s
  32. nginx-web-1   1/1     Running   0          7m12s
  33. nginx-web-2   1/1     Running   0          3m38s
  34. nginx-web-3   1/1     Running   0          3m33s
  35. nginx-web-4   1/1     Running   0          3m23s
  36. nginx-web-4   1/1     Terminating   0          3m25s
  37. nginx-web-4   0/1     Terminating   0          3m26s
  38. nginx-web-4   0/1     Terminating   0          3m32s
  39. nginx-web-4   0/1     Terminating   0          3m32s
  40. nginx-web-4   0/1     Pending       0          0s
  41. nginx-web-4   0/1     Pending       0          0s
  42. nginx-web-4   0/1     ContainerCreating   0          0s
  43. nginx-web-4   0/1     Running             0          10s
  44. nginx-web-4   1/1     Running             0          17s
  45. nginx-web-3   1/1     Terminating         0          3m59s
  46. nginx-web-3   0/1     Terminating         0          4m
  47. nginx-web-3   0/1     Terminating         0          4m1s
  48. nginx-web-3   0/1     Terminating         0          4m1s
  49. nginx-web-3   0/1     Pending             0          0s
  50. nginx-web-3   0/1     Pending             0          0s
  51. nginx-web-3   0/1     ContainerCreating   0          0s
  52. nginx-web-3   0/1     Running             0          22s
  53. nginx-web-3   1/1     Running             0          25s
  54. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":0}}}}' # 修改分区为0
  55. [root@duduniao ~]# kubectl get pod -n apps -w   # 会更新剩下来的三个Pod
  56. NAME          READY   STATUS    RESTARTS   AGE
  57. nginx-web-0   1/1     Running   0          9m10s
  58. nginx-web-1   1/1     Running   0          9m51s
  59. nginx-web-2   1/1     Running   0          6m17s
  60. nginx-web-3   1/1     Running   0          2m11s
  61. nginx-web-4   1/1     Running   0          2m30s
  62. nginx-web-2   1/1     Terminating   0          6m24s
  63. nginx-web-2   0/1     Terminating   0          6m24s
  64. nginx-web-2   0/1     Terminating   0          6m25s
  65. nginx-web-2   0/1     Terminating   0          6m25s
  66. nginx-web-2   0/1     Pending       0          0s
  67. nginx-web-2   0/1     Pending       0          0s
  68. nginx-web-2   0/1     ContainerCreating   0          0s
  69. nginx-web-2   0/1     Running             0          1s
  70. nginx-web-2   1/1     Running             0          9s
  71. nginx-web-1   1/1     Terminating         0          10m
  72. nginx-web-1   0/1     Terminating         0          10m
  73. nginx-web-1   0/1     Terminating         0          10m
  74. nginx-web-1   0/1     Terminating         0          10m
  75. nginx-web-1   0/1     Pending             0          0s
  76. nginx-web-1   0/1     Pending             0          0s
  77. nginx-web-1   0/1     ContainerCreating   0          0s
  78. nginx-web-1   0/1     Running             0          1s
  79. nginx-web-1   1/1     Running             0          8s
  80. nginx-web-0   1/1     Terminating         0          9m47s
  81. nginx-web-0   0/1     Terminating         0          9m48s
  82. nginx-web-0   0/1     Terminating         0          10m
  83. nginx-web-0   0/1     Terminating         0          10m
  84. nginx-web-0   0/1     Pending             0          0s
  85. nginx-web-0   0/1     Pending             0          0s
  86. nginx-web-0   0/1     ContainerCreating   0          0s
  87. nginx-web-0   0/1     Running             0          1s
  88. nginx-web-0   1/1     Running             0          9s

6.3.4. 删除Statefulset

与statefulset相关资源有三部分：statefulset、pv/pvc、svc，svc和pv/pvc需要手动删除。

通常删除statefulset时，推荐将其副本数置为0，实现有序停止，然后再删除statefulset，最后考虑是否需要删除svc和存储卷！

statefulset还提供了 --cascade=false 选项，表示不删除对应Pod，不过此项使用较少。

  1. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":0}}' # 有序停止
  2. [root@duduniao local-k8s-yaml]# kubectl delete -n apps sts nginx-web # 删除statefulset
  3. [root@duduniao local-k8s-yaml]# kubectl delete svc -n apps my-svc # 删除svc
  5. [root@duduniao local-k8s-yaml]# kubectl get pv -n apps
  6. NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS          REASON   AGE
  7. pvc-1461c537-e784-41fd-9e78-b1e6b212912c   2Gi        RWX            Delete           Bound    apps/nginx-web-nginx-web-1   managed-nfs-storage            11h
  8. pvc-277cff8f-c5e6-43fc-9498-ddc9ba788cb9   2Gi        RWO            Delete           Bound    apps/nginx-web-nginx-web-4   managed-nfs-storage            30m
  9. pvc-7d1a1169-6e34-4331-9549-f2f4d2bf6f94   2Gi        RWO            Delete           Bound    apps/nginx-web-nginx-web-3   managed-nfs-storage            30m
  10. pvc-b36b90f1-a268-4815-b55b-6f7fda587193   2Gi        RWX            Delete           Bound    apps/nginx-web-nginx-web-0   managed-nfs-storage            11h
  11. pvc-d804a299-95e1-486c-a9cd-73c925ed21d6   2Gi        RWO            Delete           Bound    apps/nginx-web-nginx-web-2   managed-nfs-storage            31m
  12. [root@duduniao local-k8s-yaml]# kubectl get pvc -n apps  # pv和pvc即使删除Pod还是处于绑定状态，需要手动释放
  13. NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
  14. nginx-web-nginx-web-0   Bound    pvc-b36b90f1-a268-4815-b55b-6f7fda587193   2Gi        RWX            managed-nfs-storage   11h
  15. nginx-web-nginx-web-1   Bound    pvc-1461c537-e784-41fd-9e78-b1e6b212912c   2Gi        RWX            managed-nfs-storage   11h
  16. nginx-web-nginx-web-2   Bound    pvc-d804a299-95e1-486c-a9cd-73c925ed21d6   2Gi        RWO            managed-nfs-storage   31m
  17. nginx-web-nginx-web-3   Bound    pvc-7d1a1169-6e34-4331-9549-f2f4d2bf6f94   2Gi        RWO            managed-nfs-storage   31m
  18. nginx-web-nginx-web-4   Bound    pvc-277cff8f-c5e6-43fc-9498-ddc9ba788cb9   2Gi        RWO            managed-nfs-storage   30m
  20. [root@duduniao local-k8s-yaml]# kubectl get pvc -n apps | awk '/nginx-web-nginx-web/{print "kubectl -n apps delete pvc", $1}'|bash

6.3.5. 部署有状态应用一般思路

有状态应用通常会组成集群，如MySQL。而且在启动之初还需要作初始化工作，如数据拷贝、数据导入等，通常需要initcontainer来实现。当出现故障时，如何顺利的进行故障转移也是需要考虑的，在实际操作中往往伴随着比较大的风险(相对于虚拟机部署而言)。一般性的建议是在对应官网找到kubernetes中部署方式，通常是helm脚本，通过配置合适的 values.yaml 来实现部署，但是即使如此，还是谨慎一些！

官方提供了两个示例：cassandra集群部署 、 MySQL集群部署。这里再推荐一个helm chart项目，以供学习使用：charts

7. 原理部分

7.1. Deployment创建过程

以 kubectl apply -f deployment.yaml 为例，了解 Deployment 创建的过程。

  • Kubectl 生成 API 请求

    • Kubectl 校验 deployment.yaml 中的字段，确保资源清单的合法性
    • Kubectl 检查 deployment.yaml 中的API字段，填充kubeconfig的认证信息，填充 deployment 请求体信息。
    • 将封装好的请求发送到 ApiServer

  • ApiServer 身份认证

    • apiServer 通过认证信息确认用户合法身份
    • 通过RBAC鉴权，确认用户具备资源操作权限
    • 通过准入控制，确保满足Kubernetes部分高级功能，参考文档
    • 将HTTP请求反序列化，存入ECTD中
    • 执行资源初始化逻辑，如边车容器注入、证书注入等操作

  • ControllerManager创建资源对象

    • Deployment Controller 通过ApiServer发现有Deployment需要创建
    • Deployment Controller 通过ApiServer查询对应ReplicaSet是否满足，不满则则创建新的ReplicaSet，并设置版本号
    • ReplicaSet Controller 通过ApiServer发现有新的 ReplicaSet资源，通过查询是否有满足条件的Pod，如果Pod不满足条件，则创建Pod对象

  • Scheduler 调度资源

    • Scheduler 通过ApiServer监听创建新Pod的请求，一旦需要创建新Pod，则通过调度算法选择Node
    • 先通过一组预算策略进行评估，筛选可以调度的Node
    • 再通过优选策略，在符合条件的Node中选择得分最高的Node作为运行该Pod的节点，最高分有多个节点时，随机选择一个
    • Scheduler 创建一个Binding对象并且请求ApiServer，该对象包含了Pod的唯一信息和选择的Nodename
  • Kubelet 初始化Pod

之前所有的操作，都是在操作 etcd 数据库，只有 Kubelet 这一步才开始正式创建Pod

    • Kubelet 通过ApiServer查询到当前自身节点上Pod清单，与自己缓存中记录进行比对，如果是新的Pod则进入创建流程
    • Kubelet 生成Podstatus对象，填充Pod状态(Pending，Running，Succeeded，Failed，Unkown)
    • Kubelet 检查Pod是否具备对应的准入权限，如果不具备则处于Pending状态
    • 如果Pod存在资源限制，则通过Cgroups 对Pod进行资源限制
    • 创建容器目录，并挂载存储卷的目录
    • 检索ImagePullSecret，用于拉取镜像
    • 通过CRI 接接口调用容器runtime创建容器

  • CRI 创建容器

    • 创建Pause容器，用于为其他容器提供共享的网络名称空间和启动pid名称空间，从而实现网络共享和僵尸进程回收
    • docker runtime 调用CNI插件，从IP资源池中分配一个IP地址给当前Pause容器
    • 拉取业务镜像，如果填充了 imagepullsecret 会通过指定的密钥去拉取镜像
    • 挂载configmap到容器
    • 填充Pod元数据信息
    • 启动容器，监听容器的event，执行对应钩子

03-8-K8S网络 - 图9

8. 调度

8.1. 调度器

Kubernetes 中调度是指为Pod分配合适的工作节点，该过程是由Scheduler组件完成，调度结果写入 etcd 数据库中，交由kubelet组件去完成Pod启动。scheduler 调度过程分两个阶段：

  • 预选策略：过滤出可调度节点。根据Pod清单筛选处集群中所有满足Pod运行条件的节点，这些节点称为可调度节点。
  • 优选策略：打分并选择最高分。对可调度节点进行打分，选择最高峰的节点，如果存在多个最高分则随机选一个。

Scheduler 的策略可以通过kube-scheduler进行修改，但是一般很少操作。

scheduler 有一个配置 percentageOfNodesToScore ，值是0-100之间，0表示使用默认值，100以上等价于100。当该值为N，节点数为 M时，表示当可调度节点数量达到 N*M*0.01 时，停止执行预选函数，直接进入优选打分环节，这样可以避免轮询过多的节点影响性能。scheduler 默认计算方式：在 100-节点集群 下取 50%，在 5000-节点的集群下取 10%，这个参数默认的最低值是 5%，另外如果最小可调度节点数是50，这是程序中写死的，无法调整。从官网的描述来看，当节点数在几百个甚至更少的情况，调整percentageOfNodesToScore没有明显效果。

8.2. 节点选择器

  1. # 为了方面演示，增加到三个节点，效果更加明显
  2. [root@centos-7-51 ~]# kubectl get node 
  3. NAME          STATUS   ROLES    AGE     VERSION
  4. centos-7-51   Ready    master   10d     v1.18.12
  5. centos-7-52   Ready    master   10d     v1.18.12
  6. centos-7-53   Ready    master   10d     v1.18.12
  7. centos-7-54   Ready    worker   10d     v1.18.12
  8. centos-7-55   Ready    worker   10d     v1.18.12
  9. centos-7-56   Ready    worker   7m58s   v1.18.12
  10. [root@centos-7-51 ~]# kubectl label node centos-7-54 ssd=true  # 打上标签方便区分
  11. [root@centos-7-51 ~]# kubectl label node centos-7-55 ssd=true
  12. [root@centos-7-51 ~]# kubectl label node centos-7-54 cpu=high
  13. [root@centos-7-51 ~]# kubectl label node centos-7-56 cpu=high

节点选择器有两种，一种时直接指定nodeName，另一种是通过 nodeSelector 来根据标签选择：

  • 指定nodeName
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   replicas: 5
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx-demo
  17.         image: linuxduduniao/nginx:v1.0.0
  18.       nodeName: centos-7-56
  19. [root@centos-7-51 ~]# kubectl get pod -o wide # 全部调度到 centos-7-56 节点
  20. NAME                           READY   STATUS    RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
  21. nginx-deploy-5648cd896-64ktx   1/1     Running   0          5m54s   172.16.5.11   centos-7-56   <none>           <none>
  22. nginx-deploy-5648cd896-fgx75   1/1     Running   0          5m54s   172.16.5.13   centos-7-56   <none>           <none>
  23. nginx-deploy-5648cd896-fvrlq   1/1     Running   0          5m54s   172.16.5.12   centos-7-56   <none>           <none>
  24. nginx-deploy-5648cd896-hzljl   1/1     Running   0          5m54s   172.16.5.15   centos-7-56   <none>           <none>
  25. nginx-deploy-5648cd896-qwrb5   1/1     Running   0          5m54s   172.16.5.14   centos-7-56   <none>           <none>
  • 使用nodeSelector
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   replicas: 5
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx-demo
  17.         image: linuxduduniao/nginx:v1.0.0
  18.       nodeSelector:
  19.         ssd: "true"
  20.         cpu: high
  21. [root@centos-7-51 ~]# kubectl get pod -o wide # nodeSelector多个选项之间是 and 关系
  22. NAME                            READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
  23. nginx-deploy-6d5b594bf5-b7s68   1/1     Running   0          14s   172.16.3.134   centos-7-54   <none>           <none>
  24. nginx-deploy-6d5b594bf5-kv5kn   1/1     Running   0          14s   172.16.3.132   centos-7-54   <none>           <none>
  25. nginx-deploy-6d5b594bf5-sxsgv   1/1     Running   0          11s   172.16.3.135   centos-7-54   <none>           <none>
  26. nginx-deploy-6d5b594bf5-t2p8n   1/1     Running   0          11s   172.16.3.136   centos-7-54   <none>           <none>
  27. nginx-deploy-6d5b594bf5-xrrhp   1/1     Running   0          14s   172.16.3.133   centos-7-54   <none>           <none>

8.3. 节点亲和性

节点亲和性分为强制选择(硬亲和)和优先选择(软亲和)：

  • 强制选择requiredDuringSchedulingIgnoredDuringExecution

    • 满足则调度(案例一)
    • 不满足则Pending(案例二)

  • 优先选择preferredDuringSchedulingIgnoredDuringExecution

    • 不满足则调度到其它节点(案例三)
    • 满足则优先调度到该节点，并不是一定调度到该节点，而且该节点优先级较高(案例四)
  • 案例一
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   replicas: 5
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx-demo
  17.         image: linuxduduniao/nginx:v1.0.0
  18.       affinity:
  19.         nodeAffinity:
  20.           requiredDuringSchedulingIgnoredDuringExecution:
  21.             nodeSelectorTerms:
  22.             - matchExpressions:
  23.               - key: ssd
  24.                 operator: DoesNotExist
  25.               - key: cpu
  26.                 operator: In
  27.                 values: ["high"]
  28. [root@duduniao ~]# kubectl get pod -o wide # 不存在ssd标签，并且cpu值为 high
  29. NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
  30. nginx-deploy-6f8b6d748c-4pt2x   1/1     Running   0          8s    172.16.5.20   centos-7-56   <none>           <none>
  31. nginx-deploy-6f8b6d748c-m9kb4   1/1     Running   0          8s    172.16.5.19   centos-7-56   <none>           <none>
  32. nginx-deploy-6f8b6d748c-st8mw   1/1     Running   0          5s    172.16.5.22   centos-7-56   <none>           <none>
  33. nginx-deploy-6f8b6d748c-w4mc9   1/1     Running   0          5s    172.16.5.21   centos-7-56   <none>           <none>
  34. nginx-deploy-6f8b6d748c-wjvxx   1/1     Running   0          8s    172.16.5.18   centos-7-56   <none>           <none>
  • 案例二
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   replicas: 5
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx-demo
  17.         image: linuxduduniao/nginx:v1.0.0
  18.       affinity:
  19.         nodeAffinity:
  20.           requiredDuringSchedulingIgnoredDuringExecution:
  21.             nodeSelectorTerms:
  22.             - matchExpressions:
  23.               - key: ssd
  24.                 operator: DoesNotExist
  25.               # 此处改为cpu不等于high
  26.               - key: cpu
  27.                 operator: NotIn
  28.                 values: ["high"]
  29. [root@duduniao ~]# kubectl describe pod nginx-deploy-746f88c86-96dbp # 注意message
  30. Name:           nginx-deploy-746f88c86-96dbp
  31. Namespace:      default
  32. Priority:       0
  33. Node:           <none>
  34. ......
  35. Status:         Pending
  36. ......
  37. Events:
  38.   Type     Reason            Age                From               Message
  39.   ----     ------            ----               ----               -------
  40.   Warning  FailedScheduling  30s (x3 over 32s)  default-scheduler  0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
  • 案例三
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   replicas: 5
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx-demo
  17.         image: linuxduduniao/nginx:v1.0.0
  18.       affinity:
  19.         nodeAffinity:
  20.           preferredDuringSchedulingIgnoredDuringExecution:
  21.           - preference:
  22.               matchExpressions:
  23.                 # 系统中并没有这个的Node
  24.                 - key: ssd
  25.                   operator: DoesNotExist
  26.                 - key: cpu
  27.                   operator: NotIn
  28.                   values: ["high"]
  29.             weight: 5
  30. [root@duduniao ~]# kubectl get pod -o wide # 没有满足的节点，所以随机分布了
  31. NAME                            READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
  32. nginx-deploy-5b9b5b66bc-5zrps   1/1     Running   0          31s   172.16.3.153   centos-7-54   <none>           <none>
  33. nginx-deploy-5b9b5b66bc-brwgb   1/1     Running   0          31s   172.16.4.249   centos-7-55   <none>           <none>
  34. nginx-deploy-5b9b5b66bc-f49j9   1/1     Running   0          31s   172.16.5.23    centos-7-56   <none>           <none>
  35. nginx-deploy-5b9b5b66bc-jnh45   1/1     Running   0          30s   172.16.4.250   centos-7-55   <none>           <none>
  36. nginx-deploy-5b9b5b66bc-njft8   1/1     Running   0          29s   172.16.3.154   centos-7-54   <none>           <none>
  • 案例四
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-deploy
  5. spec:
  6.   replicas: 5
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx-demo
  17.         image: linuxduduniao/nginx:v1.0.0
  18.       affinity:
  19.         nodeAffinity:
  20.           preferredDuringSchedulingIgnoredDuringExecution:
  21.           # 满足该条件的节点为 centos-7-56,当并不是一把梭全部压在该节点，而且该节点的优先级更高，通过weight可以增加优先级
  22.           - preference:
  23.               matchExpressions:
  24.               - key: ssd
  25.                 operator: DoesNotExist
  26.               - key: cpu
  27.                 operator: In
  28.                 values:
  29.                 - high
  30.             weight: 10
  31. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  32. NAME                            READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
  33. nginx-deploy-5996df99f8-4cmfv   1/1     Running   0          7s    172.16.4.4     centos-7-55   <none>           <none>
  34. nginx-deploy-5996df99f8-77k5l   1/1     Running   0          7s    172.16.5.38    centos-7-56   <none>           <none>
  35. nginx-deploy-5996df99f8-8kxvc   1/1     Running   0          7s    172.16.5.37    centos-7-56   <none>           <none>
  36. nginx-deploy-5996df99f8-t55hj   1/1     Running   0          7s    172.16.3.161   centos-7-54   <none>           <none>
  37. nginx-deploy-5996df99f8-zbpf2   1/1     Running   0          7s    172.16.5.39    centos-7-56   <none>           <none>

8.4. Pod亲和性

pod亲和性和节点亲和性类似，也非为硬亲和软亲和

  • 案例一(硬亲和)
  1. [root@duduniao local-k8s-yaml]# kubectl label node centos-7-55 cpu=slow  # 此时三个节点都存在cpu标签，仅centos-7-55为cpu=slow其它为high
  2. [root@duduniao local-k8s-yaml]# kubectl get pod -l app=nginx -o wide # 为了效果明显，建nginx-deploy减少为1个副本，当前在cnetos-7-56上
  3. NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
  4. nginx-deploy-cfbbb7cbd-jb4xz   1/1     Running   0          13m   172.16.5.40   centos-7-56   <none>           <none>
  5. apiVersion: apps/v1
  6. kind: Deployment
  7. metadata:
  8.   name: slb-deploy
  9. spec:
  10.   replicas: 5
  11.   selector:
  12.     matchLabels:
  13.       app: slb
  14.   template:
  15.     metadata:
  16.       labels:
  17.         app: slb
  18.     spec:
  19.       containers:
  20.       - name: slb-demo
  21.         image: linuxduduniao/nginx:v1.0.1
  22.       affinity:
  23.         podAffinity:
  24.           requiredDuringSchedulingIgnoredDuringExecution:
  25.           # centos-7-54和centos-7-56都满足
  26.           - labelSelector: 
  27.               matchLabels:
  28.                 app: nginx
  29.             namespaces: [default]
  30.             topologyKey: cpu
  31. [root@duduniao local-k8s-yaml]# kubectl get pod -l app=slb -o wide # centos-7-55不满足 cpu=high，所以不会调度
  32. NAME                         READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
  33. slb-deploy-7d9b6c47d-drjls   1/1     Running   0          4m19s   172.16.3.165   centos-7-54   <none>           <none>
  34. slb-deploy-7d9b6c47d-dsw7t   1/1     Running   0          4m19s   172.16.5.43    centos-7-56   <none>           <none>
  35. slb-deploy-7d9b6c47d-fn6k2   1/1     Running   0          4m19s   172.16.5.44    centos-7-56   <none>           <none>
  36. slb-deploy-7d9b6c47d-kw9vh   1/1     Running   0          4m19s   172.16.3.166   centos-7-54   <none>           <none>
  37. slb-deploy-7d9b6c47d-pl6lz   1/1     Running   0          4m19s   172.16.5.45    centos-7-56   <none>           <none>
  • 案例二(软亲和)
  1. [root@duduniao local-k8s-yaml]# kubectl label node centos-7-54 cpu=slow --overwrite # 使得仅cnetos-7-56满足cpu=high
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5.   name: slb-deploy
  6. spec:
  7.   replicas: 5
  8.   selector:
  9.     matchLabels:
  10.       app: slb
  11.   template:
  12.     metadata:
  13.       labels:
  14.         app: slb
  15.     spec:
  16.       containers:
  17.       - name: slb-demo
  18.         image: linuxduduniao/nginx:v1.0.1
  19.       affinity:
  20.         podAffinity:
  21.           preferredDuringSchedulingIgnoredDuringExecution:
  22.           - podAffinityTerm:
  23.               labelSelector: 
  24.                 matchLabels:
  25.                   app: nginx
  26.               namespaces: [default]
  27.               topologyKey: cpu
  28.             weight: 10
  29. [root@duduniao local-k8s-yaml]# kubectl get pod -l app=slb -o wide # 大部分选择了cnetos-7-56，而不是全部梭哈
  30. NAME                          READY   STATUS    RESTARTS   AGE   IP             NODE          NOMINATED NODE   READINESS GATES
  31. slb-deploy-6b5fc7bd96-78cj4   1/1     Running   0          16s   172.16.5.48    centos-7-56   <none>           <none>
  32. slb-deploy-6b5fc7bd96-clchc   1/1     Running   0          18s   172.16.3.167   centos-7-54   <none>           <none>
  33. slb-deploy-6b5fc7bd96-fxwjf   1/1     Running   0          16s   172.16.5.47    centos-7-56   <none>           <none>
  34. slb-deploy-6b5fc7bd96-ls8zr   1/1     Running   0          18s   172.16.5.46    centos-7-56   <none>           <none>
  35. slb-deploy-6b5fc7bd96-zn9t8   1/1     Running   0          18s   172.16.4.5     centos-7-55   <none>           <none>

8.5. 污点和污点容忍度

  • Node节点上污点管理
  1. 用法: 
  2.   增加污点: kubectl taint node <node_name> key=value:effect
  3.   取消污点: kubectl taint node <node_name> key=value:effect-
  4.   查看污点: kubectl describe node <node_name>
  5. effect:
  6.     PreferNoSchedule: 优先不调度，但是其它节点不满足时可以调度
  7.     NoSchedule: 禁止新的Pod调度，已经调度的Pod不会被驱逐
  8.   NoExecute: 禁止新的Pod调度，并且已经运行在该节点时的，其不能容忍污点的Pod将被驱逐
  9. # 查看master的污点
  10. [root@duduniao local-k8s-yaml]# kubectl describe node centos-7-51 
  11. Name:               centos-7-51
  12. Roles:              master
  13. Labels:             beta.kubernetes.io/arch=amd64
  14.                     beta.kubernetes.io/os=linux
  15.                     kubernetes.io/arch=amd64
  16.                     kubernetes.io/hostname=centos-7-51
  17.                     kubernetes.io/os=linux
  18.                     node-role.kubernetes.io/master=
  19. Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"52:8a:0e:48:b4:92"}
  20.                     flannel.alpha.coreos.com/backend-type: vxlan
  21.                     flannel.alpha.coreos.com/kube-subnet-manager: true
  22.                     flannel.alpha.coreos.com/public-ip: 10.4.7.51
  23.                     kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
  24.                     node.alpha.kubernetes.io/ttl: 0
  25.                     volumes.kubernetes.io/controller-managed-attach-detach: true
  26. CreationTimestamp:  Fri, 04 Dec 2020 21:49:43 +0800
  27. Taints:             node-role.kubernetes.io/master:NoSchedule  # 不可调度
  28. ......
  29. # 以此作为示例Pod
  30. apiVersion: apps/v1
  31. kind: Deployment
  32. metadata:
  33.   name: nginx-deploy
  34. spec:
  35.   replicas: 6
  36.   selector:
  37.     matchLabels:
  38.       app: nginx
  39.   template:
  40.     metadata:
  41.       labels:
  42.         app: nginx
  43.     spec:
  44.       containers:
  45.       - name: nginx-demo
  46.         image: linuxduduniao/nginx:v1.0.0
  47. # kubectl apply 部署上述的deployment后，Pod分散在三个不同的node上
  48. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  49. NAME                            READY   STATUS    RESTARTS   AGE    IP             NODE          NOMINATED NODE   READINESS GATES
  50. nginx-deploy-8697d45cb8-4x564   1/1     Running   0          4m9s   172.16.4.9     centos-7-55   <none>           <none>
  51. nginx-deploy-8697d45cb8-bxms4   1/1     Running   0          4m9s   172.16.5.55    centos-7-56   <none>           <none>
  52. nginx-deploy-8697d45cb8-c4rbf   1/1     Running   0          4m9s   172.16.3.171   centos-7-54   <none>           <none>
  53. nginx-deploy-8697d45cb8-hvs92   1/1     Running   0          4m9s   172.16.3.172   centos-7-54   <none>           <none>
  54. nginx-deploy-8697d45cb8-sbfvj   1/1     Running   0          4m9s   172.16.5.54    centos-7-56   <none>           <none>
  55. nginx-deploy-8697d45cb8-sw5m4   1/1     Running   0          4m9s   172.16.4.10    centos-7-55   <none>           <none>
  56. # 使用NoSchedule污点，发现并不会使得现有的Pod发生重调度
  57. [root@duduniao local-k8s-yaml]# kubectl taint node centos-7-54 monitor=true:NoSchedule
  58. node/centos-7-54 tainted
  59. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  60. NAME                            READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE   READINESS GATES
  61. nginx-deploy-8697d45cb8-4x564   1/1     Running   0          5m18s   172.16.4.9     centos-7-55   <none>           <none>
  62. nginx-deploy-8697d45cb8-bxms4   1/1     Running   0          5m18s   172.16.5.55    centos-7-56   <none>           <none>
  63. nginx-deploy-8697d45cb8-c4rbf   1/1     Running   0          5m18s   172.16.3.171   centos-7-54   <none>           <none>
  64. nginx-deploy-8697d45cb8-hvs92   1/1     Running   0          5m18s   172.16.3.172   centos-7-54   <none>           <none>
  65. nginx-deploy-8697d45cb8-sbfvj   1/1     Running   0          5m18s   172.16.5.54    centos-7-56   <none>           <none>
  66. nginx-deploy-8697d45cb8-sw5m4   1/1     Running   0          5m18s   172.16.4.10    centos-7-55   <none>           <none>
  68. # 更新deployment后，新的Pod不再调到 NoSchedule 节点
  69. root@duduniao local-k8s-yaml]# kubectl set image deployment nginx-deploy nginx-demo=linuxduduniao/nginx:v1.0.1
  70. deployment.apps/nginx-deploy image updated
  71. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  72. NAME                            READY   STATUS    RESTARTS   AGE   IP            NODE          NOMINATED NODE   READINESS GATES
  73. nginx-deploy-8494c5b6c5-5ddd9   1/1     Running   0          8s    172.16.4.13   centos-7-55   <none>           <none>
  74. nginx-deploy-8494c5b6c5-5zdjg   1/1     Running   0          10s   172.16.5.56   centos-7-56   <none>           <none>
  75. nginx-deploy-8494c5b6c5-bqpgn   1/1     Running   0          10s   172.16.5.57   centos-7-56   <none>           <none>
  76. nginx-deploy-8494c5b6c5-dq44w   1/1     Running   0          9s    172.16.4.12   centos-7-55   <none>           <none>
  77. nginx-deploy-8494c5b6c5-pnvw6   1/1     Running   0          8s    172.16.5.58   centos-7-56   <none>           <none>
  78. nginx-deploy-8494c5b6c5-xnf77   1/1     Running   0          10s   172.16.4.11   centos-7-55   <none>           <none>
  79. # 设置NoExecute后，现有Pod会被驱逐
  80. [root@duduniao local-k8s-yaml]# kubectl taint node centos-7-55 monitor=true:NoExecute
  81. node/centos-7-55 tainted
  82. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  83. NAME                            READY   STATUS    RESTARTS   AGE    IP            NODE          NOMINATED NODE   READINESS GATES
  84. nginx-deploy-8494c5b6c5-5zdjg   1/1     Running   0          3m8s   172.16.5.56   centos-7-56   <none>           <none>
  85. nginx-deploy-8494c5b6c5-bqpgn   1/1     Running   0          3m8s   172.16.5.57   centos-7-56   <none>           <none>
  86. nginx-deploy-8494c5b6c5-bzc2c   1/1     Running   0          14s    172.16.5.60   centos-7-56   <none>           <none>
  87. nginx-deploy-8494c5b6c5-f7k2b   1/1     Running   0          14s    172.16.5.62   centos-7-56   <none>           <none>
  88. nginx-deploy-8494c5b6c5-pnvw6   1/1     Running   0          3m6s   172.16.5.58   centos-7-56   <none>           <none>
  89. nginx-deploy-8494c5b6c5-s57tv   1/1     Running   0          14s    172.16.5.61   centos-7-56   <none>           <none>
  • Pod的污点容忍度
  1. # api-server 能容忍所有NoExecute的污点，因此能在Master上运行
  2. [root@duduniao local-k8s-yaml]# kubectl describe pod -n kube-system kube-apiserver-centos-7-51  
  3. ......
  4. Tolerations:       :NoExecute
  5. # 清除所有节点的污点后，执行以下操作。模拟Prometheus(Prometheus占用内存巨大，推荐单独部署到一个固有节点)的部署
  6. [root@duduniao local-k8s-yaml]# kubectl label node centos-7-56 prometheus=true
  7. node/centos-7-56 labeled
  8. [root@duduniao local-k8s-yaml]# kubectl taint node centos-7-56 monitor=true:NoSchedule
  9. node/centos-7-56 tainted
  10. apiVersion: apps/v1
  11. kind: Deployment
  12. metadata:
  13.   name: prometheus
  14. spec:
  15.   replicas: 1
  16.   selector:
  17.     matchLabels:
  18.       app: prometheus
  19.   template:
  20.     metadata:
  21.       labels:
  22.         app: prometheus
  23.     spec:
  24.       containers:
  25.       - name: prometheus-demo
  26.         image: linuxduduniao/nginx:v1.0.1
  27.       nodeSelector:
  28.         prometheus: "true"
  29.       tolerations:
  30.       - key: monitor
  31.         operator: Exists
  32.         effect: NoSchedule
  33. # 通过节点选择器和污点容忍度，实现独占一个节点
  34. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  35. NAME                          READY   STATUS    RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
  36. prometheus-76f64854b7-sxdq9   1/1     Running   0          2m21s   172.16.5.71   centos-7-56   <none>           <none>

8.6. Pod驱逐

在升级节点、移除节点之前，需要将该节上的Pod进行驱逐，并且保证不会有新的Pod调度进来，通常使用 kubectl drain options

节点移除：https://www.yuque.com/duduniao/k8s/togtwi#HpAMj

节点维护：https://www.yuque.com/duduniao/k8s/togtwi#Q5Ao6