03-3-Pod控制器

1. Pod

1.1. Pod介绍

1.1.1. Pod简介

Pod 是 Kubernetes 的基本构建块,它是 Kubernetes 对象模型中创建或部署的最小和最简单的单元。 Pod 表示集群上正在运行的进程。Pod 封装了应用程序容器(或者在某些情况下封装多个容器)、存储资源、唯一网络 IP 以及控制容器应该如何运行的选项。 Pod 表示部署单元:Kubernetes 中应用程序的单个实例,它可能由单个容器或少量紧密耦合并共享资源的容器组成。

一个pod内部一般仅运行一个pod,也可以运行多个pod,如果存在多个pod时,其中一个为主容器,其它作为辅助容器,也被称为边车模式。同一个pod共享一个网络名称空间和外部存储卷。

03-8-K8S网络 - 图1

1.1.2. Pod生命周期

Pod的生命周期中可以经历多个阶段,在一个Pod中在主容器(Main Container)启动前可以由init container来完成一些初始化操作。初始化完毕后,init Container 退出,Main Container启动。

在主容器启动后可以执行一些特定的指令,称为启动后钩子(PostStart),在主容器退出前也可以执行一些特殊指令完成清理工作,称为结束前钩子(PreStop)。

在主容器工作周期内,并不是刚创建就能对外提供服务,容器内部可能需要加载相关配置,因此可以使用特定命令确定容器是否就绪,称为就绪性检测(ReadinessProbe),完成就绪性检测才能成为Ready状态。

主容器对外提供服务后,可能出现意外导致容器异常,虽然此时容器仍在运行,但是不具备对外提供业务的能力,因此需要对其做存活性探测(LivenessProbe)。

03-8-K8S网络 - 图2

1.1.3. Pod状态

  • Pending: Pod 已被 Kubernetes 系统接受,但有一个或者多个容器尚未创建。
  • Running: 该 Pod 已经绑定到了一个节点上,Pod 中所有的容器都已被创建。至少有一个容器正在运行,或者正处于启动或重启状态。
  • Succeeded: Pod 中的所有容器都被成功终止,并且不会再重启。
  • Failed: Pod 中的所有容器都已终止了,并且至少有一个容器是因为失败终止。
  • Unknown: 因为某些原因无法取得 Pod 的状态,通常是因为与 Pod 所在主机通信失败。

03-8-K8S网络 - 图3

1.2. Pod模板

1.2.1. apiversion/kind

  1. apiVersion: v1
  2. kind: Pod

1.2.2. metadata

  1. metadata
  2. name <string> # 在一个名称空间内不能重复
  3. namespace <string> # 指定名称空间,默认defalut
  4. labels <map[string]string> # 标签
  5. annotations <map[string]string> # 注释,不能作为被筛选

1.2.3. spec

  1. spec
  2. containers <[]Object> -required- # 必选参数
  3. name <string> -required- # 指定容器名称,不可更新
  4. image <string> -required- # 指定镜像
  5. imagePullPolicy <string> # 指定镜像拉取方式
  6. # Always: 始终从registory拉取镜像。如果镜像标签为latest,则默认值为Always
  7. # Never: 仅使用本地镜像
  8. # IfNotPresent: 本地不存在镜像时才去registory拉取。默认值
  9. env <[]Object> # 指定环境变量
  10. name <string> -required- # 变量名称
  11. value <string> # 变量值
  12. valueFrom <Object> # 从文件中读取,不常用
  13. command <[]string> # 以数组方式指定容器运行指令,替代docker的ENTRYPOINT指令
  14. args <[]string> # 以数组方式指定容器运行参数,替代docker的CMD指令
  15. workingDir <string> # 指定工作目录,不指定则使用镜像默认值
  16. ports <[]Object> # 指定容器暴露的端口
  17. containerPort <integer> -required- # 容器的监听端口
  18. name <string> # 为端口取名,该名称可以在service种被引用
  19. protocol <string> # 指定协议:UDP, TCP, SCTP,默认TCP
  20. hostIP <string> # 绑定到宿主机的某个IP
  21. hostPort <integer> # 绑定到宿主机的端口
  22. resources <Object> # 资源设置
  23. limits <map[string]string> # 消耗的最大资源限制,通常设置cpu和memory
  24. requests <map[string]string> # 最低资源要求,在scheduler中被用到,通常设置cpu和memory
  25. volumeMounts <[]Object> # 指定存储卷挂载
  26. name <string> -required- # 存储卷名称
  27. mountPath <string> -required- # 容器内挂载路径
  28. subPath <string> # 存储卷的子目录
  29. readOnly <boolean> # 是否为只读方式挂载
  30. volumeDevices <[]Object> # 配置块设备的挂载
  31. devicePath <string> -required- # 容器内挂载路径
  32. name <string> -required- # pvc名称
  33. readinessProbe <Object> # 就绪性探测,确认就绪后提供服务
  34. initialDelaySeconds <integer> # 容器启动后到开始就绪性探测中间的等待秒数
  35. periodSeconds <integer> # 两次探测的间隔多少秒,默认值为10
  36. successThreshold <integer> # 连续多少次检测成功认为容器正常,默认值为1。不支持修改
  37. failureThreshold <integer> # 连续多少次检测失败认为容器异常,默认值为3
  38. timeoutSeconds <integer> # 探测请求超时时间
  39. exec <Object> # 通过执行特定命令来探测容器健康状态
  40. command <[]string> # 执行命令,返回值为0表示健康,不自持shell模式
  41. tcpSocket <Object> # 检测TCP套接字
  42. host <string> # 指定检测地址,默认pod的IP
  43. port <string> -required- # 指定检测端口
  44. httpGet <Object> # 以HTTP请求方式检测
  45. host <string> # 指定检测地址,默认pod的IP
  46. httpHeaders <[]Object> # 设置请求头,很少会需要填写
  47. path <string> # 设置请求的location
  48. port <string> -required- # 指定检测端口
  49. scheme <string> # 指定协议,默认HTTP
  50. livenessProbe <Object> # 存活性探测,确认pod是否具备对外服务的能力,该对象中字段和readinessProbe一致
  51. lifecycle <Object> # 生命周期
  52. postStart <Object> # pod启动后钩子,执行指令或者检测失败则退出容器或者重启容器
  53. exec <Object> # 执行指令,参考readinessProbe.exec
  54. httpGet <Object> # 执行HTTP,参考readinessProbe.httpGet
  55. tcpSocket <Object> # 检测TCP套接字,参考readinessProbe.tcpSocket
  56. preStop <Object> # pod停止前钩子,停止前执行清理工作,该对象中字段和postStart一致
  57. startupProbe <Object> # 容器启动完毕的配置,该配置与readinessProbe一致,在lifecycle和Probe之前运行,失败则重启
  58. securityContext <Object> # 与容器安全相关的配置,如运行用户、特权模式等
  59. initContainers <[]Object> # 初始化容器,执行完毕会退出,用户数据迁移、文件拷贝等
  60. volumes <[]Object> # 存储卷配置,https://www.yuque.com/duduniao/k8s/vgms23#Ptdfs
  61. restartPolicy <string> # Pod重启策略,Always, OnFailure,Never,默认Always
  62. nodeName <string> # 调度到指定的node节点, 强制要求满足
  63. nodeSelector <map[string]string> # 指定预选的node节点, 强制要求满足
  64. affinity <Object> # 调度亲和性配置
  65. nodeAffinity <Object> # node亲和性配置
  66. preferredDuringSchedulingIgnoredDuringExecution <[]Object> # 首选配置
  67. preference <Object> -required- # 亲和偏好
  68. matchExpressions <[]Object> # 表达式匹配
  69. key <string> -required- # label的key
  70. values <[]string> # label的value,当operator为Exists和DoesNotExist时为空
  71. operator <string> -required- # key和value的连接符,In,NotIn,Exists,DoesNotExist,Gt,Lt
  72. matchFields <[]Object> # 字段匹配,与matchExpressions一致
  73. weight <integer> -required- # 权重
  74. requiredDuringSchedulingIgnoredDuringExecution <Object> # 强制要求的配置
  75. nodeSelectorTerms <[]Object> -required- # nodeselect配置,与preferredDuringSchedulingIgnoredDuringExecution.preference一致
  76. podAffinity <Object> # pod亲和性配置
  77. preferredDuringSchedulingIgnoredDuringExecution <[]Object> # 首选配置
  78. podAffinityTerm <Object> -required- # 选择器
  79. labelSelector <Object> # pod标签选择器
  80. matchExpressions<[]Object> # 表达式匹配
  81. matchLabels <map[string]string> # 标签匹配
  82. namespaces <[]string> # 对方Pod的namespace,为空时表示与当前Pod同一名称空间
  83. topologyKey <string> -required- # 与对方Pod亲和的Node上具备的label名称
  84. weight <integer> -required- # 权重
  85. requiredDuringSchedulingIgnoredDuringExecution <[]Object> # 强制配置,与requiredDuringSchedulingIgnoredDuringExecution.podAffinityTerm一致
  86. podAntiAffinity <Object> # Pod反亲和性配置,与podAffinity一致
  87. tolerations <[]Object> # 污点容忍配置
  88. key <string> # 污点的Key,为空表示所有污点的Key
  89. operator <string> # key和value之间的操作符,Exists,Equal。Exists时value为空,默认值 Equal
  90. value <string> # 污点的值
  91. effect <string> # 污点的影响行为,空表示容忍所有的行为
  92. tolerationSeconds <integer> # 当Pod被节点驱逐时,延迟多少秒
  93. hostname <string> # 指定pod主机名
  94. hostIPC <boolean> # 使用宿主机的IPC名称空间,默认false
  95. hostNetwork <boolean> # 使用宿主机的网络名称空间,默认false
  96. hostPID <boolean> # 使用宿主机的PID名称空间,默认false
  97. serviceAccountName <string> # Pod运行时的使用的serviceAccount
  98. imagePullSecrets <[]Object> # 当拉取私密仓库镜像时,需要指定的密码密钥信息
  99. name <string> # secrets 对象名

1.2.4. k8s和image中的命令

03-8-K8S网络 - 图4

1.2.4. 就绪性探测和存活性探测

  • 就绪性探测失败不会重启pod,只是让pod不处于ready状态。存活性探测失败会触发pod重启。
  • 就绪性探测和存活性探测会持续进行下去,直到pod终止。

1.3. 案例

一般不会单独创建pod,而是通过控制器的方式创建。

1.3.1. 创建简单pod

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. name: pod-demo
  5. namespace: app
  6. labels:
  7. app: centos7
  8. release: stable
  9. environment: dev
  10. spec:
  11. containers:
  12. - name: centos
  13. image: harbor.od.com/public/centos:7
  14. command:
  15. - /bin/bash
  16. - -c
  17. - "sleep 3600"
  1. [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/base_resource/pods/myapp.yaml
  2. [root@hdss7-21 ~]# kubectl get pod -o wide -n app
  3. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  4. pod-demo 1/1 Running 0 16s 172.7.22.2 hdss7-22.host.com <none> <none>
  5. [root@hdss7-21 ~]# kubectl exec pod-demo -n app -- ps uax
  6. USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
  7. root 1 0.0 0.0 4364 352 ? Ss 04:41 0:00 sleep 3600
  8. root 11 0.0 0.0 51752 1696 ? Rs 04:42 0:00 ps uax
  1. [root@hdss7-21 ~]# kubectl describe pod pod-demo -n app | tail
  2. Events:
  3. Type Reason Age From Message
  4. ---- ------ ---- ---- -------
  5. Normal Scheduled 3m46s default-scheduler Successfully assigned app/pod-demo to hdss7-22.host.com
  6. Normal Pulling 3m45s kubelet, hdss7-22.host.com Pulling image "harbor.od.com/public/centos:7"
  7. Normal Pulled 3m45s kubelet, hdss7-22.host.com Successfully pulled image "harbor.od.com/public/centos:7"
  8. Normal Created 3m45s kubelet, hdss7-22.host.com Created container centos
  9. Normal Started 3m45s kubelet, hdss7-22.host.com Started container centos

1.3.2. 带健康检测的pod

  1. apiVersion: v1
  2. kind: Pod
  3. metadata:
  4. name: pod-01
  5. namespace: app
  6. labels:
  7. app: centos7
  8. release: stable
  9. version: t1
  10. spec:
  11. containers:
  12. - name: centos
  13. image: harbor.od.com/public/centos:7
  14. command:
  15. - /bin/bash
  16. - -c
  17. - "echo 'abc' > /tmp/health;sleep 60;rm -f /tmp/health;sleep 600"
  18. livenessProbe:
  19. exec:
  20. command:
  21. - /bin/bash
  22. - -c
  23. - "[ -f /tmp/health ]"

2. Deployment

2.1. 介绍

2.1.1. 简介

Pod控制器有很多种,最初的是使用 ReplicationController,即副本控制器,用于控制pod数量。随着版本升级,出现了ReplicaSet,跟ReplicationController没有本质的不同,只是名字不一样,并且ReplicaSet支持集合式的selector。ReplicaSet的核心管理对象有三种:用户期望的副本数、标签选择器、pod模板。

ReplicaSet一般不会直接使用,而是采用Deployment,Deployment是用来管理Replicaset,ReplicaSet来管理Pod。Deployment为ReplicaSet 提供了一个声明式定义(declarative)方法,用来替代以前的 ReplicationController 来方便的管理应用,比ReplicaSet的功能更加强大,且包含了ReplicaSet的功能。Deployment支持以下功能:

  • 定义Deployment来创建Pod和ReplicaSet
  • 滚动升级和回滚应用
  • 扩容和缩容
  • 暂停部署功能和手动部署

2.1.2. 部署方式

  • 蓝绿发布

如图,假设副本数是5,目标是从v1升级到v2。先部署5个v2版本的业务机器,再将SLB的流量全部切换到v2上。如果出现异常,可以快速切换到v1版本。但是实际上用的不多,因为需要消耗大量的额外机器资源。

03-8-K8S网络 - 图5

  • 滚动发布

滚动发布是逐台(批次)升级,需要占用的额外资源少。比如先升级一台,再升级一台,直到全部升级完毕。也可以每次升级10%数量的机器,逐批次升级。

03-8-K8S网络 - 图6

  • 灰度发布(金丝雀发布)

灰度发布也叫金丝雀发布,起源是,矿井工人发现,金丝雀对瓦斯气体很敏感,矿工会在下井之前,先放一只金丝雀到井中,如果金丝雀不叫了,就代表瓦斯浓度高。

灰度发布会先升级一台灰度机器,将版本升级为v2,此时先经过测试验证,确认没有问题后。从LB引入少量流量进入灰度机器,运行一段时间后,再将其它机器升级为v2版本,引入全部流量。

03-8-K8S网络 - 图7

2.1.3. Deployment升级方案

Deployment的升级方案默认是滚动升级,支持升级暂停,支持指定最大超过预期pod数量,支持指定最小低于预期pod数量。可以实现上述三种部署方案(以目标预期pod数量5个,v1版本升级到v2版本为案例):

  • 蓝绿发布场景实现方案:新创建5个v2版本pod,等待5个v2版本Pod就绪后,下掉5个v1版本pod。
  • 灰度发布场景实现案例:新创建的第一个pod最为灰度pod,此时暂定升级,等待灰度成功后再升级v1版本Pod
  • 滚动发布:通过控制超出预期pod数量和低于预期Pod数量来控制滚动发布的节奏。

如下图,预期pod数量5个,滚动升级,最大超出副本数为2个,最大低于期望值2个的升级方式:

03-8-K8S网络 - 图8

2.2. 模板

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata
  4. name <string> # 在一个名称空间不能重复
  5. namespace <string> # 指定名称空间,默认defalut
  6. labels <map[string]string> # 标签
  7. annotations <map[string]string> # 注释
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata
  4. name <string> # 在一个名称空间不能重复
  5. namespace <string> # 指定名称空间,默认defalut
  6. labels <map[string]string> # 标签
  7. annotations <map[string]string> # 注释
  8. spec
  9. replicas <integer> # 期望副本数,默认值1
  10. selector <Object> # 标签选择器
  11. matchExpressions <[]Object> # 标签选择器的一种形式,多个条件使用AND连接
  12. key <string> -required- # 标签中的Key
  13. operator <string> -required- # 操作符,支持 In, NotIn, Exists, DoesNotExist
  14. values <[]string> # value的数组集合,当操作符为In或NotIn时不能为空
  15. matchLabels <map[string]string> # 使用key/value的格式做筛选
  16. strategy <Object> # pod更新策略,即如何替换已有的pod
  17. type <string> # 更新类型,支持 Recreate, RollingUpdate。默认RollingUpdate
  18. rollingUpdate <Object> # 滚动更新策略,仅在type为RollingUpdate时使用
  19. maxSurge <string> # 最大浪涌pod数,即滚动更新时最多可多于出期望值几个pod。支持数字和百分比格式
  20. maxUnavailable <string> # 最大缺失Pod数,即滚动更新时最多可少于期望值出几个pod。支持数字和百分比格式
  21. revisionHistoryLimit <integer> # 历史版本记录数,默认为最大值(2^32)
  22. template <Object> -required- # Pod模板,和Pod管理器yaml几乎格式一致
  23. metadata <Object> # Pod的metadata
  24. spec <Object> # Pod的spec

2.3. 案例

2.3.1. 创建deployment

  1. [root@hdss7-200 deployment]# vim /data/k8s-yaml/base_resource/deployment/nginx-v1.12.yaml
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5. name: nginx-deploy
  6. namespace: app
  7. spec:
  8. replicas: 5
  9. selector:
  10. matchLabels:
  11. app: nginx
  12. release: stable
  13. tier: slb
  14. partition: website
  15. strategy:
  16. rollingUpdate:
  17. maxSurge: 1
  18. maxUnavailable: 0
  19. template:
  20. metadata:
  21. labels:
  22. app: nginx
  23. release: stable
  24. tier: slb
  25. partition: website
  26. version: v1.12
  27. spec:
  28. containers:
  29. - name: nginx-pod
  30. image: harbor.od.com/public/nginx:v1.12
  31. lifecycle:
  32. postStart:
  33. exec:
  34. command:
  35. - /bin/bash
  36. - -c
  37. - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  38. readinessProbe:
  39. initialDelaySeconds: 5
  40. httpGet:
  41. port: 80
  42. path: /health.html
  43. livenessProbe:
  44. initialDelaySeconds: 10
  45. periodSeconds: 5
  46. httpGet:
  47. port: 80
  48. path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record
  2. [root@hdss7-21 ~]# kubectl get pods -n app -l partition=website # 查看
  3. NAME READY STATUS RESTARTS AGE
  4. nginx-deploy-5597c8b45-425ms 1/1 Running 0 5m12s
  5. nginx-deploy-5597c8b45-5p2rz 1/1 Running 0 9m34s
  6. nginx-deploy-5597c8b45-dw7hd 1/1 Running 0 9m34s
  7. nginx-deploy-5597c8b45-fg82k 1/1 Running 0 5m12s
  8. nginx-deploy-5597c8b45-sfxmg 1/1 Running 0 9m34s
  9. [root@hdss7-21 ~]# kubectl get rs -n app -l partition=website -o wide
  10. NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
  11. nginx-deploy-5597c8b45 8 8 8 10m nginx-pod harbor.od.com/public/nginx:v1.12 app=nginx,partition=website,pod-template-hash=5597c8b45,release=stable,tier=slb
  12. [root@hdss7-21 ~]# kubectl get deployment -n app -o wide
  13. NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
  14. nginx-deploy 8/8 8 8 11m nginx-pod harbor.od.com/public/nginx:v1.12 app=nginx,partition=website,release=stable,tier=slb

2.3.2. 模拟蓝绿发布

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. namespace: app
  6. spec:
  7. replicas: 5
  8. selector:
  9. matchLabels:
  10. app: nginx
  11. release: stable
  12. tier: slb
  13. partition: website
  14. strategy:
  15. rollingUpdate:
  16. # 最大浪涌数量为5
  17. maxSurge: 5
  18. maxUnavailable: 0
  19. template:
  20. metadata:
  21. labels:
  22. app: nginx
  23. release: stable
  24. tier: slb
  25. partition: website
  26. # 修改版本信息,用于查看当前版本
  27. version: v1.13
  28. spec:
  29. containers:
  30. - name: nginx-pod
  31. # 修改镜像
  32. image: harbor.od.com/public/nginx:v1.13
  33. lifecycle:
  34. postStart:
  35. exec:
  36. command:
  37. - /bin/bash
  38. - -c
  39. - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  40. readinessProbe:
  41. initialDelaySeconds: 5
  42. httpGet:
  43. port: 80
  44. path: /health.html
  45. livenessProbe:
  46. initialDelaySeconds: 10
  47. periodSeconds: 5
  48. httpGet:
  49. port: 80
  50. path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  2. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app
  3. REVISION CHANGE-CAUSE
  4. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  5. 2 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  6. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version # 多个ReplicaSet对应不同版本
  7. NAME DESIRED CURRENT READY AGE VERSION
  8. nginx-deploy-5597c8b45 0 0 0 10m v1.12
  9. nginx-deploy-6bd88df699 5 5 5 9m31s v1.13
  1. # 升级过程中的状态变化:
  2. [root@hdss7-21 ~]# kubectl rollout status deployment nginx-deploy -n app
  3. Waiting for deployment "nginx-deploy" rollout to finish: 5 old replicas are pending termination...
  4. Waiting for deployment "nginx-deploy" rollout to finish: 5 old replicas are pending termination...
  5. Waiting for deployment "nginx-deploy" rollout to finish: 5 old replicas are pending termination...
  6. Waiting for deployment "nginx-deploy" rollout to finish: 4 old replicas are pending termination...
  7. Waiting for deployment "nginx-deploy" rollout to finish: 4 old replicas are pending termination...
  8. Waiting for deployment "nginx-deploy" rollout to finish: 4 old replicas are pending termination...
  9. Waiting for deployment "nginx-deploy" rollout to finish: 3 old replicas are pending termination...
  10. Waiting for deployment "nginx-deploy" rollout to finish: 3 old replicas are pending termination...
  11. Waiting for deployment "nginx-deploy" rollout to finish: 3 old replicas are pending termination...
  12. Waiting for deployment "nginx-deploy" rollout to finish: 2 old replicas are pending termination...
  13. Waiting for deployment "nginx-deploy" rollout to finish: 2 old replicas are pending termination...
  14. Waiting for deployment "nginx-deploy" rollout to finish: 2 old replicas are pending termination...
  15. Waiting for deployment "nginx-deploy" rollout to finish: 1 old replicas are pending termination...
  16. Waiting for deployment "nginx-deploy" rollout to finish: 1 old replicas are pending termination...
  17. deployment "nginx-deploy" successfully rolled out
  18. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w
  19. NAME READY STATUS RESTARTS AGE VERSION
  20. nginx-deploy-5597c8b45-t5plt 1/1 Running 0 19s v1.12
  21. nginx-deploy-5597c8b45-tcq69 1/1 Running 0 19s v1.12
  22. nginx-deploy-5597c8b45-vdjxg 1/1 Running 0 19s v1.12
  23. nginx-deploy-5597c8b45-vqn9x 1/1 Running 0 19s v1.12
  24. nginx-deploy-5597c8b45-zl6qr 1/1 Running 0 19s v1.12
  25. ---- 立刻创建5个新版本podPending调度中
  26. nginx-deploy-6bd88df699-242fr 0/1 Pending 0 0s v1.13
  27. nginx-deploy-6bd88df699-242fr 0/1 Pending 0 0s v1.13
  28. nginx-deploy-6bd88df699-8pmdg 0/1 Pending 0 0s v1.13
  29. nginx-deploy-6bd88df699-4kj8z 0/1 Pending 0 0s v1.13
  30. nginx-deploy-6bd88df699-n7x6n 0/1 Pending 0 0s v1.13
  31. nginx-deploy-6bd88df699-8pmdg 0/1 Pending 0 0s v1.13
  32. nginx-deploy-6bd88df699-4kj8z 0/1 Pending 0 0s v1.13
  33. nginx-deploy-6bd88df699-8j85n 0/1 Pending 0 0s v1.13
  34. nginx-deploy-6bd88df699-n7x6n 0/1 Pending 0 0s v1.13
  35. nginx-deploy-6bd88df699-8j85n 0/1 Pending 0 0s v1.13
  36. ---- 创建pod
  37. nginx-deploy-6bd88df699-242fr 0/1 ContainerCreating 0 0s v1.13
  38. nginx-deploy-6bd88df699-8pmdg 0/1 ContainerCreating 0 0s v1.13
  39. nginx-deploy-6bd88df699-4kj8z 0/1 ContainerCreating 0 0s v1.13
  40. nginx-deploy-6bd88df699-n7x6n 0/1 ContainerCreating 0 0s v1.13
  41. nginx-deploy-6bd88df699-8j85n 0/1 ContainerCreating 0 0s v1.13
  42. ---- 启动pod
  43. nginx-deploy-6bd88df699-242fr 0/1 Running 0 1s v1.13
  44. nginx-deploy-6bd88df699-8j85n 0/1 Running 0 1s v1.13
  45. nginx-deploy-6bd88df699-4kj8z 0/1 Running 0 1s v1.13
  46. nginx-deploy-6bd88df699-n7x6n 0/1 Running 0 1s v1.13
  47. nginx-deploy-6bd88df699-8pmdg 0/1 Running 0 1s v1.13
  48. ---- Pod逐个就绪,且替换旧版本的pod
  49. nginx-deploy-6bd88df699-242fr 1/1 Running 0 6s v1.13
  50. nginx-deploy-5597c8b45-t5plt 1/1 Terminating 0 50s v1.12
  51. nginx-deploy-6bd88df699-8j85n 1/1 Running 0 7s v1.13
  52. nginx-deploy-5597c8b45-vdjxg 1/1 Terminating 0 51s v1.12
  53. nginx-deploy-5597c8b45-t5plt 0/1 Terminating 0 51s v1.12
  54. nginx-deploy-5597c8b45-t5plt 0/1 Terminating 0 51s v1.12
  55. nginx-deploy-6bd88df699-4kj8z 1/1 Running 0 7s v1.13
  56. nginx-deploy-5597c8b45-zl6qr 1/1 Terminating 0 51s v1.12
  57. nginx-deploy-5597c8b45-vdjxg 0/1 Terminating 0 52s v1.12
  58. nginx-deploy-5597c8b45-vdjxg 0/1 Terminating 0 52s v1.12
  59. nginx-deploy-5597c8b45-zl6qr 0/1 Terminating 0 53s v1.12
  60. nginx-deploy-5597c8b45-t5plt 0/1 Terminating 0 54s v1.12
  61. nginx-deploy-5597c8b45-t5plt 0/1 Terminating 0 54s v1.12
  62. nginx-deploy-5597c8b45-zl6qr 0/1 Terminating 0 56s v1.12
  63. nginx-deploy-5597c8b45-zl6qr 0/1 Terminating 0 56s v1.12
  64. nginx-deploy-6bd88df699-n7x6n 1/1 Running 0 13s v1.13
  65. nginx-deploy-5597c8b45-tcq69 1/1 Terminating 0 57s v1.12
  66. nginx-deploy-5597c8b45-tcq69 0/1 Terminating 0 58s v1.12
  67. nginx-deploy-5597c8b45-tcq69 0/1 Terminating 0 59s v1.12
  68. nginx-deploy-6bd88df699-8pmdg 1/1 Running 0 15s v1.13
  69. nginx-deploy-5597c8b45-vqn9x 1/1 Terminating 0 59s v1.12
  70. nginx-deploy-5597c8b45-vqn9x 0/1 Terminating 0 60s v1.12
  71. nginx-deploy-5597c8b45-vqn9x 0/1 Terminating 0 61s v1.12
  72. nginx-deploy-5597c8b45-vqn9x 0/1 Terminating 0 61s v1.12
  73. nginx-deploy-5597c8b45-vdjxg 0/1 Terminating 0 64s v1.12
  74. nginx-deploy-5597c8b45-vdjxg 0/1 Terminating 0 64s v1.12
  75. nginx-deploy-5597c8b45-tcq69 0/1 Terminating 0 64s v1.12
  76. nginx-deploy-5597c8b45-tcq69 0/1 Terminating 0 64s v1.12

2.3.3. 滚动发布

通过定义 maxsurge 和 maxUnavailable 来实现滚动升级的速度,滚动升级中,可以使用 kubectl rollout pause 来实现暂停。

  1. [root@hdss7-200 deployment]# vim /data/k8s-yaml/base_resource/deployment/nginx-v1.14.yaml
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5. name: nginx-deploy
  6. namespace: app
  7. spec:
  8. replicas: 5
  9. selector:
  10. matchLabels:
  11. app: nginx
  12. release: stable
  13. tier: slb
  14. partition: website
  15. strategy:
  16. rollingUpdate:
  17. # 以下两项,控制升级速度
  18. maxSurge: 1
  19. maxUnavailable: 0
  20. template:
  21. metadata:
  22. labels:
  23. app: nginx
  24. release: stable
  25. tier: slb
  26. partition: website
  27. # 修改版本
  28. version: v1.14
  29. spec:
  30. containers:
  31. - name: nginx-pod
  32. # 修改镜像版本
  33. image: harbor.od.com/public/nginx:v1.14
  34. lifecycle:
  35. postStart:
  36. exec:
  37. command:
  38. - /bin/bash
  39. - -c
  40. - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  41. readinessProbe:
  42. initialDelaySeconds: 5
  43. httpGet:
  44. port: 80
  45. path: /health.html
  46. livenessProbe:
  47. initialDelaySeconds: 10
  48. periodSeconds: 5
  49. httpGet:
  50. port: 80
  51. path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  2. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version # replicaset 数量增加
  3. NAME DESIRED CURRENT READY AGE VERSION
  4. nginx-deploy-5597c8b45 0 0 0 155m v1.12
  5. nginx-deploy-6bd88df699 0 0 0 154m v1.13
  6. nginx-deploy-7c5976dcd9 5 5 5 83s v1.14
  7. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app # 升级记录
  8. REVISION CHANGE-CAUSE
  9. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  10. 2 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  11. 3 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  1. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w # 逐个滚动升级
  2. NAME READY STATUS RESTARTS AGE VERSION
  3. nginx-deploy-6bd88df699-242fr 1/1 Running 0 152m v1.13
  4. nginx-deploy-6bd88df699-4kj8z 1/1 Running 0 152m v1.13
  5. nginx-deploy-6bd88df699-8j85n 1/1 Running 0 152m v1.13
  6. nginx-deploy-6bd88df699-8pmdg 1/1 Running 0 152m v1.13
  7. nginx-deploy-6bd88df699-n7x6n 1/1 Running 0 152m v1.13
  8. nginx-deploy-7c5976dcd9-ttlqx 0/1 Pending 0 0s v1.14
  9. nginx-deploy-7c5976dcd9-ttlqx 0/1 Pending 0 0s v1.14
  10. nginx-deploy-7c5976dcd9-ttlqx 0/1 ContainerCreating 0 0s v1.14
  11. nginx-deploy-7c5976dcd9-ttlqx 0/1 Running 0 1s v1.14
  12. nginx-deploy-7c5976dcd9-ttlqx 1/1 Running 0 9s v1.14
  13. nginx-deploy-6bd88df699-8pmdg 1/1 Terminating 0 153m v1.13
  14. ......

2.3.4. 模拟灰度(金丝雀)发布

灰度发布在不同场景中实现方式不同,如果当前灰度机器仅对测试开放,可以定义一个新的deployment来配合service来实现。如果需要切入一部分随机真实用户的流量,可以将生产机器中一台机器作为灰度机器,通过灰度后再升级其它的机器。

  1. # nginx-v1.15.yaml 与 nginx-v1.14.yaml 一致,仅仅修改了镜像文件
  2. [root@hdss7-21 ~]# kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true && kubectl rollout pause deployment nginx-deploy -n app
  3. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app
  4. REVISION CHANGE-CAUSE
  5. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  6. 2 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  7. 3 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  8. 4 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true
  9. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version # 存在两个ReplicaSet对外提供服务
  10. NAME DESIRED CURRENT READY AGE VERSION
  11. nginx-deploy-5597c8b45 0 0 0 177m v1.12
  12. nginx-deploy-6695fd9655 1 1 1 2m22s v1.15
  13. nginx-deploy-6bd88df699 0 0 0 176m v1.13
  14. nginx-deploy-7c5976dcd9 5 5 5 23m v1.14
  15. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w # 新老共存
  16. NAME READY STATUS RESTARTS AGE VERSION
  17. nginx-deploy-6695fd9655-tcm76 1/1 Running 0 17s v1.15
  18. nginx-deploy-7c5976dcd9-4tnv4 1/1 Running 0 21m v1.14
  19. nginx-deploy-7c5976dcd9-bpjc2 1/1 Running 0 20m v1.14
  20. nginx-deploy-7c5976dcd9-gv8qm 1/1 Running 0 20m v1.14
  21. nginx-deploy-7c5976dcd9-ttlqx 1/1 Running 0 21m v1.14
  22. nginx-deploy-7c5976dcd9-xq2qs 1/1 Running 0 21m v1.14
  1. # 手动暂停
  2. [root@hdss7-21 ~]# kubectl rollout resume deployment nginx-deploy -n app && kubectl rollout pause deployment nginx-deploy -n app
  3. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version -w
  4. NAME READY STATUS RESTARTS AGE VERSION
  5. nginx-deploy-6695fd9655-jmb94 1/1 Running 0 19s v1.15
  6. nginx-deploy-6695fd9655-tcm76 1/1 Running 0 6m19s v1.15
  7. nginx-deploy-7c5976dcd9-4tnv4 1/1 Running 0 27m v1.14
  8. nginx-deploy-7c5976dcd9-gv8qm 1/1 Running 0 26m v1.14
  9. nginx-deploy-7c5976dcd9-ttlqx 1/1 Running 0 27m v1.14
  10. nginx-deploy-7c5976dcd9-xq2qs 1/1 Running 0 27m v1.14
  11. # 升级剩余所有机器
  12. [root@hdss7-21 ~]# kubectl rollout resume deployment nginx-deploy -n app

2.3.5. 版本回滚

当升级出现异常时,执行回滚即可。

  1. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app # 查看历史版本记录
  2. deployment.extensions/nginx-deploy
  3. REVISION CHANGE-CAUSE
  4. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  5. 2 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  6. 3 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  7. 4 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true
  8. [root@hdss7-21 ~]# kubectl rollout undo deployment nginx-deploy -n app
  9. [root@hdss7-21 ~]# kubectl rollout history deployment nginx-deploy -n app # 版本3已经被版本5替代
  10. deployment.extensions/nginx-deploy
  11. REVISION CHANGE-CAUSE
  12. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.12.yaml --record=true
  13. 2 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.13.yaml --record=true
  14. 4 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true
  15. 5 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.14.yaml --record=true
  16. [root@hdss7-21 ~]# kubectl get pod -n app -l partition=website -L version
  17. NAME READY STATUS RESTARTS AGE VERSION
  18. nginx-deploy-7c5976dcd9-2kps8 1/1 Running 0 2m20s v1.14
  19. nginx-deploy-7c5976dcd9-bqs28 1/1 Running 0 2m6s v1.14
  20. nginx-deploy-7c5976dcd9-jdvps 1/1 Running 0 2m13s v1.14
  21. nginx-deploy-7c5976dcd9-vs8l4 1/1 Running 0 116s v1.14
  22. nginx-deploy-7c5976dcd9-z99mb 1/1 Running 0 101s v1.14
  23. [root@hdss7-21 ~]# kubectl get rs -n app -l tier=slb -L version
  24. NAME DESIRED CURRENT READY AGE VERSION
  25. nginx-deploy-5597c8b45 0 0 0 3h7m v1.12
  26. nginx-deploy-6695fd9655 0 0 0 12m v1.15
  27. nginx-deploy-6bd88df699 0 0 0 3h7m v1.13
  28. nginx-deploy-7c5976dcd9 5 5 5 34m v1.14

2.3.6. 常用命令

  1. kubectl rollout status deployment nginx-deploy -n app # 查看版本升级过程
  2. kubectl rollout history deployment nginx-deploy -n app # 查看版本升级历史
  3. kubectl apply -f http://k8s-yaml.od.com/base_resource/deployment/nginx-v1.15.yaml --record=true # 升级且记录升级命令
  4. kubectl rollout undo deployment nginx-deploy -n app # 回滚到上个版本
  5. kubectl rollout undo deployment nginx-deploy --to-revision=3 -n app # 回滚到版本3

3. DaemonSet

3.1. DaemonSet介绍

DaemonSet 确保全部(或者一些)Node 上运行一个 Pod 的副本。当有 Node 加入集群时,也会为他们新增一个 Pod 。当有 Node 从集群移除时,这些 Pod 也会被回收。删除 DaemonSet 将会删除它创建的所有 Pod。使用 DaemonSet 的一些典型用法:

  • 运行集群存储 daemon,例如在每个 Node 上运行 glusterd、ceph。
  • 在每个 Node 上运行日志收集 daemon,例如fluentd、logstash。
  • 在每个 Node 上运行监控 daemon,例如 Prometheus Node Exporter。

3.2. 模板

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata
  4. name <string> # 在一个名称空间不能重复
  5. namespace <string> # 指定名称空间,默认defalut
  6. labels <map[string]string> # 标签
  7. annotations <map[string]string> # 注释
  8. spec
  9. selector <Object> # 标签选择器
  10. matchExpressions <[]Object> # 标签选择器的一种形式,多个条件使用AND连接
  11. key <string> -required- # 标签中的Key
  12. operator <string> -required- # 操作符,支持 In, NotIn, Exists, DoesNotExist
  13. values <[]string> # value的数组集合,当操作符为In或NotIn时不能为空
  14. matchLabels <map[string]string> # 使用key/value的格式做筛选
  15. updateStrategy <Object> # 更新策略
  16. type <string> # 更新类型,支持 Recreate, RollingUpdate。默认RollingUpdate
  17. rollingUpdate <Object> # 滚动更新策略,仅在type为RollingUpdate时使用
  18. maxUnavailable <string> # 最大缺失Pod数,即滚动更新时最多可少于期望值出几个pod。支持数字和百分比格式
  19. template <Object> -required- # Pod模板,和Pod管理器yaml几乎格式一致
  20. metadata <Object> # Pod的metadata
  21. spec <Object> # Pod的spec

3.3. 案例

3.3.1. 创建daemonset

  1. [root@hdss7-200 base_resource]# cat /data/k8s-yaml/base_resource/daemonset/proxy-v1.12.yaml
  2. apiVersion: apps/v1
  3. kind: DaemonSet
  4. metadata:
  5. name: proxy-daemonset
  6. namespace: app
  7. labels:
  8. app: nginx
  9. release: stable
  10. partition: CRM
  11. spec:
  12. selector:
  13. matchLabels:
  14. app: nginx
  15. release: stable
  16. tier: proxy
  17. partition: CRM
  18. updateStrategy:
  19. rollingUpdate:
  20. maxUnavailable: 1
  21. template:
  22. metadata:
  23. labels:
  24. app: nginx
  25. release: stable
  26. tier: proxy
  27. partition: CRM
  28. version: v1.12
  29. spec:
  30. containers:
  31. - name: nginx-proxy
  32. image: harbor.od.com/public/nginx:v1.12
  33. ports:
  34. - name: http
  35. containerPort: 80
  36. hostPort: 10080
  37. lifecycle:
  38. postStart:
  39. exec:
  40. command:
  41. - /bin/bash
  42. - -c
  43. - "echo 'health check ok!' > /usr/share/nginx/html/health.html"
  44. readinessProbe:
  45. initialDelaySeconds: 5
  46. httpGet:
  47. port: 80
  48. path: /health.html
  49. livenessProbe:
  50. initialDelaySeconds: 10
  51. periodSeconds: 5
  52. httpGet:
  53. port: 80
  54. path: /health.html
  1. [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.12.yaml --record
  2. [root@hdss7-21 ~]# kubectl get daemonset -n app
  3. NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  4. proxy-daemonset 2 2 2 2 2 <none> 56s
  1. [root@hdss7-21 ~]# kubectl get pod -n app -l tier=proxy -o wide
  2. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  3. proxy-daemonset-7stgs 1/1 Running 0 8m31s 172.7.22.9 hdss7-22.host.com <none> <none>
  4. proxy-daemonset-dxgdp 1/1 Running 0 8m31s 172.7.21.10 hdss7-21.host.com <none> <none>
  5. [root@hdss7-21 ~]# curl -s 10.4.7.22:10080/info # 通过宿主机的端口访问
  6. 2020-01-22T13:15:58+00:00|172.7.22.9|nginx:v1.12
  7. [root@hdss7-21 ~]# curl -s 10.4.7.21:10080/info
  8. 2020-01-22T13:16:05+00:00|172.7.21.10|nginx:v1.12

3.3.2. 升级daemonset

daemonset的升级方式和deployment一致

  1. [root@hdss7-21 ~]# kubectl rollout history daemonset proxy-daemonset -n app
  2. REVISION CHANGE-CAUSE
  3. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.12.yaml --record=true
  4. [root@hdss7-21 ~]# kubectl apply -f http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.13.yaml --record
  5. [root@hdss7-21 ~]# kubectl rollout history daemonset proxy-daemonset -n app
  6. daemonset.extensions/proxy-daemonset
  7. REVISION CHANGE-CAUSE
  8. 1 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.12.yaml --record=true
  9. 2 kubectl apply --filename=http://k8s-yaml.od.com/base_resource/daemonset/proxy-v1.13.yaml --record=true
  10. [root@hdss7-21 ~]# kubectl get pod -n app -l tier=proxy -L version
  11. NAME READY STATUS RESTARTS AGE VERSION
  12. proxy-daemonset-7wr4f 1/1 Running 0 119s v1.13
  13. proxy-daemonset-clhqk 1/1 Running 0 2m11s v1.13

4. Job

4.1. Job介绍

Job也是一种很常用的控制器,在执行自动化脚本的时候就可以使用,比如使用ansible脚本job创建一个新的k8s集群,使用job创建MySQL账号密码,使用job去执行helm脚本等等。

4.2. 模板

Job的spec中没有selector!

  1. apiVersion: batch/v1
  2. kind: Job
  3. metadata
  4. name <string> # 在一个名称空间不能重复
  5. namespace <string> # 指定名称空间,默认defalut
  6. labels <map[string]string> # 标签
  7. annotations <map[string]string> # 注释
  8. spec
  9. backoffLimit <integer> # 重试次数,默认6
  10. completions <integer> # 指定当前job要执行多少次pod
  11. parallelism <integer> # 指定多个pod执行时的并发数
  12. template <Object> -required- # Pod模板,和Pod管理器yaml几乎格式一致
  13. metadata <Object> # Pod的metadata
  14. spec <Object> # Pod的spec

4.3. 案例

  1. [root@centos-7-51 ~]# cat /tmp/job.yaml
  2. apiVersion: batch/v1
  3. kind: Job
  4. metadata:
  5. name: test-job
  6. namespace: default
  7. spec:
  8. template:
  9. metadata:
  10. labels:
  11. jobType: sleep
  12. spec:
  13. restartPolicy: Never
  14. containers:
  15. - name: job-sleep
  16. image: busybox:latest
  17. command:
  18. - sleep
  19. - "60"
  20. [root@centos-7-51 ~]# kubectl apply -f /tmp/job.yaml
  21. [root@centos-7-51 ~]# kubectl get pod -o wide| grep job
  22. test-job-2fccp 0/1 Completed 0 3m21s 172.16.4.43 centos-7-55 <none> <none>
  23. [root@centos-7-51 ~]# kubectl get job -o wide
  24. NAME COMPLETIONS DURATION AGE CONTAINERS IMAGES SELECTOR
  25. test-job 1/1 68s 3m26s job-sleep busybox:latest controller-uid=3ba88fdd-ad1c-46c7-86b4-dc6e641ebdb4

5. CronJob

5.1. cronjob介绍

cronJob 是类似于Linux中的计划任务,用于执行周期性任务,如数据备份、周期性采集相关信息并发送邮件等。

https://kubernetes.io/zh/docs/tasks/job/automated-tasks-with-cron-jobs/

5.2. 模板

  1. apiVersion: batch/v1beta1
  2. kind: batch/v1beta1
  3. metadata <Object>
  4. name <string> # 在一个名称空间不能重复
  5. namespace <string> # 指定名称空间,默认defalut
  6. labels <map[string]string> # 标签
  7. annotations <map[string]string> # 注释
  8. spec <Object>
  9. concurrencyPolicy <string> # 新任务执行时上个任务还未结束时如何处理,Allow(默认值):允许新老任务并发执行;Forbid:忽略新任务;Replace:停止老任务,运行新任务
  10. failedJobsHistoryLimit <integer> # 失败历史记录保留次数,默认3
  11. successfulJobsHistoryLimit <integer> # 成功历史记录保留次数,默认1
  12. schedule <string> -required- # 计划任务的cron,与Linux的一致
  13. startingDeadlineSeconds <integer> # 如果任务错过调度的时间超过秒数表示任务失败,超过100次不再调度,参考 https://kubernetes.io/zh/docs/tasks/job/automated-tasks-with-cron-jobs/#starting-deadline
  14. suspend <boolean> # https://kubernetes.io/zh/docs/tasks/job/automated-tasks-with-cron-jobs/#%E6%8C%82%E8%B5%B7
  15. jobTemplate <Object> -required- # job 的任务模板
  16. metadata <Object>
  17. spec <Object>

5.3. 案例

  1. apiVersion: batch/v1beta1
  2. kind: CronJob
  3. metadata:
  4. name: hello
  5. spec:
  6. schedule: "*/1 * * * *"
  7. concurrencyPolicy: Forbid
  8. successfulJobsHistoryLimit: 3
  9. failedJobsHistoryLimit: 10
  10. jobTemplate:
  11. spec:
  12. template:
  13. spec:
  14. containers:
  15. - name: hello
  16. image: busybox
  17. imagePullPolicy: IfNotPresent
  18. args:
  19. - /bin/sh
  20. - -c
  21. - date; echo Hello from the Kubernetes cluster
  22. restartPolicy: OnFailure
  23. [root@centos-7-52 ~]# kubectl apply -f /tmp/cronjob.yaml
  24. [root@centos-7-52 ~]# kubectl get cronjob -o wide #查看cronjob
  25. NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE CONTAINERS IMAGES SELECTOR
  26. hello */1 * * * * False 0 52s 6m36s hello busybox <none>
  27. [root@centos-7-52 ~]# kubectl get job -o wide | grep hello # 查看执行的job历史,Job名称中的数字为时间戳
  28. hello-1607757060 1/1 1s 2m48s hello busybox controller-uid=194350a4-21ca-4d01-b264-34a963e78ce7
  29. hello-1607757120 1/1 1s 108s hello busybox controller-uid=e30a318c-6c0f-4d1d-b200-3dfaffe083c5
  30. hello-1607757180 1/1 1s 48s hello busybox controller-uid=a00d02fc-8d7e-40b3-8e2a-25620ecab427
  31. [root@centos-7-52 ~]# date -d @1607757180
  32. Sat Dec 12 15:13:00 CST 2020

6. StatefulSet

6.1. 介绍

Deployment 是用来发布和管理无状态服务的,所有节点之间是相同的,他们挂载相同的共享存储,且Pod名称后缀也是随机生成的,异常后可以随时重启。而有状态应用程序,他们的Pod之间是有区别的,启动也是有顺序要求,而且往往需要挂载不同的存储卷。由于有状态的应用程序本身比较复杂,尤其是发生故障重启时可能需要人为干预,因此复杂的有状态程序不建议放到K8s集群中,如MySQL集群。

K8S中StatefulSet 是用来管理有状态应用的工作负载 API 对象,StatefulSet 用来管理某 Pod 集合的部署和扩缩, 并为这些 Pod 提供持久存储和持久标识符。和 Deployment 类似, StatefulSet 管理基于相同容器规约的一组 Pod。但和 Deployment 不同的是, StatefulSet 为它们的每个 Pod 维护了一个有粘性的 ID。这些 Pod 是基于相同的规约来创建的, 但是不能相互替换:无论怎么调度,每个 Pod 都有一个永久不变的 ID。如果希望使用存储卷为工作负载提供持久存储,可以使用 StatefulSet 作为解决方案的一部分。 尽管 StatefulSet 中的单个 Pod 仍可能出现故障, 但持久的 Pod 标识符使得将现有卷与替换已失败 Pod 的新 Pod 相匹配变得更加容易。

6.1.1. Statefulset使用场景

  • 稳定的、唯一的网络标识符,即每个Pod有自己独有的域名,无论重启多少次都不会变
  • 稳定的、持久的存储,即每个Pod有自己独享的PVC
  • 有序的、优雅的部署和缩放,即每个Pod会按照序号从小到大启动,按照需要从大到小被停止
  • 有序的、自动的滚动更新,即每个Pod按照需要从大到小进行重启升级

6.1.2. Statefulset注意项

  • 需要定义pvc申请模板,每个pod使用不同的pvc。可以通过storageclass分配或者提前创建好pv
  • 需要一个headless的service作为服务发现
  • 如果需要删除statefulset,推荐将副本数置为0再删除
  • 删除statefulset,并不会删除申请的pvc和pv
  • 升级过程中,如果出现问题,可能需要人为干预

6.2. 模板

  1. apiVersion: apps/v1
  2. kind: StatefulSet
  3. metadata <Object>
  4. spec:
  5. replicas <integer>
  6. serviceName <string> -required-
  7. selector <Object> -required- # Pod选择器
  8. template <Object> -required- # Pod模板
  9. volumeClaimTemplates <[]Object>
  10. metadata <Object>
  11. spec <Object>
  12. accessModes <[]string> # RWO:单路读写;ROX:单路只读;RWX:多路读写
  13. resources <Object> # 指定当前PVC需要的系统最小资源限制
  14. limits <map[string]string> # 资源限制,一般不配置
  15. requests <map[string]string> # 资源限制,常用为 storage: xGi
  16. storageClassName <string> # 声明使用的存储类,用于动态分配
  17. volumeMode <string> # 使用卷的文件系统还是当成块设备来用,一般不配置
  18. selector <Object> # 标签选择器,选择PV的标签,默认在所有PV中寻找
  19. volumeName <string> # 指定PV名称,直接绑定PV
  20. revisionHistoryLimit <integer> # 保留历史版本数量,默认10
  21. podManagementPolicy <string> # 控制Pod启停顺序,默认OrderedReady,启动顺序,停止倒序。Parallel表示并行
  22. updateStrategy <Object> # 更新策略
  23. rollingUpdate <Object> # 滚动更新策略
  24. partition <integer> # 指定更新到哪个分区,N表示更新序号大于等于N的Pod
  25. type <string> # 更新策略,默认滚动更新RollingUpdate.

6.3. 案例

6.3.1. 创建Statefulset资源

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5. name: my-svc
  6. namespace: apps
  7. spec:
  8. selector:
  9. app: nginx-web
  10. role: web
  11. state: "true"
  12. clusterIP: None
  13. ports:
  14. - name: http
  15. port: 80
  16. targetPort: 80
  17. ---
  18. apiVersion: apps/v1
  19. kind: StatefulSet
  20. metadata:
  21. name: nginx-web
  22. namespace: apps
  23. labels:
  24. app: nginx
  25. role: web
  26. spec:
  27. replicas: 2
  28. serviceName: my-svc
  29. selector:
  30. matchLabels:
  31. app: nginx-web
  32. role: web
  33. state: "true"
  34. template:
  35. metadata:
  36. labels:
  37. app: nginx-web
  38. role: web
  39. state: "true"
  40. spec:
  41. containers:
  42. - name: nginx-web
  43. image: linuxduduniao/nginx:v1.0.0
  44. ports:
  45. - name: http
  46. containerPort: 80
  47. readinessProbe:
  48. httpGet:
  49. port: 80
  50. path: /health
  51. volumeMounts:
  52. - name: nginx-web
  53. mountPath: /usr/share/nginx/html
  54. volumeClaimTemplates:
  55. - metadata:
  56. name: nginx-web
  57. namespace: apps
  58. spec:
  59. accessModes: ["ReadWriteOnce"]
  60. resources:
  61. requests:
  62. storage: 2Gi
  63. storageClassName: managed-nfs-storage # nfs classstorage,参考https://www.yuque.com/duduniao/k8s/vgms23#3W9oz
  64. [root@duduniao local-k8s-yaml]# kubectl -n apps get sts -o wide # 当前sts 状态
  65. NAME READY AGE CONTAINERS IMAGES
  66. nginx-web 2/2 19s nginx-web linuxduduniao/nginx:v1.0.0
  67. [root@duduniao local-k8s-yaml]# kubectl -n apps get pod -o wide # 注意Pod名称
  68. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  69. nginx-web-0 1/1 Running 0 86s 172.16.4.158 centos-7-55 <none> <none>
  70. nginx-web-1 1/1 Running 0 77s 172.16.3.98 centos-7-54 <none> <none>
  71. [root@duduniao local-k8s-yaml]# kubectl -n apps get pvc -o wide # 注意pvc名称
  72. NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE VOLUMEMODE
  73. nginx-web-nginx-web-0 Bound pvc-b36b90f1-a268-4815-b55b-6f7fda587193 2Gi RWX managed-nfs-storage 93s Filesystem
  74. nginx-web-nginx-web-1 Bound pvc-1461c537-e784-41fd-9e78-b1e6b212912c 2Gi RWX managed-nfs-storage 84s Filesystem
  75. [root@duduniao local-k8s-yaml]# kubectl -n apps get pv -o wide # 注意pv名称
  76. NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE VOLUMEMODE
  77. pvc-1461c537-e784-41fd-9e78-b1e6b212912c 2Gi RWX Delete Bound apps/nginx-web-nginx-web-1 managed-nfs-storage 88s Filesystem
  78. pvc-b36b90f1-a268-4815-b55b-6f7fda587193 2Gi RWX Delete Bound apps/nginx-web-nginx-web-0 managed-nfs-storage 97s Filesystem
  79. [root@duduniao local-k8s-yaml]# kubectl -n apps describe svc my-svc
  80. Name: my-svc
  81. Namespace: apps
  82. Labels: <none>
  83. Annotations: Selector: app=nginx-web,role=web,state=true
  84. Type: ClusterIP
  85. IP: None
  86. Port: http 80/TCP
  87. TargetPort: 80/TCP
  88. Endpoints: 172.16.3.98:80,172.16.4.158:80
  89. Session Affinity: None
  90. Events: <none>
  91. [root@centos-7-51 ~]# dig -t A my-svc.apps.svc.cluster.local @10.96.0.10 +short # 无头服务特性,用于服务发现
  92. 172.16.4.216
  93. 172.16.3.105
  94. [root@centos-7-51 ~]# dig -t A nginx-web-0.my-svc.apps.svc.cluster.local @10.96.0.10 +short # statefulset 特性,可以用pod名称解析到Pod的地址
  95. 172.16.4.216
  96. [root@centos-7-51 ~]# dig -t A nginx-web-1.my-svc.apps.svc.cluster.local @10.96.0.10 +short
  97. 172.16.3.105

6.3.2. 扩缩容

在默认的策略下,statefulset启停Pod都是按照顺序执行的,新建Pod按照序号从小到大进行,删除Pod从大到小进行,通常需要在Pod配置就绪性探针,严格保证次序。

  1. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":5}}' # 增加副本数
  2. [root@duduniao ~]# kubectl get pod -n apps -w # 观察启动顺序,前一个就绪后,再启动下一个
  3. NAME READY STATUS RESTARTS AGE
  4. nginx-web-0 1/1 Running 2 10h
  5. nginx-web-1 1/1 Running 2 10h
  6. nginx-web-2 0/1 Pending 0 0s
  7. nginx-web-2 0/1 Pending 0 0s
  8. nginx-web-2 0/1 Pending 0 1s
  9. nginx-web-2 0/1 ContainerCreating 0 2s
  10. nginx-web-2 0/1 Running 0 3s
  11. nginx-web-2 1/1 Running 0 5s
  12. nginx-web-3 0/1 Pending 0 0s
  13. nginx-web-3 0/1 Pending 0 0s
  14. nginx-web-3 0/1 Pending 0 1s
  15. nginx-web-3 0/1 ContainerCreating 0 2s
  16. nginx-web-3 0/1 Running 0 3s
  17. nginx-web-3 1/1 Running 0 9s
  18. nginx-web-4 0/1 Pending 0 0s
  19. nginx-web-4 0/1 Pending 0 0s
  20. nginx-web-4 0/1 Pending 0 1s
  21. nginx-web-4 0/1 ContainerCreating 0 2s
  22. nginx-web-4 0/1 Running 0 3s
  23. nginx-web-4 1/1 Running 0 7s
  24. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":2}}' # 减少副本数
  25. [root@duduniao ~]# kubectl get pod -n apps -w # 观察停止顺序,按序号倒序停止
  26. NAME READY STATUS RESTARTS AGE
  27. nginx-web-0 1/1 Running 2 10h
  28. nginx-web-1 1/1 Running 2 10h
  29. nginx-web-2 1/1 Running 0 117s
  30. nginx-web-3 1/1 Running 0 112s
  31. nginx-web-4 1/1 Running 0 103s
  32. nginx-web-4 1/1 Terminating 0 109s
  33. nginx-web-4 0/1 Terminating 0 110s
  34. nginx-web-4 0/1 Terminating 0 111s
  35. nginx-web-4 0/1 Terminating 0 111s
  36. nginx-web-4 0/1 Terminating 0 111s
  37. nginx-web-3 1/1 Terminating 0 2m
  38. nginx-web-3 0/1 Terminating 0 2m
  39. nginx-web-3 0/1 Terminating 0 2m1s
  40. nginx-web-3 0/1 Terminating 0 2m1s
  41. nginx-web-2 1/1 Terminating 0 2m7s
  42. nginx-web-2 0/1 Terminating 0 2m7s
  43. nginx-web-2 0/1 Terminating 0 2m13s
  44. nginx-web-2 0/1 Terminating 0 2m13s

6.3.3. 滚动更新

滚动更新是按照倒序方式进行,可以指定分区号,如指定分区号N,则表示更新序号大于等于N的Pod,默认分区为0,表示更新所有Pod

  1. [root@duduniao local-k8s-yaml]# kubectl -n apps set image sts nginx-web nginx-web=linuxduduniao/nginx:v1.0.1
  2. [root@duduniao ~]# kubectl get pod -n apps -w # 注意是倒序更新所有Pod
  3. NAME READY STATUS RESTARTS AGE
  4. nginx-web-0 1/1 Running 2 10h
  5. nginx-web-1 1/1 Running 2 10h
  6. nginx-web-1 1/1 Terminating 2 10h
  7. nginx-web-1 0/1 Terminating 2 10h
  8. nginx-web-1 0/1 Terminating 2 10h
  9. nginx-web-1 0/1 Terminating 2 10h
  10. nginx-web-1 0/1 Pending 0 0s
  11. nginx-web-1 0/1 Pending 0 0s
  12. nginx-web-1 0/1 ContainerCreating 0 0s
  13. nginx-web-1 0/1 Running 0 22s
  14. nginx-web-1 1/1 Running 0 27s
  15. nginx-web-0 1/1 Terminating 2 10h
  16. nginx-web-0 0/1 Terminating 2 10h
  17. nginx-web-0 0/1 Terminating 2 10h
  18. nginx-web-0 0/1 Terminating 2 10h
  19. nginx-web-0 0/1 Pending 0 0s
  20. nginx-web-0 0/1 Pending 0 0s
  21. nginx-web-0 0/1 ContainerCreating 0 0s
  22. nginx-web-0 0/1 ErrImagePull 0 29s
  23. nginx-web-0 0/1 ImagePullBackOff 0 44s
  24. nginx-web-0 0/1 Running 0 62s
  25. nginx-web-0 1/1 Running 0 70s
  26. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":5}}' # 扩容到5个副本,方便演示
  27. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":3}}}}' # 指定更新分区为3
  28. [root@duduniao local-k8s-yaml]# kubectl -n apps set image sts nginx-web nginx-web=linuxduduniao/nginx:v1.0.2 # 开始更新
  29. [root@duduniao ~]# kubectl get pod -n apps -w # 只更新序号4和3的Pod
  30. NAME READY STATUS RESTARTS AGE
  31. nginx-web-0 1/1 Running 0 6m31s
  32. nginx-web-1 1/1 Running 0 7m12s
  33. nginx-web-2 1/1 Running 0 3m38s
  34. nginx-web-3 1/1 Running 0 3m33s
  35. nginx-web-4 1/1 Running 0 3m23s
  36. nginx-web-4 1/1 Terminating 0 3m25s
  37. nginx-web-4 0/1 Terminating 0 3m26s
  38. nginx-web-4 0/1 Terminating 0 3m32s
  39. nginx-web-4 0/1 Terminating 0 3m32s
  40. nginx-web-4 0/1 Pending 0 0s
  41. nginx-web-4 0/1 Pending 0 0s
  42. nginx-web-4 0/1 ContainerCreating 0 0s
  43. nginx-web-4 0/1 Running 0 10s
  44. nginx-web-4 1/1 Running 0 17s
  45. nginx-web-3 1/1 Terminating 0 3m59s
  46. nginx-web-3 0/1 Terminating 0 4m
  47. nginx-web-3 0/1 Terminating 0 4m1s
  48. nginx-web-3 0/1 Terminating 0 4m1s
  49. nginx-web-3 0/1 Pending 0 0s
  50. nginx-web-3 0/1 Pending 0 0s
  51. nginx-web-3 0/1 ContainerCreating 0 0s
  52. nginx-web-3 0/1 Running 0 22s
  53. nginx-web-3 1/1 Running 0 25s
  54. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":0}}}}' # 修改分区为0
  55. [root@duduniao ~]# kubectl get pod -n apps -w # 会更新剩下来的三个Pod
  56. NAME READY STATUS RESTARTS AGE
  57. nginx-web-0 1/1 Running 0 9m10s
  58. nginx-web-1 1/1 Running 0 9m51s
  59. nginx-web-2 1/1 Running 0 6m17s
  60. nginx-web-3 1/1 Running 0 2m11s
  61. nginx-web-4 1/1 Running 0 2m30s
  62. nginx-web-2 1/1 Terminating 0 6m24s
  63. nginx-web-2 0/1 Terminating 0 6m24s
  64. nginx-web-2 0/1 Terminating 0 6m25s
  65. nginx-web-2 0/1 Terminating 0 6m25s
  66. nginx-web-2 0/1 Pending 0 0s
  67. nginx-web-2 0/1 Pending 0 0s
  68. nginx-web-2 0/1 ContainerCreating 0 0s
  69. nginx-web-2 0/1 Running 0 1s
  70. nginx-web-2 1/1 Running 0 9s
  71. nginx-web-1 1/1 Terminating 0 10m
  72. nginx-web-1 0/1 Terminating 0 10m
  73. nginx-web-1 0/1 Terminating 0 10m
  74. nginx-web-1 0/1 Terminating 0 10m
  75. nginx-web-1 0/1 Pending 0 0s
  76. nginx-web-1 0/1 Pending 0 0s
  77. nginx-web-1 0/1 ContainerCreating 0 0s
  78. nginx-web-1 0/1 Running 0 1s
  79. nginx-web-1 1/1 Running 0 8s
  80. nginx-web-0 1/1 Terminating 0 9m47s
  81. nginx-web-0 0/1 Terminating 0 9m48s
  82. nginx-web-0 0/1 Terminating 0 10m
  83. nginx-web-0 0/1 Terminating 0 10m
  84. nginx-web-0 0/1 Pending 0 0s
  85. nginx-web-0 0/1 Pending 0 0s
  86. nginx-web-0 0/1 ContainerCreating 0 0s
  87. nginx-web-0 0/1 Running 0 1s
  88. nginx-web-0 1/1 Running 0 9s

6.3.4. 删除Statefulset

与statefulset相关资源有三部分:statefulset、pv/pvc、svc,svc和pv/pvc需要手动删除。

通常删除statefulset时,推荐将其副本数置为0,实现有序停止,然后再删除statefulset,最后考虑是否需要删除svc和存储卷!

statefulset还提供了 --cascade=false 选项,表示不删除对应Pod,不过此项使用较少。

  1. [root@duduniao local-k8s-yaml]# kubectl -n apps patch sts nginx-web -p '{"spec":{"replicas":0}}' # 有序停止
  2. [root@duduniao local-k8s-yaml]# kubectl delete -n apps sts nginx-web # 删除statefulset
  3. [root@duduniao local-k8s-yaml]# kubectl delete svc -n apps my-svc # 删除svc
  4. [root@duduniao local-k8s-yaml]# kubectl get pv -n apps
  5. NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
  6. pvc-1461c537-e784-41fd-9e78-b1e6b212912c 2Gi RWX Delete Bound apps/nginx-web-nginx-web-1 managed-nfs-storage 11h
  7. pvc-277cff8f-c5e6-43fc-9498-ddc9ba788cb9 2Gi RWO Delete Bound apps/nginx-web-nginx-web-4 managed-nfs-storage 30m
  8. pvc-7d1a1169-6e34-4331-9549-f2f4d2bf6f94 2Gi RWO Delete Bound apps/nginx-web-nginx-web-3 managed-nfs-storage 30m
  9. pvc-b36b90f1-a268-4815-b55b-6f7fda587193 2Gi RWX Delete Bound apps/nginx-web-nginx-web-0 managed-nfs-storage 11h
  10. pvc-d804a299-95e1-486c-a9cd-73c925ed21d6 2Gi RWO Delete Bound apps/nginx-web-nginx-web-2 managed-nfs-storage 31m
  11. [root@duduniao local-k8s-yaml]# kubectl get pvc -n apps # pv和pvc即使删除Pod还是处于绑定状态,需要手动释放
  12. NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
  13. nginx-web-nginx-web-0 Bound pvc-b36b90f1-a268-4815-b55b-6f7fda587193 2Gi RWX managed-nfs-storage 11h
  14. nginx-web-nginx-web-1 Bound pvc-1461c537-e784-41fd-9e78-b1e6b212912c 2Gi RWX managed-nfs-storage 11h
  15. nginx-web-nginx-web-2 Bound pvc-d804a299-95e1-486c-a9cd-73c925ed21d6 2Gi RWO managed-nfs-storage 31m
  16. nginx-web-nginx-web-3 Bound pvc-7d1a1169-6e34-4331-9549-f2f4d2bf6f94 2Gi RWO managed-nfs-storage 31m
  17. nginx-web-nginx-web-4 Bound pvc-277cff8f-c5e6-43fc-9498-ddc9ba788cb9 2Gi RWO managed-nfs-storage 30m
  18. [root@duduniao local-k8s-yaml]# kubectl get pvc -n apps | awk '/nginx-web-nginx-web/{print "kubectl -n apps delete pvc", $1}'|bash

6.3.5. 部署有状态应用一般思路

有状态应用通常会组成集群,如MySQL。而且在启动之初还需要作初始化工作,如数据拷贝、数据导入等,通常需要initcontainer来实现。当出现故障时,如何顺利的进行故障转移也是需要考虑的,在实际操作中往往伴随着比较大的风险(相对于虚拟机部署而言)。一般性的建议是在对应官网找到kubernetes中部署方式,通常是helm脚本,通过配置合适的 values.yaml 来实现部署,但是即使如此,还是谨慎一些!

官方提供了两个示例:cassandra集群部署MySQL集群部署。这里再推荐一个helm chart项目,以供学习使用:charts

7. 原理部分

7.1. Deployment创建过程

kubectl apply -f deployment.yaml 为例,了解 Deployment 创建的过程。

  • Kubectl 生成 API 请求

    • Kubectl 校验 deployment.yaml 中的字段,确保资源清单的合法性
    • Kubectl 检查 deployment.yaml 中的API字段,填充kubeconfig的认证信息,填充 deployment 请求体信息。
    • 将封装好的请求发送到 ApiServer
  • ApiServer 身份认证

    • apiServer 通过认证信息确认用户合法身份
    • 通过RBAC鉴权,确认用户具备资源操作权限
    • 通过准入控制,确保满足Kubernetes部分高级功能,参考文档
    • 将HTTP请求反序列化,存入ECTD中
    • 执行资源初始化逻辑,如边车容器注入、证书注入等操作
  • ControllerManager创建资源对象

    • Deployment Controller 通过ApiServer发现有Deployment需要创建
    • Deployment Controller 通过ApiServer查询对应ReplicaSet是否满足,不满则则创建新的ReplicaSet,并设置版本号
    • ReplicaSet Controller 通过ApiServer发现有新的 ReplicaSet资源,通过查询是否有满足条件的Pod,如果Pod不满足条件,则创建Pod对象
  • Scheduler 调度资源

    • Scheduler 通过ApiServer监听创建新Pod的请求,一旦需要创建新Pod,则通过调度算法选择Node
    • 先通过一组预算策略进行评估,筛选可以调度的Node
    • 再通过优选策略,在符合条件的Node中选择得分最高的Node作为运行该Pod的节点,最高分有多个节点时,随机选择一个
    • Scheduler 创建一个Binding对象并且请求ApiServer,该对象包含了Pod的唯一信息和选择的Nodename
  • Kubelet 初始化Pod

之前所有的操作,都是在操作 etcd 数据库,只有 Kubelet 这一步才开始正式创建Pod

    • Kubelet 通过ApiServer查询到当前自身节点上Pod清单,与自己缓存中记录进行比对,如果是新的Pod则进入创建流程
    • Kubelet 生成Podstatus对象,填充Pod状态(Pending,Running,Succeeded,Failed,Unkown)
    • Kubelet 检查Pod是否具备对应的准入权限,如果不具备则处于Pending状态
    • 如果Pod存在资源限制,则通过Cgroups 对Pod进行资源限制
    • 创建容器目录,并挂载存储卷的目录
    • 检索ImagePullSecret,用于拉取镜像
    • 通过CRI 接接口调用容器runtime创建容器
  • CRI 创建容器

    • 创建Pause容器,用于为其他容器提供共享的网络名称空间和启动pid名称空间,从而实现网络共享和僵尸进程回收
    • docker runtime 调用CNI插件,从IP资源池中分配一个IP地址给当前Pause容器
    • 拉取业务镜像,如果填充了 imagepullsecret 会通过指定的密钥去拉取镜像
    • 挂载configmap到容器
    • 填充Pod元数据信息
    • 启动容器,监听容器的event,执行对应钩子

03-8-K8S网络 - 图9

8. 调度

8.1. 调度器

Kubernetes 中调度是指为Pod分配合适的工作节点,该过程是由Scheduler组件完成,调度结果写入 etcd 数据库中,交由kubelet组件去完成Pod启动。scheduler 调度过程分两个阶段:

  • 预选策略:过滤出可调度节点。根据Pod清单筛选处集群中所有满足Pod运行条件的节点,这些节点称为可调度节点。
  • 优选策略:打分并选择最高分。对可调度节点进行打分,选择最高峰的节点,如果存在多个最高分则随机选一个。

Scheduler 的策略可以通过kube-scheduler进行修改,但是一般很少操作。

scheduler 有一个配置 percentageOfNodesToScore ,值是0-100之间,0表示使用默认值,100以上等价于100。当该值为N,节点数为 M时,表示当可调度节点数量达到 N*M*0.01 时,停止执行预选函数,直接进入优选打分环节,这样可以避免轮询过多的节点影响性能。scheduler 默认计算方式:在 100-节点集群 下取 50%,在 5000-节点的集群下取 10%,这个参数默认的最低值是 5%,另外如果最小可调度节点数是50,这是程序中写死的,无法调整。从官网的描述来看,当节点数在几百个甚至更少的情况,调整percentageOfNodesToScore没有明显效果。

8.2. 节点选择器

  1. # 为了方面演示,增加到三个节点,效果更加明显
  2. [root@centos-7-51 ~]# kubectl get node
  3. NAME STATUS ROLES AGE VERSION
  4. centos-7-51 Ready master 10d v1.18.12
  5. centos-7-52 Ready master 10d v1.18.12
  6. centos-7-53 Ready master 10d v1.18.12
  7. centos-7-54 Ready worker 10d v1.18.12
  8. centos-7-55 Ready worker 10d v1.18.12
  9. centos-7-56 Ready worker 7m58s v1.18.12
  10. [root@centos-7-51 ~]# kubectl label node centos-7-54 ssd=true # 打上标签方便区分
  11. [root@centos-7-51 ~]# kubectl label node centos-7-55 ssd=true
  12. [root@centos-7-51 ~]# kubectl label node centos-7-54 cpu=high
  13. [root@centos-7-51 ~]# kubectl label node centos-7-56 cpu=high

节点选择器有两种,一种时直接指定nodeName,另一种是通过 nodeSelector 来根据标签选择:

  • 指定nodeName
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. spec:
  6. replicas: 5
  7. selector:
  8. matchLabels:
  9. app: nginx
  10. template:
  11. metadata:
  12. labels:
  13. app: nginx
  14. spec:
  15. containers:
  16. - name: nginx-demo
  17. image: linuxduduniao/nginx:v1.0.0
  18. nodeName: centos-7-56
  19. [root@centos-7-51 ~]# kubectl get pod -o wide # 全部调度到 centos-7-56 节点
  20. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  21. nginx-deploy-5648cd896-64ktx 1/1 Running 0 5m54s 172.16.5.11 centos-7-56 <none> <none>
  22. nginx-deploy-5648cd896-fgx75 1/1 Running 0 5m54s 172.16.5.13 centos-7-56 <none> <none>
  23. nginx-deploy-5648cd896-fvrlq 1/1 Running 0 5m54s 172.16.5.12 centos-7-56 <none> <none>
  24. nginx-deploy-5648cd896-hzljl 1/1 Running 0 5m54s 172.16.5.15 centos-7-56 <none> <none>
  25. nginx-deploy-5648cd896-qwrb5 1/1 Running 0 5m54s 172.16.5.14 centos-7-56 <none> <none>
  • 使用nodeSelector
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. spec:
  6. replicas: 5
  7. selector:
  8. matchLabels:
  9. app: nginx
  10. template:
  11. metadata:
  12. labels:
  13. app: nginx
  14. spec:
  15. containers:
  16. - name: nginx-demo
  17. image: linuxduduniao/nginx:v1.0.0
  18. nodeSelector:
  19. ssd: "true"
  20. cpu: high
  21. [root@centos-7-51 ~]# kubectl get pod -o wide # nodeSelector多个选项之间是 and 关系
  22. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  23. nginx-deploy-6d5b594bf5-b7s68 1/1 Running 0 14s 172.16.3.134 centos-7-54 <none> <none>
  24. nginx-deploy-6d5b594bf5-kv5kn 1/1 Running 0 14s 172.16.3.132 centos-7-54 <none> <none>
  25. nginx-deploy-6d5b594bf5-sxsgv 1/1 Running 0 11s 172.16.3.135 centos-7-54 <none> <none>
  26. nginx-deploy-6d5b594bf5-t2p8n 1/1 Running 0 11s 172.16.3.136 centos-7-54 <none> <none>
  27. nginx-deploy-6d5b594bf5-xrrhp 1/1 Running 0 14s 172.16.3.133 centos-7-54 <none> <none>

8.3. 节点亲和性

节点亲和性分为强制选择(硬亲和)和优先选择(软亲和):

  • 强制选择requiredDuringSchedulingIgnoredDuringExecution

    • 满足则调度(案例一)
    • 不满足则Pending(案例二)
  • 优先选择preferredDuringSchedulingIgnoredDuringExecution

    • 不满足则调度到其它节点(案例三)
    • 满足则优先调度到该节点,并不是一定调度到该节点,而且该节点优先级较高(案例四)
  • 案例一
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. spec:
  6. replicas: 5
  7. selector:
  8. matchLabels:
  9. app: nginx
  10. template:
  11. metadata:
  12. labels:
  13. app: nginx
  14. spec:
  15. containers:
  16. - name: nginx-demo
  17. image: linuxduduniao/nginx:v1.0.0
  18. affinity:
  19. nodeAffinity:
  20. requiredDuringSchedulingIgnoredDuringExecution:
  21. nodeSelectorTerms:
  22. - matchExpressions:
  23. - key: ssd
  24. operator: DoesNotExist
  25. - key: cpu
  26. operator: In
  27. values: ["high"]
  28. [root@duduniao ~]# kubectl get pod -o wide # 不存在ssd标签,并且cpu值为 high
  29. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  30. nginx-deploy-6f8b6d748c-4pt2x 1/1 Running 0 8s 172.16.5.20 centos-7-56 <none> <none>
  31. nginx-deploy-6f8b6d748c-m9kb4 1/1 Running 0 8s 172.16.5.19 centos-7-56 <none> <none>
  32. nginx-deploy-6f8b6d748c-st8mw 1/1 Running 0 5s 172.16.5.22 centos-7-56 <none> <none>
  33. nginx-deploy-6f8b6d748c-w4mc9 1/1 Running 0 5s 172.16.5.21 centos-7-56 <none> <none>
  34. nginx-deploy-6f8b6d748c-wjvxx 1/1 Running 0 8s 172.16.5.18 centos-7-56 <none> <none>
  • 案例二
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. spec:
  6. replicas: 5
  7. selector:
  8. matchLabels:
  9. app: nginx
  10. template:
  11. metadata:
  12. labels:
  13. app: nginx
  14. spec:
  15. containers:
  16. - name: nginx-demo
  17. image: linuxduduniao/nginx:v1.0.0
  18. affinity:
  19. nodeAffinity:
  20. requiredDuringSchedulingIgnoredDuringExecution:
  21. nodeSelectorTerms:
  22. - matchExpressions:
  23. - key: ssd
  24. operator: DoesNotExist
  25. # 此处改为cpu不等于high
  26. - key: cpu
  27. operator: NotIn
  28. values: ["high"]
  29. [root@duduniao ~]# kubectl describe pod nginx-deploy-746f88c86-96dbp # 注意message
  30. Name: nginx-deploy-746f88c86-96dbp
  31. Namespace: default
  32. Priority: 0
  33. Node: <none>
  34. ......
  35. Status: Pending
  36. ......
  37. Events:
  38. Type Reason Age From Message
  39. ---- ------ ---- ---- -------
  40. Warning FailedScheduling 30s (x3 over 32s) default-scheduler 0/6 nodes are available: 3 node(s) didn't match node selector, 3 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate.
  • 案例三
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. spec:
  6. replicas: 5
  7. selector:
  8. matchLabels:
  9. app: nginx
  10. template:
  11. metadata:
  12. labels:
  13. app: nginx
  14. spec:
  15. containers:
  16. - name: nginx-demo
  17. image: linuxduduniao/nginx:v1.0.0
  18. affinity:
  19. nodeAffinity:
  20. preferredDuringSchedulingIgnoredDuringExecution:
  21. - preference:
  22. matchExpressions:
  23. # 系统中并没有这个的Node
  24. - key: ssd
  25. operator: DoesNotExist
  26. - key: cpu
  27. operator: NotIn
  28. values: ["high"]
  29. weight: 5
  30. [root@duduniao ~]# kubectl get pod -o wide # 没有满足的节点,所以随机分布了
  31. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  32. nginx-deploy-5b9b5b66bc-5zrps 1/1 Running 0 31s 172.16.3.153 centos-7-54 <none> <none>
  33. nginx-deploy-5b9b5b66bc-brwgb 1/1 Running 0 31s 172.16.4.249 centos-7-55 <none> <none>
  34. nginx-deploy-5b9b5b66bc-f49j9 1/1 Running 0 31s 172.16.5.23 centos-7-56 <none> <none>
  35. nginx-deploy-5b9b5b66bc-jnh45 1/1 Running 0 30s 172.16.4.250 centos-7-55 <none> <none>
  36. nginx-deploy-5b9b5b66bc-njft8 1/1 Running 0 29s 172.16.3.154 centos-7-54 <none> <none>
  • 案例四
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4. name: nginx-deploy
  5. spec:
  6. replicas: 5
  7. selector:
  8. matchLabels:
  9. app: nginx
  10. template:
  11. metadata:
  12. labels:
  13. app: nginx
  14. spec:
  15. containers:
  16. - name: nginx-demo
  17. image: linuxduduniao/nginx:v1.0.0
  18. affinity:
  19. nodeAffinity:
  20. preferredDuringSchedulingIgnoredDuringExecution:
  21. # 满足该条件的节点为 centos-7-56,当并不是一把梭全部压在该节点,而且该节点的优先级更高,通过weight可以增加优先级
  22. - preference:
  23. matchExpressions:
  24. - key: ssd
  25. operator: DoesNotExist
  26. - key: cpu
  27. operator: In
  28. values:
  29. - high
  30. weight: 10
  31. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  32. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  33. nginx-deploy-5996df99f8-4cmfv 1/1 Running 0 7s 172.16.4.4 centos-7-55 <none> <none>
  34. nginx-deploy-5996df99f8-77k5l 1/1 Running 0 7s 172.16.5.38 centos-7-56 <none> <none>
  35. nginx-deploy-5996df99f8-8kxvc 1/1 Running 0 7s 172.16.5.37 centos-7-56 <none> <none>
  36. nginx-deploy-5996df99f8-t55hj 1/1 Running 0 7s 172.16.3.161 centos-7-54 <none> <none>
  37. nginx-deploy-5996df99f8-zbpf2 1/1 Running 0 7s 172.16.5.39 centos-7-56 <none> <none>

8.4. Pod亲和性

pod亲和性和节点亲和性类似,也非为硬亲和软亲和

  • 案例一(硬亲和)
  1. [root@duduniao local-k8s-yaml]# kubectl label node centos-7-55 cpu=slow # 此时三个节点都存在cpu标签,仅centos-7-55为cpu=slow其它为high
  2. [root@duduniao local-k8s-yaml]# kubectl get pod -l app=nginx -o wide # 为了效果明显,建nginx-deploy减少为1个副本,当前在cnetos-7-56上
  3. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  4. nginx-deploy-cfbbb7cbd-jb4xz 1/1 Running 0 13m 172.16.5.40 centos-7-56 <none> <none>
  5. apiVersion: apps/v1
  6. kind: Deployment
  7. metadata:
  8. name: slb-deploy
  9. spec:
  10. replicas: 5
  11. selector:
  12. matchLabels:
  13. app: slb
  14. template:
  15. metadata:
  16. labels:
  17. app: slb
  18. spec:
  19. containers:
  20. - name: slb-demo
  21. image: linuxduduniao/nginx:v1.0.1
  22. affinity:
  23. podAffinity:
  24. requiredDuringSchedulingIgnoredDuringExecution:
  25. # centos-7-54和centos-7-56都满足
  26. - labelSelector:
  27. matchLabels:
  28. app: nginx
  29. namespaces: [default]
  30. topologyKey: cpu
  31. [root@duduniao local-k8s-yaml]# kubectl get pod -l app=slb -o wide # centos-7-55不满足 cpu=high,所以不会调度
  32. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  33. slb-deploy-7d9b6c47d-drjls 1/1 Running 0 4m19s 172.16.3.165 centos-7-54 <none> <none>
  34. slb-deploy-7d9b6c47d-dsw7t 1/1 Running 0 4m19s 172.16.5.43 centos-7-56 <none> <none>
  35. slb-deploy-7d9b6c47d-fn6k2 1/1 Running 0 4m19s 172.16.5.44 centos-7-56 <none> <none>
  36. slb-deploy-7d9b6c47d-kw9vh 1/1 Running 0 4m19s 172.16.3.166 centos-7-54 <none> <none>
  37. slb-deploy-7d9b6c47d-pl6lz 1/1 Running 0 4m19s 172.16.5.45 centos-7-56 <none> <none>
  • 案例二(软亲和)
  1. [root@duduniao local-k8s-yaml]# kubectl label node centos-7-54 cpu=slow --overwrite # 使得仅cnetos-7-56满足cpu=high
  2. apiVersion: apps/v1
  3. kind: Deployment
  4. metadata:
  5. name: slb-deploy
  6. spec:
  7. replicas: 5
  8. selector:
  9. matchLabels:
  10. app: slb
  11. template:
  12. metadata:
  13. labels:
  14. app: slb
  15. spec:
  16. containers:
  17. - name: slb-demo
  18. image: linuxduduniao/nginx:v1.0.1
  19. affinity:
  20. podAffinity:
  21. preferredDuringSchedulingIgnoredDuringExecution:
  22. - podAffinityTerm:
  23. labelSelector:
  24. matchLabels:
  25. app: nginx
  26. namespaces: [default]
  27. topologyKey: cpu
  28. weight: 10
  29. [root@duduniao local-k8s-yaml]# kubectl get pod -l app=slb -o wide # 大部分选择了cnetos-7-56,而不是全部梭哈
  30. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  31. slb-deploy-6b5fc7bd96-78cj4 1/1 Running 0 16s 172.16.5.48 centos-7-56 <none> <none>
  32. slb-deploy-6b5fc7bd96-clchc 1/1 Running 0 18s 172.16.3.167 centos-7-54 <none> <none>
  33. slb-deploy-6b5fc7bd96-fxwjf 1/1 Running 0 16s 172.16.5.47 centos-7-56 <none> <none>
  34. slb-deploy-6b5fc7bd96-ls8zr 1/1 Running 0 18s 172.16.5.46 centos-7-56 <none> <none>
  35. slb-deploy-6b5fc7bd96-zn9t8 1/1 Running 0 18s 172.16.4.5 centos-7-55 <none> <none>

8.5. 污点和污点容忍度

  • Node节点上污点管理
  1. 用法:
  2. 增加污点: kubectl taint node <node_name> key=value:effect
  3. 取消污点: kubectl taint node <node_name> key=value:effect-
  4. 查看污点: kubectl describe node <node_name>
  5. effect:
  6. PreferNoSchedule: 优先不调度,但是其它节点不满足时可以调度
  7. NoSchedule: 禁止新的Pod调度,已经调度的Pod不会被驱逐
  8. NoExecute: 禁止新的Pod调度,并且已经运行在该节点时的,其不能容忍污点的Pod将被驱逐
  9. # 查看master的污点
  10. [root@duduniao local-k8s-yaml]# kubectl describe node centos-7-51
  11. Name: centos-7-51
  12. Roles: master
  13. Labels: beta.kubernetes.io/arch=amd64
  14. beta.kubernetes.io/os=linux
  15. kubernetes.io/arch=amd64
  16. kubernetes.io/hostname=centos-7-51
  17. kubernetes.io/os=linux
  18. node-role.kubernetes.io/master=
  19. Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"52:8a:0e:48:b4:92"}
  20. flannel.alpha.coreos.com/backend-type: vxlan
  21. flannel.alpha.coreos.com/kube-subnet-manager: true
  22. flannel.alpha.coreos.com/public-ip: 10.4.7.51
  23. kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
  24. node.alpha.kubernetes.io/ttl: 0
  25. volumes.kubernetes.io/controller-managed-attach-detach: true
  26. CreationTimestamp: Fri, 04 Dec 2020 21:49:43 +0800
  27. Taints: node-role.kubernetes.io/master:NoSchedule # 不可调度
  28. ......
  29. # 以此作为示例Pod
  30. apiVersion: apps/v1
  31. kind: Deployment
  32. metadata:
  33. name: nginx-deploy
  34. spec:
  35. replicas: 6
  36. selector:
  37. matchLabels:
  38. app: nginx
  39. template:
  40. metadata:
  41. labels:
  42. app: nginx
  43. spec:
  44. containers:
  45. - name: nginx-demo
  46. image: linuxduduniao/nginx:v1.0.0
  47. # kubectl apply 部署上述的deployment后,Pod分散在三个不同的node上
  48. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  49. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  50. nginx-deploy-8697d45cb8-4x564 1/1 Running 0 4m9s 172.16.4.9 centos-7-55 <none> <none>
  51. nginx-deploy-8697d45cb8-bxms4 1/1 Running 0 4m9s 172.16.5.55 centos-7-56 <none> <none>
  52. nginx-deploy-8697d45cb8-c4rbf 1/1 Running 0 4m9s 172.16.3.171 centos-7-54 <none> <none>
  53. nginx-deploy-8697d45cb8-hvs92 1/1 Running 0 4m9s 172.16.3.172 centos-7-54 <none> <none>
  54. nginx-deploy-8697d45cb8-sbfvj 1/1 Running 0 4m9s 172.16.5.54 centos-7-56 <none> <none>
  55. nginx-deploy-8697d45cb8-sw5m4 1/1 Running 0 4m9s 172.16.4.10 centos-7-55 <none> <none>
  56. # 使用NoSchedule污点,发现并不会使得现有的Pod发生重调度
  57. [root@duduniao local-k8s-yaml]# kubectl taint node centos-7-54 monitor=true:NoSchedule
  58. node/centos-7-54 tainted
  59. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  60. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  61. nginx-deploy-8697d45cb8-4x564 1/1 Running 0 5m18s 172.16.4.9 centos-7-55 <none> <none>
  62. nginx-deploy-8697d45cb8-bxms4 1/1 Running 0 5m18s 172.16.5.55 centos-7-56 <none> <none>
  63. nginx-deploy-8697d45cb8-c4rbf 1/1 Running 0 5m18s 172.16.3.171 centos-7-54 <none> <none>
  64. nginx-deploy-8697d45cb8-hvs92 1/1 Running 0 5m18s 172.16.3.172 centos-7-54 <none> <none>
  65. nginx-deploy-8697d45cb8-sbfvj 1/1 Running 0 5m18s 172.16.5.54 centos-7-56 <none> <none>
  66. nginx-deploy-8697d45cb8-sw5m4 1/1 Running 0 5m18s 172.16.4.10 centos-7-55 <none> <none>
  67. # 更新deployment后,新的Pod不再调到 NoSchedule 节点
  68. root@duduniao local-k8s-yaml]# kubectl set image deployment nginx-deploy nginx-demo=linuxduduniao/nginx:v1.0.1
  69. deployment.apps/nginx-deploy image updated
  70. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  71. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  72. nginx-deploy-8494c5b6c5-5ddd9 1/1 Running 0 8s 172.16.4.13 centos-7-55 <none> <none>
  73. nginx-deploy-8494c5b6c5-5zdjg 1/1 Running 0 10s 172.16.5.56 centos-7-56 <none> <none>
  74. nginx-deploy-8494c5b6c5-bqpgn 1/1 Running 0 10s 172.16.5.57 centos-7-56 <none> <none>
  75. nginx-deploy-8494c5b6c5-dq44w 1/1 Running 0 9s 172.16.4.12 centos-7-55 <none> <none>
  76. nginx-deploy-8494c5b6c5-pnvw6 1/1 Running 0 8s 172.16.5.58 centos-7-56 <none> <none>
  77. nginx-deploy-8494c5b6c5-xnf77 1/1 Running 0 10s 172.16.4.11 centos-7-55 <none> <none>
  78. # 设置NoExecute后,现有Pod会被驱逐
  79. [root@duduniao local-k8s-yaml]# kubectl taint node centos-7-55 monitor=true:NoExecute
  80. node/centos-7-55 tainted
  81. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  82. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  83. nginx-deploy-8494c5b6c5-5zdjg 1/1 Running 0 3m8s 172.16.5.56 centos-7-56 <none> <none>
  84. nginx-deploy-8494c5b6c5-bqpgn 1/1 Running 0 3m8s 172.16.5.57 centos-7-56 <none> <none>
  85. nginx-deploy-8494c5b6c5-bzc2c 1/1 Running 0 14s 172.16.5.60 centos-7-56 <none> <none>
  86. nginx-deploy-8494c5b6c5-f7k2b 1/1 Running 0 14s 172.16.5.62 centos-7-56 <none> <none>
  87. nginx-deploy-8494c5b6c5-pnvw6 1/1 Running 0 3m6s 172.16.5.58 centos-7-56 <none> <none>
  88. nginx-deploy-8494c5b6c5-s57tv 1/1 Running 0 14s 172.16.5.61 centos-7-56 <none> <none>
  • Pod的污点容忍度
  1. # api-server 能容忍所有NoExecute的污点,因此能在Master上运行
  2. [root@duduniao local-k8s-yaml]# kubectl describe pod -n kube-system kube-apiserver-centos-7-51
  3. ......
  4. Tolerations: :NoExecute
  5. # 清除所有节点的污点后,执行以下操作。模拟Prometheus(Prometheus占用内存巨大,推荐单独部署到一个固有节点)的部署
  6. [root@duduniao local-k8s-yaml]# kubectl label node centos-7-56 prometheus=true
  7. node/centos-7-56 labeled
  8. [root@duduniao local-k8s-yaml]# kubectl taint node centos-7-56 monitor=true:NoSchedule
  9. node/centos-7-56 tainted
  10. apiVersion: apps/v1
  11. kind: Deployment
  12. metadata:
  13. name: prometheus
  14. spec:
  15. replicas: 1
  16. selector:
  17. matchLabels:
  18. app: prometheus
  19. template:
  20. metadata:
  21. labels:
  22. app: prometheus
  23. spec:
  24. containers:
  25. - name: prometheus-demo
  26. image: linuxduduniao/nginx:v1.0.1
  27. nodeSelector:
  28. prometheus: "true"
  29. tolerations:
  30. - key: monitor
  31. operator: Exists
  32. effect: NoSchedule
  33. # 通过节点选择器和污点容忍度,实现独占一个节点
  34. [root@duduniao local-k8s-yaml]# kubectl get pod -o wide
  35. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
  36. prometheus-76f64854b7-sxdq9 1/1 Running 0 2m21s 172.16.5.71 centos-7-56 <none> <none>

8.6. Pod驱逐

在升级节点、移除节点之前,需要将该节上的Pod进行驱逐,并且保证不会有新的Pod调度进来,通常使用 kubectl drain options

节点移除:https://www.yuque.com/duduniao/k8s/togtwi#HpAMj

节点维护:https://www.yuque.com/duduniao/k8s/togtwi#Q5Ao6