kubeadm 安装官方文档

Kubeadm 高可用部署

Kubernetes 高可用部署实际上针对的是 apiServer 访问的高可用和 ETCD 集群的数据高可用。apiServer 高可用可以通过搭建多个 Master 节点，然后在 Kubernetes 集群前面增加一个负载均衡来实现。ETCD 的高可用
这里为了节省机器，将负载均衡和 ETCD 集群一同搭建在 Kubernetes Master 节点上了，生产中还是建议将这些服务部署在单独的服务器上。

1. 环境架构

1.1 角色划分

主机名 IP 角色
k8s-master01 10.0.0.10 K8s-master01、Keepalived、Haproxy、Etcd-node01
k8s-master02 10.0.0.20 K8s-master02、Keepalived、Haproxy、Etcd-node02
k8s-master03 10.0.0.30 K8s-master03、Keepalived、Haproxy、Etcd-node03
k8s-node01 10.0.0.101 K8s-node01
k8s-node02 10.0.0.102 K8s-node02
虚拟IP 10.0.0.200 Keepalived提供

1.2 拓扑图

kubeadm高可用架构.png

2. 操作系统初始化

2.1 关闭防火墙

  1. systemctl disable --now firewalld

2.2 关闭 selinux

  1. setenforce 0
  2. sed -i 's/enforcing/disabled/' /etc/selinux/config

2.3 关闭 swap

  1. swapoff -a && sysctl -w vm.swappiness=0
  2. sed -ri 's/.*swap.*/#&/' /etc/fstab

2.4 修改主机名和 hosts

  1. hostnamectl set-hostname <hostname>
  3. cat >> /etc/hosts << EOF
  4. 10.0.0.10 k8s-master01
  5. 10.0.0.20 k8s-master02
  6. 10.0.0.30 k8s-master03
  7. 10.0.0.101 k8s-node01
  8. 10.0.0.102 k8s-node02
  9. 10.0.0.200 k8s-lb
  10. EOF

2.5 时间同步

  1. # 硬件时钟和系统时间同步
  2. hwclock -s
  4. # 修改时区
  5. ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  6. echo "Asia/Shanghai" > /etc/timezone
  8. # 安装ntp服务
  9. which ntpdate &>/dev/null || yum -y install ntp
  11. # 时间同步
  12. ntpdate ntp.aliyun.com
  14. # 添加时间同步计划任务
  15. echo "$((RANDOM%60)) $((RANDOM%24)) * * * /usr/sbin/ntpdate time1.aliyun.com" >> /var/spool/cron/root

2.6 配置 limit

  1. ulimit -SHn 65535
  3. # 注意：Ubuntu系统不支持*
  4. cat >> /etc/security/limits.conf <<EOF
  5. * soft nofile 65536
  6. * hard nofile 131072
  7. * soft nproc 65535
  8. * hard nproc 655350
  9. * soft memlock unlimited
  10. * hard memlock unlimited
  11. * soft core unlimited
  12. * hard core unlimited
  13. EOF

2.7 ssh 免密登录（可选）

  1. # 在k8s-master01上执行以下操作
  2. ssh-keygen
  3. ssh-copy-id 127.0.0.1
  5. scp -r .ssh k8s-master02:/root/
  6. scp -r .ssh k8s-master03:/root/
  7. scp -r .ssh k8s-node01:/root/
  8. scp -r .ssh k8s-node02:/root/

2.8 升级内核版本

CentOS7 需要升级内核至 4.18+，但也不要太新，太新可能会出现老机器无法启动的情况。
这里选择 5.4 版本，手动下载内核进行安装。
CentOS7 内核下载地址

  1. # 更新软件包并重启
  2. yum update -y --exclude=kernel* && reboot
  4. # 下载内核（下载速度较慢，建议提前下载好）
  5. wget https://elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-lt-5.4.180-1.el7.elrepo.x86_64.rpm
  6. wget https://elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-lt-devel-5.4.180-1.el7.elrepo.x86_64.rpm
  8. # 手动升级内核
  9. yum localinstall -y kernel-lt*
  11. # 设置默认启动内核
  12. grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
  13. grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
  15. # 查看默认内核版本
  16. grubby --default-kernel
  18. # 重启
  19. reboot
  21. # 查看内核版本
  22. uname -a

2.9 安装 ipvs 并加载到内核模块

在内核 4.19+ 版本 nf_conntrack_ipv4 已经改为 nf_conntrack，另外ip_vs_fo调度算法是内核 4.15 版本之后才有的，如果没有升级内核会报错。

  1. # 安装工具包
  2. yum install -y ipvsadm ipset sysstat conntrack libseccomp
  4. # 开机加载模块
  5. cat > /etc/modules-load.d/ipvs.conf << EOF
  6. ip_vs
  7. ip_vs_lc
  8. ip_vs_wlc
  9. ip_vs_rr
  10. ip_vs_wrr
  11. ip_vs_lblc
  12. ip_vs_lblcr
  13. ip_vs_dh
  14. ip_vs_sh
  15. ip_vs_fo
  16. ip_vs_nq
  17. ip_vs_sed
  18. ip_vs_ftp
  19. ip_vs_sh
  20. nf_conntrack
  21. ip_tables
  22. ip_set
  23. xt_set
  24. ipt_set
  25. ipt_rpfilter
  26. ipt_REJECT
  27. ipip
  28. EOF
  30. # 开机启动模块自动加载服务
  31. systemctl enable --now systemd-modules-load.service

2.10 修改内核参数

  1. cat  > /etc/sysctl.d/k8s.conf <<EOF
  2. net.ipv4.ip_forward = 1
  3. net.bridge.bridge-nf-call-iptables = 1
  4. net.bridge.bridge-nf-call-ip6tables = 1
  5. fs.may_detach_mounts = 1
  6. vm.overcommit_memory=1
  7. vm.panic_on_oom=0
  8. fs.inotify.max_user_watches=89100
  9. fs.file-max=52706963
  10. fs.nr_open=52706963
  11. net.netfilter.nf_conntrack_max=2310720
  13. net.ipv4.tcp_keepalive_time = 600
  14. net.ipv4.tcp_keepalive_probes = 3
  15. net.ipv4.tcp_keepalive_intvl =15
  16. net.ipv4.tcp_max_tw_buckets = 36000
  17. net.ipv4.tcp_tw_reuse = 1
  18. net.ipv4.tcp_max_orphans = 327680
  19. net.ipv4.tcp_orphan_retries = 3
  20. net.ipv4.tcp_syncookies = 1
  21. net.ipv4.tcp_max_syn_backlog = 16384
  22. net.ipv4.ip_conntrack_max = 65536
  23. net.ipv4.tcp_max_syn_backlog = 16384
  24. net.ipv4.tcp_timestamps = 0
  25. net.core.somaxconn = 16384
  26. EOF
  27. sysctl --system
  29. # 重启
  30. reboot

3. 安装 Docker

  1. # 添加 Docker repo 源
  2. wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  4. # 查看版本
  5. yum list --showduplicates docker-ce
  7. # 安装
  8. yum -y install docker-ce
  11. # 单独挂载一块盘到/var/lib/docker目录
  12. # 修改镜像源和使用systemd管理cgroup
  13. mkdir /etc/docker
  14. cat > /etc/docker/daemon.json <<EOF
  15. {
  16.   "exec-opts": ["native.cgroupdriver=systemd"],
  17.   "log-driver": "json-file",
  18.   "log-opts": {
  19.     "max-size": "100m"
  20.   }, 
  21.   "data-root": "/var/lib/docker",
  22.   "storage-driver": "overlay2",
  23.   "registry-mirrors": [
  24.         "https://mirror.ccs.tencentyun.com",
  25.         "https://docker.mirrors.ustc.edu.cn",
  26.         "https://registry.docker-cn.com"]
  27. }
  28. EOF
  31. # 启动docker
  32. systemctl daemon-reload
  33. systemctl enable --now docker

4. 安装 kubeadm，kubelet 和 kubectl

  1. # 添加阿里云镜像源
  2. cat > /etc/yum.repos.d/kubernetes.repo << EOF
  3. [kubernetes]
  4. name=Kubernetes
  5. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
  6. enabled=1
  7. gpgcheck=0
  8. repo_gpgcheck=0
  9. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  10. EOF
  12. # 查看版本
  13. yum list --showduplicates kubeadm
  15. # 安装
  16. yum -y install kubeadm-1.21* kubelet-1.21* kubectl-1.21*
  18. # 配置kubelet使用阿里云的pause镜像
  19. cat > /etc/sysconfig/kubelet <<EOF
  20. KUBELET_EXTRA_ARGS="--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5"
  21. EOF
  23. # 启动kubelet
  24. systemctl daemon-reload
  25. systemctl enable --now kubelet

5. Haproxy 部署

只在 Master 节点进行以下操作：

5.1 安装

  1. yum -y install haproxy

5.2 编辑配置文件

注意：所有 Master 节点配置文件相同。
vim /etc/haproxy/haproxy.cfg

  1. global
  2.     daemon
  3.     log         127.0.0.1 local2 err        # 全局rsyslog定义
  4.     chroot      /var/lib/haproxy
  5.     pidfile     /var/run/haproxy.pid
  6.     maxconn     20000                                                # 每个haproxy进程所接受的最大并发连接数
  7.     user        haproxy
  8.     group       haproxy
  9.     stats socket /var/lib/haproxy/stats
  10.     spread-checks 2                                                 # 后端server状态check随机提前或延迟百分比时间，建议2-5(20%-50%)之间
  12. defaults
  13.     log     global
  14.       mode  http                                            # 使用七层代理
  15.       option      httplog                                # 在日志中记录http请求、session信息等
  16.       option        dontlognull                        # 不要在日志中记录空连接
  17.     option    redispatch                        # 用于cookie保持的环境中。默认情况下，HAProxy会将其请求的后端服务器的serverID插入cookie中，以保证会话的session持久性。如果后端服务器出现故障，客户端的cookie是不会刷新的，这就会造成无法访问。此时，如果设置了此参数，就会将客户的请求强制定向到另外一台健康的后端服务器上，以保证服务正常。
  18.       timeout     connect 10s                        # haproxy和服务端建立连接的最大时长，设置为1秒就足够了。局域网内建立连接一般都是瞬间的
  19.       timeout     client  3m                        # 和客户端保持空闲连接的超时时长，高并发下可稍微短一点，可设置为10秒以尽快释放连接
  20.       timeout     server  1m                        # 和服务端保持空闲连接的超时时长，局域网内建立连接很快，所以尽量设置短一些，特别是并发时，如设置为1-3秒
  21.       timeout     http-request 15s            #  等待客户端发送完整请求的最大时长，应该设置较短些防止洪水攻击,优先级高于timeout client
  22.       timeout     http-keep-alive 15s        # 和客户端保持长连接的最大时长。优先级高于timeout
  24. frontend monitor-in
  25.       bind *:33305
  26.       mode http
  27.       option httplog
  28.       monitor-uri /monitor
  30. frontend k8s-master
  31.       bind 0.0.0.0:16443
  32.       bind 127.0.0.1:16443
  33.       mode tcp
  34.       option tcplog
  35.       tcp-request inspect-delay 5s
  36.       default_backend k8s-master
  38. backend k8s-master
  39.       mode tcp
  40.       option tcplog
  41.       option tcp-check
  42.     balance roundrobin
  43.       default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  44.       server k8s-master01    10.0.0.10:6443 check
  45.       server k8s-master02    10.0.0.20:6443 check
  46.          server k8s-master03    10.0.0.30:6443 check

5.3 启动服务

  1. systemctl enable --now haproxy.service

6. Keepalived 部署

只在 Master 节点进行以下操作：

6.1 编译安装

官方网站

  1. # 安装依赖包
  2. yum install -y gcc openssl-devel libnl3-devel net-snmp-devel
  4. # 下载
  5. wget https://www.keepalived.org/software/keepalived-2.2.7.tar.gz
  7. # 解压
  8. tar xvf keepalived-2.2.7.tar.gz -C /usr/local/src
  10. # 编译安装
  11. cd /usr/local/src/keepalived-2.2.7/
  12. ./configure --prefix=/usr/local/keepalived
  13. make && make install
  15. # 创建/etc/keepalived目录
  16. mkdir /etc/keepalived

6.2 编辑配置文件

vim /etc/keepalived/keepalived.conf
注意：三个Master节点的配置文件稍有不同。

  • k8s-master01 配置文件： ```bash global_defs { script_user root # 脚本执行用户 router_id k8s-master01 # keepalived 主机唯一标识，建议使用当前主机名 vrrp_skip_check_adv_addr # 如果收到的通告报文和上一个报文是同一个路由，则跳过检查，默认为检查所有报文 enable_script_security # 如果脚本对任一非root用户来说具有可写权限，则不以root身份运行脚本 }

vrrp_script check_apiserver { script “/etc/keepalived/check_apiserver.sh” interval 5 # 检测间隔时间 weight -10 fall 2 rise 1 timeout 2 # 超时时间 }

vrrp_instance VIP_Nginx { state MASTER interface eth0 mcast_src_ip 10.0.0.10 # 发送多播包的地址，如果没有设置，默认使用绑定网卡的primary ip virtual_router_id 88 # 虚拟路由器惟一标识，同一个虚拟路由器的多个keepalived节点此值必须相同 priority 100 # Keepalived优先级 advert_int 2 authentication { # 使用普通密码认证 auth_type PASS auth_pass K8S_AUTH }

  1. virtual_ipaddress {
  2.             10.0.0.200                        # 虚拟IP
  3. }
  5. track_script {                        # 健康检查
  6.     check_apiserver
  7. }

}

  2. - k8s-master02 配置文件：
  3. ```bash
  4. global_defs {
  5.    script_user root                            # 脚本执行用户
  6.    router_id k8s-master02              # keepalived 主机唯一标识，建议使用当前主机名
  7.    vrrp_skip_check_adv_addr            # 如果收到的通告报文和上一个报文是同一个路由，则跳过检查，默认为检查所有报文
  8.    enable_script_security                # 如果脚本对任一非root用户来说具有可写权限，则不以root身份运行脚本
  9. }
  11. vrrp_script check_apiserver {
  12.     script "/etc/keepalived/check_apiserver.sh"
  13.     interval 5            # 检测间隔时间
  14.     weight -10
  15.     fall 2
  16.     rise 1
  17.     timeout 2                # 超时时间
  18. }
  20. vrrp_instance VIP_Nginx {
  21.     state BACKUP
  22.     interface eth0
  23.     mcast_src_ip 10.0.0.20        # 发送多播包的地址，如果没有设置，默认使用绑定网卡的primary ip
  24.     virtual_router_id 88            # 虚拟路由器惟一标识，同一个虚拟路由器的多个keepalived节点此值必须相同
  25.     priority 95                                # Keepalived优先级
  26.     advert_int 2
  27.     authentication {                    # 使用普通密码认证
  28.         auth_type PASS
  29.         auth_pass K8S_MASTER_AUTH
  30.     }
  32.     virtual_ipaddress {
  33.                 10.0.0.200                        # 虚拟IP
  34.     }
  36.     track_script {                        # 健康检查
  37.         check_apiserver
  38.     }
  39. }
  • k8s-master03配置文件： ```bash global_defs { script_user root # 脚本执行用户 router_id k8s-master03 # keepalived 主机唯一标识，建议使用当前主机名 vrrp_skip_check_adv_addr # 如果收到的通告报文和上一个报文是同一个路由，则跳过检查，默认为检查所有报文 enable_script_security # 如果脚本对任一非root用户来说具有可写权限，则不以root身份运行脚本 }

vrrp_script check_apiserver { script “/etc/keepalived/check_apiserver.sh” interval 5 # 检测间隔时间 weight -10 fall 2 rise 1 timeout 2 # 超时时间 }

vrrp_instance VIP_Nginx { state BACKUP interface eth0 mcast_src_ip 10.0.0.30 # 发送多播包的地址，如果没有设置，默认使用绑定网卡的primary ip virtual_router_id 88 # 虚拟路由器惟一标识，同一个虚拟路由器的多个keepalived节点此值必须相同 priority 95 # Keepalived优先级 advert_int 2 authentication { # 使用普通密码认证 auth_type PASS auth_pass K8S_MASTER_AUTH }

  1. virtual_ipaddress {
  2.             10.0.0.200                        # 虚拟IP
  3. }
  5. track_script {                        # 健康检查
  6.     check_apiserver
  7. }

}

  1. **最后在每台 Master 节点上添加 api-server 健康检查脚本：**<br />`vim /etc/keepalived/check_apiserver.sh`
  2. ```bash
  3. #!/bin/bash
  5. err=0
  6. for k in $(seq 1 3)
  7. do
  8.     check_code=$(pgrep haproxy)
  9.     if [[ $check_code == "" ]]; then
  10.         err=$(expr $err + 1)
  11.         sleep 2
  12.         continue
  13.     else
  14.         err=0
  15.         break
  16.     fi
  17. done
  19. if [[ $err != "0" ]]; then
  20.     echo "systemctl stop keepalived"
  21.     /usr/bin/systemctl stop keepalived
  22.     exit 1
  23. else
  24.     exit 0
  25. fi

添加执行权限：

  1. chmod +x /etc/keepalived/check_apiserver.sh

6.2 启动 Keepalived

  1. systemctl enable --now keepalived

7. Etcd 集群搭建

7.1 准备 cfssl 证书工具

Gitlab 项目地址

  1. # 下载工具包
  2. wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
  3. wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
  4. wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
  6. # 添加执行权限
  7. chmod +x cfssl*
  9. # 移动并重命名
  10. mv cfssl_linux-amd64 /usr/local/bin/cfssl
  11. mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
  12. mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

7.2 生成 Etcd 证书

创建证书目录：

  1. [root@k8s-master01 ~]# mkdir -p ~/tls/etcd

7.2.1 生成自签CA

  1. # 切换目录
  2. cd ~/tls/etcd
  4. # 创建证书配置文件
  5. cat > etcd-ca-config.json << EOF
  6. {
  7.     "signing": {
  8.         "default": {
  9.             "expiry": "87600h"
  10.         },
  11.         "profiles": {
  12.             "www": {
  13.             "expiry": "87600h",
  14.             "usages": [
  15.                 "signing",    
  16.                 "key encipherment",
  17.                 "server auth",
  18.                 "client auth"
  19.                 ]
  20.             }
  21.         }
  22.     }
  23. }
  24. EOF
  26. # 创建证书申请文件
  27. cat > etcd-ca-csr.json << EOF
  28. {
  29.     "CN": "etcd CA",
  30.     "key": {
  31.         "algo": "rsa",
  32.         "size": 2048
  33.     },
  34.     "names": [
  35.         {
  36.             "C": "CN",
  37.             "L": "Shanghai",
  38.             "ST": "Shanghai",
  39.             "O": "etcd",
  40.             "OU": "Etcd Security"
  41.         }
  42.     ]
  43. }
  44. EOF
  46. # 生成证书
  47. cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca -

7.2.2 生成 Etcd Server 证书

  1. # 创建证书申请文件
  2. # hosts字段中的IP为所有etcd节点的集群内部通信IP，为了方便后期扩容可以多写几个预留IP
  3. cat > etcd-server-csr.json << EOF
  4. {
  5.     "CN": "etcd",
  6.     "hosts": [
  7.     "k8s-master01",
  8.     "k8s-master02",
  9.     "k8s-master03",
  10.     "10.0.0.10",
  11.     "10.0.0.20",
  12.     "10.0.0.30",
  13.     "10.0.0.40"
  14.     ],
  15.     "key": {
  16.         "algo": "rsa",
  17.         "size": 2048 
  18.     },
  19.     "names": [
  20.         {
  21.             "C": "CN",
  22.             "L": "Shanghai",
  23.             "ST": "Shanghai"
  24.         }
  25.     ]
  26. }
  27. EOF
  29. # 生成证书
  30. cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=etcd-ca-config.json -profile=www etcd-server-csr.json | cfssljson -bare ./etcd-server

7.3 部署 Etcd 集群

Gitlab 项目地址

7.3.1 下载&配置

  1. # 下载
  2. wget https://github.com/etcd-io/etcd/releases/download/v3.4.18/etcd-v3.4.18-linux-amd64.tar.gz
  4. # 解压
  5. tar xvf etcd-v* -C /opt/
  7. # 软连接
  8. mv /opt/etcd-v* /opt/etcd
  10. # 创建目录
  11. mkdir -p /opt/etcd/{bin,conf,certs}
  13. # 移动可执行文件
  14. mv /opt/etcd/etcd* /opt/etcd/bin/
  16. # 修改属主
  17. chown -R root.root /opt/etcd/

7.3.2 创建 etcd 配置文件

  1. # 创建配置文件
  2. cat > /opt/etcd/conf/etcd.conf << EOF
  3. #[Member]
  4. ETCD_NAME="etcd-1"
  5. ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
  6. ETCD_LISTEN_PEER_URLS="https://10.0.0.10:2380"
  7. ETCD_LISTEN_CLIENT_URLS="https://10.0.0.10:2379"
  9. #[Clustering]
  10. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.10:2380"
  11. ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.10:2379"
  12. ETCD_INITIAL_CLUSTER="etcd-1=https://10.0.0.10:2380,etcd-2=https://10.0.0.20:2380,etcd-3=https://10.0.0.30:2380"
  13. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  14. ETCD_INITIAL_CLUSTER_STATE="new"
  15. EOF
  • ETCD_NAME：节点名称，集群中唯一
  • ETCD_DATA_DIR：数据目录
  • ETCD_LISTEN_PEER_URLS：集群通信监听地址
  • ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
  • ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
  • ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
  • ETCD_INITIAL_CLUSTER：集群节点地址池
  • ETCD_INITIALCLUSTER_TOKEN：集群Token
  • ETCD_INITIALCLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

    7.3.3 创建 service 文件

    ```bash cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target

[Service] Type=notify EnvironmentFile=/opt/etcd/conf/etcd.conf ExecStart=/opt/etcd/bin/etcd \ —cert-file=/opt/etcd/certs/etcd-server.pem \ —key-file=/opt/etcd/certs/etcd-server-key.pem \ —peer-cert-file=/opt/etcd/certs/etcd-server.pem \ —peer-key-file=/opt/etcd/certs/etcd-server-key.pem \ —trusted-ca-file=/opt/etcd/certs/etcd-ca.pem \ —peer-trusted-ca-file=/opt/etcd/certs/etcd-ca.pem \ —logger=zap Restart=on-failure LimitNOFILE=65536

[Install] WantedBy=multi-user.target EOF

  1. <a name="5a288613"></a>
  2. #### 7.3.4 拷贝证书到指定目录
  3. ```bash
  4. cp ~/tls/etcd/*pem /opt/etcd/certs/

7.3.5 配置其他节点

  1. # 将etcd目录和service文件拷贝到其他节点
  2. scp -r /opt/etcd/ root@k8s-master02:/opt/
  3. scp -r /opt/etcd/ root@k8s-master03:/opt/
  4. scp /usr/lib/systemd/system/etcd.service root@k8s-master02:/usr/lib/systemd/system/
  5. scp /usr/lib/systemd/system/etcd.service root@k8s-master03:/usr/lib/systemd/system/
  8. # 文件拷贝完成后修改etcd.conf配置文件中的节点名称和 IP。
  9. vim /opt/etcd/conf/etcd.conf
  10. #[Member]
  11. ETCD_NAME="etcd-2"                                                                        <== 节点名称
  12. ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
  13. ETCD_LISTEN_PEER_URLS="https://10.0.0.20:2380"            <== 节点IP
  14. ETCD_LISTEN_CLIENT_URLS="https://10.0.0.20:2379"        <== 节点IP
  16. #[Clustering]
  17. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.20:2380"        <== 节点IP
  18. ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.20:2379"                    <== 节点IP
  19. ETCD_INITIAL_CLUSTER="etcd-1=https://10.0.0.10:2380,etcd-2=https://10.0.0.20:2380,etcd-3=https://10.0.0.30:2380"
  20. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
  21. ETCD_INITIAL_CLUSTER_STATE="new"

7.3.6 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now etcd

7.3.7 查看集群状态

  1. ETCDCTL_API=3 /opt/etcd/bin/etcdctl \
  2.   --cacert=/opt/etcd/certs/etcd-ca.pem \
  3.   --cert=/opt/etcd/certs/etcd-server.pem \
  4.   --key=/opt/etcd/certs/etcd-server-key.pem \
  5.   --endpoints="https://10.0.0.10:2379,https://10.0.0.20:2379,https://10.0.0.30:2379" \
  6.   endpoint health \
  7.   --write-out=table
  10. # 出现下面的表格即说明集群部署成功
  11. # 如果有问题请查看日志进行排查：/var/log/message 或 journalctl -u etcd
  12. +------------------------+--------+------------+-------+
  13. |        ENDPOINT        | HEALTH |    TOOK    | ERROR |
  14. +------------------------+--------+------------+-------+
  15. | https://10.0.0.10:2379 |   true | 8.957952ms |       |
  16. | https://10.0.0.30:2379 |   true | 9.993095ms |       |
  17. | https://10.0.0.20:2379 |   true | 9.813129ms |       |
  18. +------------------------+--------+------------+-------+

8. 集群初始化

注意：以下操作只在 k8s-master01 节点执行。
Kubernetes 集群支持两种初始化方式，第一种是直接使用命令行指定参数进行初始化，第二种是使用配置文件的方式加载参数进行初始化，由于需要修改的参数较多，这里选用第二种方式，使用配置文件进行初始化。
使用命令kubeadm config print init-defaults > initConfig.yaml可以生成初始化配置文件模板，可以根据需要进行修改。下面是修改后的配置示例文件：

  1. apiVersion: kubeadm.k8s.io/v1beta2
  2. bootstrapTokens:
  3. - groups:
  4.   - system:bootstrappers:kubeadm:default-node-token
  5.   token: abcdef.0123456789abcdef
  6.   ttl: 24h0m0s
  7.   usages:
  8.   - signing
  9.   - authentication
  10. kind: InitConfiguration
  11. localAPIEndpoint:
  12.   advertiseAddress: 10.0.0.10                            # 主机IP
  13.   bindPort: 6443                                                    # apiServer端口
  14. nodeRegistration:
  15.   criSocket: /var/run/dockershim.sock
  16.   name: k8s-master01                                            # hostname
  17.   taints:                                                                    # 污点
  18.   - effect: NoSchedule
  19.     key: node-role.kubernetes.io/master
  20. ---
  21. apiServer:
  22.   certSANs:                # 访问apiServer的白名单，包含所有Master/LB/VIP，为了方便后期扩容可以多写几个预留IP
  23.   - k8s-master01
  24.   - k8s-master02
  25.   - k8s-master03
  26.   - k8s-lb
  27.   - 10.0.0.200
  28.   - 10.0.0.222
  29.   timeoutForControlPlane: 4m0s
  30. apiVersion: kubeadm.k8s.io/v1beta2
  31. certificatesDir: /etc/kubernetes/pki
  32. clusterName: kubernetes                                        # 集群名称
  33. controlPlaneEndpoint: k8s-lb:16443                # 控制平面endpoint
  34. controllerManager: {}
  35. dns:
  36.   type: CoreDNS
  37. etcd:
  38.   external:                                      # 使用外部etcd
  39.     endpoints:                                # etcd后端节点
  40.     - https://10.0.0.10:2379 
  41.     - https://10.0.0.20:2379
  42.     - https://10.0.0.30:2379
  43.     caFile: /opt/etcd/certs/etcd-ca.pem
  44.     certFile: /opt/etcd/certs/etcd-server.pem
  45.     keyFile: /opt/etcd/certs/etcd-server-key.pem
  46. imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers        # 镜像仓库设置
  47. kind: ClusterConfiguration
  48. kubernetesVersion: v1.21.10                                # 版本号
  49. networking:
  50.   dnsDomain: cluster.local
  51.   podSubnet: 172.16.0.0/12                                # Pod子网设置
  52.   serviceSubnet: 192.168.0.0/16                        # service子网设置
  53. scheduler: {}

8.1 更新 initConfig.yaml 文件

  1. kubeadm config migrate --old-config initConfig.yaml --new-config kubeadm-config.yaml

8.2 k8s-Master01 节点初始化

初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他 Master 节点加入 k8s-master01 节点即可。

  1. kubeadm init --config /root/kubeadm-config.yaml --upload-certs
  4. # 可以使用下面的命令提前下载镜像
  5. kubeadm config images list --config initConfig.yaml
  6. kubeadm config images pull --config /root/kubeadm-config.yaml

注意：如果初始化失败，可以排错后进行重置后，然后再次初始化，重置命令如下：

  1. kubeadm reset -f; ipvsadm --clear; rm -rf ~/.kube

初始化成功后，会给出一下提示操作，以及加入集群的 Token 令牌信息，如下所示：

  1. Your Kubernetes control-plane has initialized successfully!
  3. To start using your cluster, you need to run the following as a regular user:
  5.   mkdir -p $HOME/.kube
  6.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  7.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  9. Alternatively, if you are the root user, you can run:
  11.   export KUBECONFIG=/etc/kubernetes/admin.conf
  13. You should now deploy a pod network to the cluster.
  14. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  15.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  17. You can now join any number of the control-plane node running the following command on each as root:
  19.   kubeadm join k8s-lb:16443 --token abcdef.0123456789abcdef \
  20.     --discovery-token-ca-cert-hash sha256:c622312d44fbe54fe49ff0b19c72eacf163edaab49e3cb5d0104e15f68d1afc3 \
  21.     --control-plane --certificate-key 7d996915da0ab463b0ccc5b76877565866d77a43c767753fa94171c73d5b1d47
  23. Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
  24. As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
  25. "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
  27. Then you can join any number of worker nodes by running the following on each as root:
  29. kubeadm join k8s-lb:16443 --token abcdef.0123456789abcdef \
  30.     --discovery-token-ca-cert-hash sha256:c622312d44fbe54fe49ff0b19c72eacf163edaab49e3cb5d0104e15f68d1afc3

8.3 配置 kubeconfig 文件

  • 方式一：

    1. mkdir -p $HOME/.kube
    2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

  • 方式二：

    1. cat >> /root/.bashrc <<EOF
    2. export KUBECONFIG=/etc/kubernetes/admin.conf
    3. EOF
    4. source /root/.bashrc

    8.4 查看节点状态

    ```bash [root@k8s-master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 NotReady control-plane,master 12m v1.21.10

[root@k8s-master01 ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-6f6b8cc4f6-wrbrp 0/1 Pending 0 18m coredns-6f6b8cc4f6-z6nvb 0/1 Pending 0 18m etcd-k8s-master01 1/1 Running 0 19m kube-apiserver-k8s-master01 1/1 Running 0 19m kube-controller-manager-k8s-master01 1/1 Running 0 19m kube-proxy-klc6q 1/1 Running 0 18m kube-scheduler-k8s-master01 1/1 Running 0 19m

  1. 这里节点未就绪是正常的，因为网络插件还没有部署。
  2. <a name="Gb1JI"></a>
  3. ### 8.5 其他 Master 节点加入集群
  4. 使用提示中的命令，在其他 Master 节点执行：
  5. ```bash
  6. kubeadm join k8s-lb:16443 --token abcdef.0123456789abcdef \
  7.     --discovery-token-ca-cert-hash sha256:c622312d44fbe54fe49ff0b19c72eacf163edaab49e3cb5d0104e15f68d1afc3 \
  8.     --control-plane --certificate-key 7d996915da0ab463b0ccc5b76877565866d77a43c767753fa94171c73d5b1d47

注意：这里的 Token 有效期为 24h，过了有效期后可以使用以下命令重新生成 Token：

  1. kubeadm token create --print-join-command
  3. # Master节点还需要生成--certificate-key
  4. kubeadm init phase upload-certs --upload-certs

8.6 Node 节点加入集群

  1. kubeadm join k8s-lb:16443 --token abcdef.0123456789abcdef \
  2.     --discovery-token-ca-cert-hash sha256:c622312d44fbe54fe49ff0b19c72eacf163edaab49e3cb5d0104e15f68d1afc3

9 部署网络插件

常用的网络插件有 Calico 和 Flannel，选择其中一种部署即可。（推荐使用 Calico）

9.1 Calico

下载 yaml 文件：

  1. wget https://docs.projectcalico.org/manifests/calico.yaml

下载完后需要修改里面定义Pod网络（CALICO_IPV4POOL_CIDR），与初始化时的 Pod 网络一致。

  1. sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: "172.16.0.0/12"@g' calico.yaml

为了加快部署，将 yaml 中的镜像替换成国内镜像。注意：下面是我个人提前准备好的镜像，如果因为版本问题导致部署失败请自行更换镜像。

  1. sed -r -i '/calico\/cni:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-cni:v3.22.2#' calico.yaml
  3. sed -r -i '/calico\/pod2daemon-flexvol:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-pod2daemon-flexvol:v3.22.2#' calico.yaml
  5. sed -r -i '/calico\/node:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-node:v3.22.2#' calico.yaml
  7. sed -r -i '/calico\/kube-controllers:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-kube-controllers:v3.22.2#' calico.yaml

部署 yaml 文件：

  1. kubectl apply -f calico.yaml
  3. # 查看calico pod运行情况
  4. kubectl get pod -n kube-system

等 calico pod 全部正常运行，再次查看 Node 节点，发现已经变为 Ready 状态：

  1. [root@k8s-master01 ~]# kubectl get nodes
  2. NAME           STATUS   ROLES                  AGE     VERSION
  3. k8s-master01   Ready    control-plane,master   23m         v1.21.10
  4. k8s-master02   Ready    control-plane,master   35m        v1.21.10
  5. k8s-master03   Ready    control-plane,master   37m        v1.21.10
  6. k8s-node01     Ready    <none>                 45m        v1.21.10
  7. k8s-node02     Ready    <none>                 47m       v1.21.10

9.2 Flannel

下载 Yaml 文件：

  1. wget https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

10. 测试集群

创建一个 Nginx deployment 和 svc，访问 http://NodeIP:Port。 

  1. kubectl create deployment nginx --image=nginx
  3. kubectl expose deployment nginx --port=80 --type=NodePort
  5. kubectl get pod,svc
  7. curl <pod-ip>:80

二进制高可用部署

1. 环境架构

1.1 角色划分

主机名 IP 角色 服务组件
k8s-master01 10.0.0.11 k8s-master节点 Kube-apiserver、Kube-controller-manager、Kube-scheduler、Etcd、Docker、Kubelet、Kubectl
k8s-master02 10.0.0.12 k8s-master节点 Kube-apiserver、Kube-controller-manager、Kube-scheduler、Etcd、Docker、Kubelet、Kubectl
k8s-master03 10.0.0.13 k8s-master节点 Kube-apiserver、Kube-controller-manager、Kube-scheduler、Etcd、Docker、Kubelet、Kubectl
k8s-node01 10.0.0.21 k8s-node节点 Docker、Kubelet、Kube-proxy
k8s-node02 10.0.0.22 k8s-node节点 Docker、Kubelet、Kube-proxy
Haproxy01 10.0.0.31 Haproxy + Keepalived Keepalived+Haproxy
Haproxy02 10.0.0.32 Haproxy + Keepalived Keepalived+Haproxy

10.0.0.99 虚拟IP Keepalived

1.2 拓扑图

k8s高可用拓扑图.png

2. 操作系统初始化

2.1 关闭防火墙

  1. systemctl stop firewalld
  2. systemctl disable firewalld

2.2 关闭 selinux

  1. setenforce 0
  2. sed -i 's/enforcing/disabled/' /etc/selinux/config

2.3 关闭 swap

  1. swapoff -a && sysctl -w vm.swappiness=0
  2. sed -ri 's/.*swap.*/#&/' /etc/fstab

2.4 修改 hosts

  1. cat >> /etc/hosts << EOF
  2. 10.0.0.11 k8s-master01
  3. 10.0.0.12 k8s-master02
  4. 10.0.0.13 k8s-master03
  5. 10.0.0.21 k8s-node01
  6. 10.0.0.22 k8s-node02
  7. 10.0.0.99 k8s-master-lb
  8. EOF

2.5 时间同步

  1. # 硬件时钟和系统时间同步
  2. hwclock -s
  4. # 修改时区
  5. ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  6. echo "Asia/Shanghai" > /etc/timezone
  8. # 安装 chrony 时间同步服务
  9. yum install -y chrony
  11. # 修改时间服务配置文件
  12. sed -i '/iburst/d' /etc/chrony.conf
  13. sed -i '2aserver ntp1.aliyun.com iburst\nserver ntp2.aliyun.com iburst\nserver time1.cloud.tencent.com iburst\nserver time2.cloud.tencent.com iburst' /etc/chrony.conf
  15. # 启动服务
  16. systemctl enable --now chronyd
  18. # 查看是否生效
  19. chronyc sources -v

2.6 配置 limit

  1. ulimit -SHn 65535
  3. cat >> /etc/security/limits.conf <<EOF
  4. * soft nofile 65536
  5. * hard nofile 131072
  6. * soft nproc 65535
  7. * hard nproc 655350
  8. * soft memlock unlimited
  9. * hard memlock unlimited
  10. * soft core unlimited
  11. * hard core unlimited
  12. EOF

2.7 ssh 免密登录（可选）

  1. # 在k8s-master01上执行以下操作
  2. ssh-keygen
  3. ssh-copy-id 127.0.0.1
  5. scp -r .ssh k8s-master02:/root/
  6. scp -r .ssh k8s-master03:/root/
  7. scp -r .ssh k8s-node01:/root/
  8. scp -r .ssh k8s-node02:/root/

2.8 升级内核版本

CentOS7 需要升级内核至 4.18+，但也不要太新，太新可能会出现老机器无法启动的情况。
这里选择 5.4 版本，手动下载内核进行安装。
CentOS7 内核下载地址

  1. # 更新软件包并重启
  2. yum update -y --exclude=kernel*
  4. # 下载内核（下载速度较慢，建议提前下载好）
  5. wget https://elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-lt-5.4.xxx-1.el7.elrepo.x86_64.rpm
  6. wget https://elrepo.org/linux/kernel/el7/x86_64/RPMS/kernel-lt-devel-5.4.xxx-1.el7.elrepo.x86_64.rpm
  8. # 手动升级内核
  9. yum localinstall -y kernel-lt*
  11. # 设置默认启动内核
  12. grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg
  13. grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
  15. # 查看默认内核版本
  16. grubby --default-kernel
  18. # 重启
  19. reboot
  21. # 查看内核版本
  22. uname -a

2.9 安装 ipvs 并加载到内核模块

在内核 4.19+ 版本 nf_conntrack_ipv4 已经改为 nf_conntrack，另外ip_vs_fo调度算法是内核 4.15 版本之后才有的，如果没有升级内核会报错。

  1. # 安装工具包
  2. yum install -y ipvsadm ipset sysstat conntrack libseccomp
  4. # 开机加载模块
  5. cat > /etc/modules-load.d/ipvs.conf << EOF
  6. ip_vs
  7. ip_vs_lc
  8. ip_vs_wlc
  9. ip_vs_rr
  10. ip_vs_wrr
  11. ip_vs_lblc
  12. ip_vs_lblcr
  13. ip_vs_dh
  14. ip_vs_sh
  15. ip_vs_fo
  16. ip_vs_nq
  17. ip_vs_sed
  18. ip_vs_ftp
  19. ip_vs_sh
  20. nf_conntrack
  21. ip_tables
  22. ip_set
  23. xt_set
  24. ipt_set
  25. ipt_rpfilter
  26. ipt_REJECT
  27. ipip
  28. EOF
  30. # 开机启动模块自动加载服务
  31. systemctl enable --now systemd-modules-load.service
  33. # 重启后查看模块加载情况
  34. lsmod | grep ip_vs

2.10 修改内核参数

  1. cat  > /etc/sysctl.d/k8s.conf <<EOF
  2. net.ipv4.ip_forward = 1
  3. net.bridge.bridge-nf-call-iptables = 1
  4. net.bridge.bridge-nf-call-ip6tables = 1
  5. fs.may_detach_mounts = 1
  6. net.ipv4.conf.all.route_localnet = 1
  7. vm.overcommit_memory=1
  8. vm.panic_on_oom=0
  9. fs.inotify.max_user_watches=89100
  10. fs.file-max=52706963
  11. fs.nr_open=52706963
  12. net.netfilter.nf_conntrack_max=2310720
  14. net.ipv4.tcp_keepalive_time = 600
  15. net.ipv4.tcp_keepalive_probes = 3
  16. net.ipv4.tcp_keepalive_intvl =15
  17. net.ipv4.tcp_max_tw_buckets = 36000
  18. net.ipv4.tcp_tw_reuse = 1
  19. net.ipv4.tcp_max_orphans = 327680
  20. net.ipv4.tcp_orphan_retries = 3
  21. net.ipv4.tcp_syncookies = 1
  22. net.ipv4.tcp_max_syn_backlog = 16384
  23. net.ipv4.ip_conntrack_max = 65536
  24. net.ipv4.tcp_max_syn_backlog = 16384
  25. net.ipv4.tcp_timestamps = 0
  26. net.core.somaxconn = 16384
  27. EOF
  28. sysctl --system
  30. # 重启
  31. reboot

3. 安装 Docker

  1. # 添加 Docker repo 源
  2. wget -P /etc/yum.repos.d/ https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  4. # 查看版本
  5. yum list --showduplicates docker-ce
  7. # 安装
  8. yum -y install docker-ce
  10. # 修改镜像源和使用systemd管理cgroup
  11. ## 建议单独挂载一块盘到/var/lib/docker目录
  12. mkdir /etc/docker
  13. cat > /etc/docker/daemon.json <<EOF
  14. {
  15.   "exec-opts": ["native.cgroupdriver=systemd"],
  16.   "log-driver": "json-file",
  17.   "log-opts": {
  18.     "max-size": "300m",
  19.     "max-file": "2"
  20.   },
  21.   "data-root": "/var/lib/docker",
  22.   "storage-driver": "overlay2",
  23.   "max-concurrent-uploads": 5,
  24.   "max-concurrent-downloads": 10,
  25.   "registry-mirrors": [
  26.         "https://mirror.ccs.tencentyun.com", 
  27.         "https://docker.mirrors.ustc.edu.cn",
  28.         "https://registry.docker-cn.com"
  29.   ],
  30.   "live-restore": true
  31. }
  32. EOF
  35. # 启动docker
  36. systemctl daemon-reload
  37. systemctl enable --now docker

4. Etcd 集群搭建

4.1 准备 cfssl 证书工具

Gitlab 项目地址

  1. # 下载工具包
  2. wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
  3. wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
  4. wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
  6. # 添加执行权限
  7. chmod +x cfssl*
  9. # 移动并重命名
  10. mv cfssl_linux-amd64 /usr/local/bin/cfssl
  11. mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
  12. mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

4.2 生成 Etcd 证书

创建证书目录：

  1. mkdir -p ~/tls/etcd

4.2.1 生成自签CA

  1. # 切换目录
  2. cd ~/tls/etcd
  4. # 创建证书配置文件
  5. cat > etcd-ca-config.json << EOF
  6. {
  7.     "signing": {
  8.         "default": {
  9.             "expiry": "87600h"
  10.         },
  11.         "profiles": {
  12.             "www": {
  13.             "expiry": "87600h",
  14.             "usages": [
  15.                 "signing",    
  16.                 "key encipherment",
  17.                 "server auth",
  18.                 "client auth"
  19.                 ]
  20.             }
  21.         }
  22.     }
  23. }
  24. EOF
  26. # 创建证书申请文件
  27. cat > etcd-ca-csr.json << EOF
  28. {
  29.     "CN": "etcd CA",
  30.     "key": {
  31.         "algo": "rsa",
  32.         "size": 2048
  33.     },
  34.     "names": [
  35.         {
  36.             "C": "CN",
  37.             "L": "Shanghai",
  38.             "ST": "Shanghai",
  39.             "O": "etcd",
  40.             "OU": "Etcd Security"
  41.         }
  42.     ]
  43. }
  44. EOF
  46. # 生成证书
  47. cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare ./etcd-ca -

4.2.2 生成 Etcd Server 证书

  1. # 创建证书申请文件
  2. # hosts字段中的IP为所有etcd节点的集群内部通信IP，为了方便后期扩容可以多写几个预留IP
  3. cat > etcd-server-csr.json << EOF
  4. {
  5.     "CN": "etcd",
  6.     "hosts": [
  7.     "k8s-master01",
  8.     "k8s-master02",
  9.     "k8s-master03",
  10.     "10.0.0.11",
  11.     "10.0.0.12",
  12.     "10.0.0.13",
  13.     "10.0.0.14",
  14.     "10.0.0.15"
  15.     ],
  16.     "key": {
  17.         "algo": "rsa",
  18.         "size": 2048 
  19.     },
  20.     "names": [
  21.         {
  22.             "C": "CN",
  23.             "L": "Shanghai",
  24.             "ST": "Shanghai"
  25.         }
  26.     ]
  27. }
  28. EOF
  30. # 生成证书
  31. cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=etcd-ca-config.json -profile=www etcd-server-csr.json | cfssljson -bare ./etcd-server

4.3 部署 Etcd 集群

Gitlab 项目地址

4.3.1 下载&配置

  1. # 下载安装包（建议手动下载后上传）
  2. wget https://github.com/etcd-io/etcd/releases/download/v3.4.18/etcd-v3.4.18-linux-amd64.tar.gz
  4. # 解压
  5. tar xvf etcd-v* -C /opt/
  7. # 重命名
  8. mv /opt/etcd-v* /opt/etcd
  10. # 创建目录
  11. mkdir -p /opt/etcd/{bin,conf,certs}
  13. # 移动可执行文件
  14. mv /opt/etcd/etcd* /opt/etcd/bin/
  16. # 修改属主
  17. chown -R root.root /opt/etcd/

4.3.2 创建 etcd 配置文件

ETCD 中文文档

  1. # 创建配置文件
  2. cat > /opt/etcd/conf/etcd.config.yml <<-EOF
  3. name: 'etcd-node01'
  4. data-dir: /opt/data
  5. wal-dir:  /opt/data/wal
  6. snapshot-count: 5000
  7. heartbeta-interval: 100
  8. election-timeout: 1000
  9. quota-backend-bytes: 0
  10. listen-peer-urls: 'https://10.0.0.11:2380'
  11. listen-client-urls: 'https://10.0.0.11:2379,https://127.0.0.1:2379'
  12. max-snapshots: 3
  13. max-wals: 5
  14. initial-advertise-peer-urls: 'https://10.0.0.11:2380'
  15. advertise-client-urls: 'https://10.0.0.11:2379'
  16. initial-cluster: 'etcd-node01=https://10.0.0.11:2380,etcd-node02=https://10.0.0.12:2380,etcd-node03=https://10.0.0.13:2380'
  17. initial-cluster-token: 'etcd-k8s-cluster'
  18. initial-cluster-state: 'new'
  19. strict-reconfig-check: false
  20. enable-pprof: true
  21. client-transport-security:
  22.   cert-file:  '/opt/etcd/certs/etcd-server.pem'
  23.   key-file:  '/opt/etcd/certs/etcd-server-key.pem'
  24.   client-cert-auth: true
  25.   trusted-ca-file:  '/opt/etcd/certs/etcd-ca.pem'
  26.   auto-tls: true
  27. peer-transport-security:
  28.   cert-file: '/opt/etcd/certs/etcd-server.pem'
  29.   key-file: '/opt/etcd/certs/etcd-server-key.pem'
  30.   peer-client-cert-auth: true
  31.   trusted-ca-file:  '/opt/etcd/certs/etcd-ca.pem'
  32.   auto-tls: true
  33. debug: false
  34. logger: zap
  35. log-level: warn
  36. log-outputs: [default]
  37. force-new-cluster: false
  38. EOF

4.3.3 创建 service 文件

  1. cat > /usr/lib/systemd/system/etcd.service << EOF
  2. [Unit]
  3. Description=Etcd Server
  4. Documentation=https://etcd.io/docs/
  5. After=network.target
  6. After=network-online.target
  7. Wants=network-online.target
  9. [Service]
  10. Type=notify
  11. ExecStart=/opt/etcd/bin/etcd --config-file=/opt/etcd/conf/etcd.config.yml
  12. Restart=on-failure
  13. RestartSec=10
  14. LimitNOFILE=65536
  16. [Install]
  17. WantedBy=multi-user.target
  18. EOF

4.3.4 拷贝证书到指定目录

  1. cp ~/tls/etcd/*.pem /opt/etcd/certs/

4.3.5 配置其他节点

将 etcd 目录和 service 文件拷贝到其他节点：

  1. scp -r /opt/etcd/ root@k8s-master02:/opt/
  2. scp -r /opt/etcd/ root@k8s-master03:/opt/
  3. scp /usr/lib/systemd/system/etcd.service root@k8s-master02:/usr/lib/systemd/system/
  4. scp /usr/lib/systemd/system/etcd.service root@k8s-master03:/usr/lib/systemd/system/

文件拷贝完成后修改 etcd.conf 配置文件中的节点名称和 IP：

  1. vim /opt/etcd/conf/etcd.config.yml
  3. name: 'etcd-node02'
  4. ...
  5. listen-peer-urls: 'https://10.0.0.12:2380'
  6. listen-client-urls: 'https://10.0.0.12:2379,https://127.0.0.1:2379'
  7. ...
  8. initial-advertise-peer-urls: 'https://10.0.0.12:2380'
  9. advertise-client-urls: 'https://10.0.0.12:2379'

4.3.6 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now etcd

4.3.7 查看集群状态

  1. ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/certs/etcd-ca.pem --cert=/opt/etcd/certs/etcd-server.pem --key=/opt/etcd/certs/etcd-server-key.pem --endpoints="https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379" endpoint health --write-out=table
  3. # 出现下面的表格即说明集群部署成功
  4. # 如果有问题请查看日志进行排查：/var/log/message 或 journalctl -u etcd
  5. +------------------------+--------+------------+-------+
  6. |        ENDPOINT        | HEALTH |    TOOK    | ERROR |
  7. +------------------------+--------+------------+-------+
  8. | https://10.0.0.11:2379 |   true | 8.957952ms |       |
  9. | https://10.0.0.12:2379 |   true | 9.993095ms |       |
  10. | https://10.0.0.13:2379 |   true | 9.813129ms |       |
  11. +------------------------+--------+------------+-------+

5. 部署 Haproxy 和 Keepalived

5.1 安装 Haproxy 和 Keepalived

  1. [root@haproxy-node01 ~]# yum -y install keepalived haproxy
  2. [root@haproxy-node02 ~]# yum -y install keepalived haproxy

5.2 配置 Haproxy

两个节点配置相同

  1. [root@haproxy-node01 ~]# vim /etc/haproxy/haproxy.cfg
  2. global
  3.     daemon
  4.     log         127.0.0.1 local2 err        # 全局rsyslog定义
  5.     chroot      /var/lib/haproxy
  6.     pidfile     /var/run/haproxy.pid
  7.     maxconn     20000                                                # 每个haproxy进程所接受的最大并发连接数
  8.     user        haproxy
  9.     group       haproxy
  10.     stats socket /var/lib/haproxy/stats
  11.     spread-checks 2                                                 # 后端server状态check随机提前或延迟百分比时间，建议2-5(20%-50%)之间
  13. defaults
  14.     log     global
  15.       mode  http                                            # 使用七层代理
  16.       option      httplog                                # 在日志中记录http请求、session信息等
  17.       option        dontlognull                        # 不要在日志中记录空连接
  18.     option    redispatch                        # 用于cookie保持的环境中。默认情况下，HAProxy会将其请求的后端服务器的serverID插入cookie中，以保证会话的session持久性。如果后端服务器出现故障，客户端的cookie是不会刷新的，这就会造成无法访问。此时，如果设置了此参数，就会将客户的请求强制定向到另外一台健康的后端服务器上，以保证服务正常。
  19.       timeout     connect 10s                        # haproxy和服务端建立连接的最大时长，设置为1秒就足够了。局域网内建立连接一般都是瞬间的
  20.       timeout     client  3m                        # 和客户端保持空闲连接的超时时长，高并发下可稍微短一点，可设置为10秒以尽快释放连接
  21.       timeout     server  1m                        # 和服务端保持空闲连接的超时时长，局域网内建立连接很快，所以尽量设置短一些，特别是并发时，如设置为1-3秒
  22.       timeout     http-request 15s            #  等待客户端发送完整请求的最大时长，应该设置较短些防止洪水攻击,优先级高于timeout client
  23.       timeout     http-keep-alive 15s        # 和客户端保持长连接的最大时长。优先级高于timeout
  25. frontend monitor-in
  26.       bind *:33305
  27.       mode http
  28.       option httplog
  29.       monitor-uri /monitor
  31. listen stats
  32.   bind     *:8006
  33.   mode     http
  34.   stats    enable
  35.   stats    hide-version
  36.   stats    uri           /stats
  37.   stats    refresh       30s
  38.   stats    realm         Haproxy Statistics
  39.   stats    auth          admin:admin123
  41. frontend k8s-master
  42.       bind 0.0.0.0:6443
  43.       bind 127.0.0.1:6443
  44.       mode tcp
  45.       option tcplog
  46.       tcp-request inspect-delay 5s
  47.       default_backend k8s-master
  49. backend k8s-master
  50.       mode tcp
  51.       option tcplog
  52.       option tcp-check
  53.     balance roundrobin
  54.       default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  55.       server k8s-master01    10.0.0.11:6443 check
  56.       server k8s-master02    10.0.0.12:6443 check
  57.          server k8s-master03    10.0.0.13:6443 check

5.3 启动 Haproxy

  1. [root@haproxy-node01 ~]# systemctl enable --now haproxy
  2. [root@haproxy-node02 ~]# systemctl enable --now haproxy

5.4 配置 Keepalived

编辑配置文件： 

  1. [root@haproxy-node01 ~]# vim /etc/keepalived/keepalived.conf
  2. global_defs {
  3.    script_user root             # 脚本执行用户
  4.    router_id haproxy01          # keepalived 主机唯一标识，建议使用当前主机名
  5.    vrrp_skip_check_adv_addr     # 如果收到的通告报文和上一个报文是同一个路由，则跳过检查，默认为检查所有报文
  6.    enable_script_security       # 如果脚本对任一非root用户来说具有可写权限，则不以root身份运行脚本
  7. }
  9. vrrp_script check_apiserver {
  10.     script "/etc/keepalived/check_apiserver.sh"
  11.     interval 5      # 检测间隔时间
  12.     weight -10
  13.     fall 2
  14.     rise 1
  15.     timeout 2       # 超时时间
  16. }
  18. vrrp_instance VIP_API_SERVER {
  19.     state MASTER
  20.     interface eth0
  21.     mcast_src_ip 10.0.0.31    # 发送多播包的地址，如果没有设置，默认使用绑定网卡的primary ip
  22.     virtual_router_id 88      # 虚拟路由器惟一标识，同一个虚拟路由器的多个keepalived节点此值必须相同
  23.     priority 100              # Keepalived优先级
  24.     advert_int 2
  25.     authentication {          # 使用普通密码认证
  26.         auth_type PASS
  27.         auth_pass API_AUTH
  28.     }
  30.     virtual_ipaddress {
  31.         10.0.0.99 dev eth0    # 虚拟IP
  32.     }
  34.     track_script {            # 健康检查
  35.         check_apiserver
  36.     }
  37. }

准备 Haproxy 检测脚本：

  1. [root@haproxy-node01 ~]# vim /etc/keepalived/check_apiserver.sh
  2. #!/bin/bash
  4. err=0
  5. for k in $(seq 1 3)
  6. do
  7.     check_code=$(pgrep haproxy)
  8.     if [[ $check_code == "" ]]; then
  9.         err=$(expr $err + 1)
  10.         sleep 1
  11.         continue
  12.     else
  13.         err=0
  14.         break
  15.     fi
  16. done
  18. if [[ $err != "0" ]]; then
  19.     echo "systemctl stop keepalived"
  20.     /usr/bin/systemctl stop keepalived
  21.     exit 1
  22. else
  23.     exit 0
  24. fi
  1. 添加执行权限：
  1. chmod +x /etc/keepalived/check_apiserver.sh

将配置文件拷贝到 haproxy-node02 上：

  1. [root@haproxy-node01 ~]# scp /etc/keepalived/* haproxy-node02:/etc/keepalived/

修改 haproxy-node02 配置文件：

  1. [root@haproxy-node02 ~]# vim /etc/keepalived/keepalived.conf
  2. ...
  3.    router_id hapeoxy-node02     # keepalived 主机唯一标识，建议使用当前主机名
  4.    state BACKUP
  5.    mcast_src_ip 10.0.0.32                # 发送组播包的地址，如果没有设置，默认使用绑定网卡的primary ip
  6.    priority 95                    # Keepalived优先级
  7. ...

5.5 启动 Keepalived

  1. systemctl enable --now keepalived

6. 准备 k8s 相关证书

创建证书目录：

  1. mkdir -p ~/tls/k8s/{ca,kube-apiserver,kube-controller-manager,kube-scheduler,kubectl,front-proxy,service-account}

6.1 生成自签 CA

  1. # 切换目录
  2. cd ~/tls/k8s/ca
  4. # 创建证书配置文件
  5. cat > k8s-ca-config.json << EOF
  6. {
  7.     "signing": {
  8.         "default": {
  9.             "expiry": "876000h"
  10.         },
  11.         "profiles": {
  12.             "kubernetes": {
  13.                 "expiry": "876000h",
  14.                 "usages": [
  15.                     "signing",
  16.                     "key encipherment",
  17.                     "server auth",
  18.                     "client auth"
  19.                 ]
  20.             }
  21.         }
  22.     }
  23. }
  24. EOF
  26. # 创建证书申请文件 
  27. cat > k8s-ca-csr.json << EOF
  28. {
  29.     "CN": "kubernetes",
  30.     "key": {
  31.         "algo": "rsa",
  32.         "size": 2048
  33.     },
  34.     "names": [
  35.         {
  36.             "C": "CN",
  37.             "L": "Shanghai",
  38.             "ST": "Shanghai",
  39.             "O": "Kubernetes",
  40.             "OU": "Kubernetes-bin"
  41.         }
  42.     ],
  43.     "ca": {
  44.         "expiry": "876000h"
  45.     }
  46. }
  47. EOF
  49. # 生成证书
  50. cfssl gencert -initca k8s-ca-csr.json | cfssljson -bare ./k8s-ca -

6.2 生成 kube-apiServer 证书

  1. # 切换目录
  2. cd ~/tls/k8s/kube-apiserver
  4. # 创建证书申请文件
  5. cat > kube-apiserver-csr.json << EOF
  6. {
  7.     "CN": "kube-apiserver",
  8.     "hosts": [
  9.         "k8s-master01",
  10.         "k8s-master02",
  11.         "k8s-master03",
  12.         "kubernetes",
  13.         "kubernetes.default",
  14.         "kubernetes.default.svc",
  15.         "kubernetes.default.svc.cluster",
  16.         "kubernetes.default.svc.cluster.local",
  17.         "127.0.0.1",
  18.         "192.168.0.1",
  19.         "10.0.0.11",
  20.         "10.0.0.12",
  21.         "10.0.0.13",
  22.         "10.0.0.99"
  23.     ],
  24.     "key": {
  25.         "algo": "rsa",
  26.         "size": 2048
  27.     },    
  28.     "names": [
  29.         {
  30.             "C": "CN",
  31.             "ST": "Shanghai",
  32.             "L": "Shanghai",
  33.             "O": "Kubernetes",
  34.             "OU": "Kubernetes-bin"
  35.         }
  36.     ]
  37. }
  38. EOF
  40. # 生成证书
  41. cfssl gencert -ca=../ca/k8s-ca.pem -ca-key=../ca/k8s-ca-key.pem -config=../ca/k8s-ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare ./kube-apiserver

6.3 生成 apiServer 聚合证书

  1. # 切换目录
  2. cd ~/tls/k8s/front-proxy
  4. # 创建CA申请文件
  5. cat > front-proxy-ca-csr.json <<-EOF
  6. {
  7.   "CN": "kubernetes",
  8.   "key": {
  9.      "algo": "rsa",
  10.      "size": 2048
  11.   },
  12.   "ca": {
  13.     "expiry": "876000h"
  14.   }
  15. }
  16. EOF
  18. # 生成front-proxy-ca证书
  19. cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare ./front-proxy-ca
  21. # 创建证书申请文件
  22. cat > front-proxy-client-csr.json <<-EOF
  23. {
  24.   "CN": "front-proxy-client",
  25.   "key": {
  26.      "algo": "rsa",
  27.      "size": 2048
  28.   }
  29. }
  30. EOF
  32. cfssl gencert -ca=./front-proxy-ca.pem -ca-key=./front-proxy-ca-key.pem -config=../ca/k8s-ca-config.json -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare ./front-proxy-client

6.4 生成 kube-controller-manager 证书

  1. # 切换工作目录
  2. cd ~/tls/k8s/kube-controller-manager
  4. # 创建证书请求文件
  5. cat > kube-controller-manager-csr.json << EOF
  6. {
  7.     "CN": "system:kube-controller-manager",
  8.     "key": {
  9.         "algo": "rsa",
  10.         "size": 2048
  11.     },
  12.     "names": [
  13.         {
  14.             "C": "CN",
  15.             "L": "Shanghai", 
  16.             "ST": "Shanghai",
  17.             "O": "system:kube-controller-manager",
  18.             "OU": "Kubernetes-bin"
  19.         }
  20.     ]
  21. }
  22. EOF
  24. # 生成证书
  25. cfssl gencert -ca=../ca/k8s-ca.pem -ca-key=../ca/k8s-ca-key.pem -config=../ca/k8s-ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare ./kube-controller-manager

6.5 生成 kube-schedule 证书

  1. # 切换工作目录
  2. cd ~/tls/k8s/kube-scheduler
  4. # 创建证书请求文件
  5. cat > kube-scheduler-csr.json << EOF
  6. {
  7.   "CN": "system:kube-scheduler",
  8.   "hosts": [],
  9.   "key": {
  10.     "algo": "rsa",
  11.     "size": 2048
  12.   },
  13.   "names": [
  14.     {
  15.       "C": "CN",
  16.       "L": "Shanghai",
  17.       "ST": "Shanghai",
  18.       "O": "system:kube-scheduler",
  19.       "OU": "Kubernetes-bin"
  20.     }
  21.   ]
  22. }
  23. EOF
  25. # 生成证书
  26. cfssl gencert -ca=../ca/k8s-ca.pem -ca-key=../ca/k8s-ca-key.pem -config=../ca/k8s-ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare ./kube-scheduler

6.6 生成 kubectl 证书

  1. # 切换工作目录
  2. cd ~/tls/k8s/kubectl
  4. # 创建证书请求文件
  5. cat > admin-csr.json <<EOF
  6. {
  7.   "CN": "admin",
  8.   "hosts": [],
  9.   "key": {
  10.     "algo": "rsa",
  11.     "size": 2048
  12.   },
  13.   "names": [
  14.     {
  15.       "C": "CN",
  16.       "L": "Shanghai",
  17.       "ST": "Shanghai",
  18.       "O": "system:masters",
  19.       "OU": "Kubernetes-bin"
  20.     }
  21.   ]
  22. }
  23. EOF
  25. # 生成证书
  26. cfssl gencert -ca=../ca/k8s-ca.pem -ca-key=../ca/k8s-ca-key.pem -config=../ca/k8s-ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare ./admin

6.7 生成 Service Account 密钥对

  1. # 切换工作目录
  2. cd ~/tls/k8s/service-account
  4. # 创建密钥对
  5. openssl genrsa -out ./sa.key 2048
  6. openssl rsa -in ./sa.key -pubout -out ./sa.pub

7. 部署 Master 节点组件

7.1 下载 Kubernetes 二进制文件

Kubernetes 下载地址
上面的地址是 Kubernetes github 各版本下载地址，选择对应的版本，下载 Server Binaries 即可，里面包含了 Master 和 Worker Node 的所有二进制文件。（这里选择的是 v1.21.11 版本）

  1. # 回到家目录
  2. cd
  4. # 创建k8s工作目录
  5. mkdir -p /opt/kubernetes/{bin,conf,certs,logs,manifests}
  7. # 下载（速度较慢，建议手动下载后上传）
  8. wget https://dl.k8s.io/v1.21.11/kubernetes-server-linux-amd64.tar.gz
  10. # 解压
  11. tar xvf kubernetes-server-linux-amd64.tar.gz
  13. # 将命令行工具拷贝到对应目录
  14. cp ~/kubernetes/server/bin/kube{-apiserver,-controller-manager,-scheduler,let,-proxy,ctl} /opt/kubernetes/bin
  16. # 创建kubectl软连接
  17. ln -s /opt/kubernetes/bin/kubectl /usr/local/bin/kubectl

7.2 部署 kube-apiServer

7.2.1 创建 service 文件

kube-apiserver启动参数官方文档

  1. cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
  2. [Unit]
  3. Description=Kubernetes API Server
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=network.target
  7. [Service]
  8. ExecStart=/opt/kubernetes/bin/kube-apiserver \\
  9.   --logtostderr=false \\
  10.   --v=2 \\
  11.   --log-file=/opt/kubernetes/logs/kube-apiserver.log \\
  12.   --etcd-servers=https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379 \\
  13.   --etcd-cafile=/opt/etcd/certs/etcd-ca.pem \\
  14.   --etcd-certfile=/opt/etcd/certs/etcd-server.pem \\
  15.   --etcd-keyfile=/opt/etcd/certs/etcd-server-key.pem \\
  16.   --bind-address=0.0.0.0 \\
  17.   --secure-port=6443 \\
  18.   --advertise-address=10.0.0.11 \\
  19.   --allow-privileged=true \\
  20.   --service-cluster-ip-range=192.168.0.0/16 \\
  21.   --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota,NodeRestriction \\
  22.   --authorization-mode=RBAC,Node \\
  23.   --enable-bootstrap-token-auth=true \\
  24.   --service-node-port-range=30000-32767 \\
  25.   --kubelet-client-certificate=/opt/kubernetes/certs/kube-apiserver.pem \\
  26.   --kubelet-client-key=/opt/kubernetes/certs/kube-apiserver-key.pem \\
  27.   --tls-cert-file=/opt/kubernetes/certs/kube-apiserver.pem  \\
  28.   --tls-private-key-file=/opt/kubernetes/certs/kube-apiserver-key.pem \\
  29.   --client-ca-file=/opt/kubernetes/certs/k8s-ca.pem \\
  30.   --service-account-key-file=/opt/kubernetes/certs/sa.pub \\
  31.   --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
  32.   --service-account-signing-key-file=/opt/kubernetes/certs/sa.key \\
  33.   --requestheader-client-ca-file=/opt/kubernetes/certs/front-proxy-ca.pem \\
  34.   --proxy-client-cert-file=/opt/kubernetes/certs/front-proxy-client.pem \\
  35.   --proxy-client-key-file=/opt/kubernetes/certs/front-proxy-client-key.pem \\
  36.   --requestheader-allowed-names=aggregator \\
  37.   --requestheader-extra-headers-prefix=X-Remote-Extra- \\
  38.   --requestheader-group-headers=X-Remote-Group \\
  39.   --requestheader-username-headers=X-Remote-User \\
  40.   --enable-aggregator-routing=true \\
  41.   --audit-log-maxage=30 \\
  42.   --audit-log-maxbackup=3 \\
  43.   --audit-log-maxsize=100 \\
  44.   --audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
  45. Restart=on-failure
  46. RestartSec=10s
  47. LimitNOFILE=65535
  49. [Install]
  50. WantedBy=multi-user.target
  51. EOF

7.2.2 拷贝证书文件

  1. cp ~/tls/k8s/ca/k8s-ca*pem ~/tls/k8s/kube-apiserver/kube-apiserver*pem ~/tls/k8s/front-proxy/front-proxy*.pem ~/tls/k8s/service-account/* /opt/kubernetes/certs/

7.2.3 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now kube-apiserver
  3. systemctl status kube-apiserver

7.3 部署 kube-controller-manager

7.3.1 拷贝证书文件

  1. cp ~/tls/k8s/kube-controller-manager/kube-controller-manager*.pem /opt/kubernetes/certs/

7.3.2 生成 kubeconfig 文件

  1. KUBE_CONFIG="/opt/kubernetes/conf/kube-controller-manager.kubeconfig"
  2. KUBE_APISERVER="https://10.0.0.99:6443"
  4. # 设置cluster信息
  5. kubectl config set-cluster kubernetes \
  6.   --certificate-authority=/opt/kubernetes/certs/k8s-ca.pem \
  7.   --embed-certs=true \
  8.   --server=${KUBE_APISERVER} \
  9.   --kubeconfig=${KUBE_CONFIG}
  11. # 设置用户信息
  12. kubectl config set-credentials kube-controller-manager \
  13.   --client-certificate=/opt/kubernetes/certs/kube-controller-manager.pem \
  14.   --client-key=/opt/kubernetes/certs/kube-controller-manager-key.pem \
  15.   --embed-certs=true \
  16.   --kubeconfig=${KUBE_CONFIG}
  18. # 设置上下文信息
  19. kubectl config set-context kube-controller-manager@kubernetes \
  20.   --cluster=kubernetes \
  21.   --user=kube-controller-manager \
  22.   --kubeconfig=${KUBE_CONFIG}
  24. # 选择要使用的上下文信息
  25. kubectl config use-context kube-controller-manager@kubernetes --kubeconfig=${KUBE_CONFIG}

7.3.3 创建 service 文件

注意：**--cluster-cidr=172.16.0.0/12**该参数为集群 Pod 子网，请确保设置正确。

  1. cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
  2. [Unit]
  3. Description=Kubernetes Controller Manager
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=network.target
  7. [Service]
  8. ExecStart=/opt/kubernetes/bin/kube-controller-manager \\
  9.   --v=2 \\
  10.   --logtostderr=false \\
  11.   --log-file=/opt/kubernetes/logs/kube-controller-manager.log \\
  12.   --address=0.0.0.0 \\
  13.   --root-ca-file=/opt/kubernetes/certs/k8s-ca.pem \\
  14.   --cluster-signing-cert-file=/opt/kubernetes/certs/k8s-ca.pem \\
  15.   --cluster-signing-key-file=/opt/kubernetes/certs/k8s-ca-key.pem \\
  16.   --service-account-private-key-file=/opt/kubernetes/certs/sa.key \\
  17.   --kubeconfig=/opt/kubernetes/conf/kube-controller-manager.kubeconfig \\
  18.   --leader-elect=true \\
  19.   --use-service-account-credentials=true \\
  20.   --node-monitor-grace-period=40s \\
  21.   --node-monitor-period=5s \\
  22.   --pod-eviction-timeout=2m0s \\
  23.   --controllers=*,bootstrapsigner,tokencleaner \\
  24.   --allocate-node-cidrs=true \\
  25.   --cluster-cidr=172.16.0.0/12 \\
  26.   --requestheader-client-ca-file=/opt/kubernetes/certs/front-proxy-ca.pem \\
  27.   --cluster-signing-duration=876000h0m0s \\
  28.   --node-cidr-mask-size=24
  29. Restart=always
  30. RestartSec=10s
  32. [Install]
  33. WantedBy=multi-user.target
  34. EOF

7.3.4 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now kube-controller-manager
  3. systemctl status kube-controller-manager

7.4 部署 kube-scheduler

7.4.1 拷贝证书文件

  1. cp ~/tls/k8s/kube-scheduler/kube-scheduler*.pem /opt/kubernetes/certs/

7.4.2 生成 kubeconfig 文件

  1. KUBE_CONFIG="/opt/kubernetes/conf/kube-scheduler.kubeconfig"
  2. KUBE_APISERVER="https://10.0.0.99:6443"
  4. # 设置cluster信息
  5. kubectl config set-cluster kubernetes \
  6.   --certificate-authority=/opt/kubernetes/certs/k8s-ca.pem \
  7.   --embed-certs=true \
  8.   --server=${KUBE_APISERVER} \
  9.   --kubeconfig=${KUBE_CONFIG}
  11. # 设置用户信息
  12. kubectl config set-credentials kube-scheduler \
  13.   --client-certificate=/opt/kubernetes/certs/kube-scheduler.pem \
  14.   --client-key=/opt/kubernetes/certs/kube-scheduler-key.pem \
  15.   --embed-certs=true \
  16.   --kubeconfig=${KUBE_CONFIG}
  18. # 设置上下文信息
  19. kubectl config set-context kube-scheduler@kubernetes \
  20.   --cluster=kubernetes \
  21.   --user=kube-scheduler \
  22.   --kubeconfig=${KUBE_CONFIG}
  24. # 选择要使用的上下文信息
  25. kubectl config use-context kube-scheduler@kubernetes --kubeconfig=${KUBE_CONFIG}

7.4.3 创建 service 文件

  1. cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
  2. [Unit]
  3. Description=Kubernetes Scheduler
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=network.target
  7. [Service]
  8. ExecStart=/opt/kubernetes/bin/kube-scheduler \\
  9.   --logtostderr=false \\
  10.   --v=2 \\
  11.   --log-file=/opt/kubernetes/logs/kube-scheduler.log \\
  12.   --leader-elect=true \\
  13.   --kubeconfig=/opt/kubernetes/conf/kube-scheduler.kubeconfig \\
  14.   --bind-address=127.0.0.1
  15. Restart=always
  16. RestartSec=10s
  18. [Install]
  19. WantedBy=multi-user.target
  20. EOF

7.4.4 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now kube-scheduler
  3. systemctl status kube-scheduler

7.5 部署 kubectl

7.5.1 拷贝证书文件

  1. cp ~/tls/k8s/kubectl/admin*.pem /opt/kubernetes/certs/

7.5.2 生成 kubeconfig 文件

  1. KUBE_CONFIG="/opt/kubernetes/conf/kubectl.kubeconfig"
  2. KUBE_APISERVER="https://10.0.0.99:6443"
  4. # 设置cluster信息
  5. kubectl config set-cluster kubernetes \
  6.   --certificate-authority=/opt/kubernetes/certs/k8s-ca.pem \
  7.   --embed-certs=true \
  8.   --server=${KUBE_APISERVER} \
  9.   --kubeconfig=${KUBE_CONFIG}
  11. # 设置用户信息
  12. kubectl config set-credentials kubernetes-admin \
  13.   --client-certificate=/opt/kubernetes/certs/admin.pem \
  14.   --client-key=/opt/kubernetes/certs/admin-key.pem \
  15.   --embed-certs=true \
  16.   --kubeconfig=${KUBE_CONFIG}
  18. # 设置上下文信息
  19. kubectl config set-context kubernetes-admin@kubernetes \
  20.   --cluster=kubernetes \
  21.   --user=kubernetes-admin \
  22.   --kubeconfig=${KUBE_CONFIG}
  24. # 选择要使用的上下文信息
  25. kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=${KUBE_CONFIG}
  27. # 创建kubectl config目录
  28. mkdir /root/.kube
  30. # 拷贝kubeconfig文件到默认目录
  31. cp /opt/kubernetes/conf/kubectl.kubeconfig /root/.kube/config

7.5.3 查看集群状态

  1. [root@k8s-master01 ~]# kubectl get cs
  2. Warning: v1 ComponentStatus is deprecated in v1.19+
  3. NAME                 STATUS    MESSAGE             ERROR
  4. controller-manager   Healthy   ok
  5. scheduler            Healthy   ok
  6. etcd-2               Healthy   {"health":"true"}
  7. etcd-1               Healthy   {"health":"true"}
  8. etcd-0               Healthy   {"health":"true"}

如上输出则 Master 节点正常运行。

7.6 为 kubelet-bootstrap 授权

kubelet-bootstrap TLS启动引导

7.6.1 TLS Bootstrapping 配置

vim ~/tls-bootstrapping.yaml

  1. apiVersion: v1
  2. kind: Secret
  3. metadata:
  4.   name: bootstrap-token-8e2e05
  5.   namespace: kube-system
  6. type: bootstrap.kubernetes.io/token
  7. stringData:
  8.   description: "The default bootstrap token generated by 'kubelet '."
  9.   token-id: 8e2e05
  10.   token-secret: 1bfcc0c4bb0dbf2c
  11.   usage-bootstrap-authentication: "true"
  12.   usage-bootstrap-signing: "true"
  13.   auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress
  14. ---
  15. apiVersion: rbac.authorization.k8s.io/v1
  16. kind: ClusterRoleBinding
  17. metadata:
  18.   name: kubelet-bootstrap
  19. roleRef:
  20.   apiGroup: rbac.authorization.k8s.io
  21.   kind: ClusterRole
  22.   name: system:node-bootstrapper
  23. subjects:
  24. - apiGroup: rbac.authorization.k8s.io
  25.   kind: Group
  26.   name: system:bootstrappers:default-node-token
  27. ---
  28. apiVersion: rbac.authorization.k8s.io/v1
  29. kind: ClusterRoleBinding
  30. metadata:
  31.   name: node-autoapprove-bootstrap
  32. roleRef:
  33.   apiGroup: rbac.authorization.k8s.io
  34.   kind: ClusterRole
  35.   name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  36. subjects:
  37. - apiGroup: rbac.authorization.k8s.io
  38.   kind: Group
  39.   name: system:bootstrappers:default-node-token
  40. ---
  41. apiVersion: rbac.authorization.k8s.io/v1
  42. kind: ClusterRoleBinding
  43. metadata:
  44.   name: node-autoapprove-certificate-rotation
  45. roleRef:
  46.   apiGroup: rbac.authorization.k8s.io
  47.   kind: ClusterRole
  48.   name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  49. subjects:
  50. - apiGroup: rbac.authorization.k8s.io
  51.   kind: Group
  52.   name: system:nodes
  53. ---
  54. apiVersion: rbac.authorization.k8s.io/v1
  55. kind: ClusterRole
  56. metadata:
  57.   annotations:
  58.     rbac.authorization.kubernetes.io/autoupdate: "true"
  59.   labels:
  60.     kubernetes.io/bootstrapping: rbac-defaults
  61.   name: system:kube-apiserver-to-kubelet
  62. rules:
  63.   - apiGroups:
  64.       - ""
  65.     resources:
  66.       - nodes/proxy
  67.       - nodes/stats
  68.       - nodes/log
  69.       - nodes/spec
  70.       - nodes/metrics
  71.     verbs:
  72.       - "*"
  73. ---
  74. apiVersion: rbac.authorization.k8s.io/v1
  75. kind: ClusterRoleBinding
  76. metadata:
  77.   name: system:kube-apiserver
  78.   namespace: ""
  79. roleRef:
  80.   apiGroup: rbac.authorization.k8s.io
  81.   kind: ClusterRole
  82.   name: system:kube-apiserver-to-kubelet
  83. subjects:
  84.   - apiGroup: rbac.authorization.k8s.io
  85.     kind: User
  86.     name: kube-apiserver

上面的 yaml 文件大致可以分为三个部分，第一个部分是名为bootstrap-token-<token id>的 secret，用于验证 Node 节点身份；第二个部分为中间的三个ClusterRoleBinding，分别实现为 Node 节点进行身份绑定、controller manager自动颁发证书和证书过期后自动续签证书的功能；最后一个部分则创建了一个system:kube-apiserver-to-kubelet的集群角色，并将其与kube-apiserver用户绑定，作用是授权 apiServer 去访问 kubelet。

7.6.2 应用 yaml 文件

  1. kubectl apply -f tls-bootstrapping.yaml

7.6.3 TLS Bootstrapping 原理（选修）

TLS Bootstraping：Kubernetes 集群默认启用 TLS 认证进行连接，Node 节点 kubelet 和 kube-proxy 组件要和 apiServer 进行通信，必须使用集群 CA 签发的有效证书才可以，当 Node 节点很多时，如果手动颁发证书，会给管理员带来巨大的工作量，不利于集群的扩展。因此，Kubernetes 引入了 TLS bootstraping 机制来为 Node 节点自动颁发证书，kubelet 会以一个低权限令牌自动连接到 apiServer 并申请证书，kubelet 的证书由 controllerManager 自动进行验证和签署。
整个过程具体如下：

  1. kubelet 服务启动后会先去寻找**bootstrap-kubeconfig**文件。
  2. kubelet 读取该文件，从中获取到 apiServer 的 URL 地址和一个权限比较低 Token 令牌（仅用于申请证书）。
  3. kubelet 连接到 apiServer ，并使用这个 Token 令牌做身份验证。
    1. 在上面 7.6.1章节的 yaml 文件中，第一部分就是集群Secret，验证过程主要就是验证bootstrap-kubeconfig（可以去8.1.1章节查看bootstrap-kubeconfig的创建）文件中的Token令牌，与集群的Secret是否一致。
    2. 引导令牌使用 abcdef.0123456789abcdef 的形式，它需要符合正则表达式 [a-z0-9]{6}\.[a-z0-9]{16}。令牌的第一部分（小数点前面部分）是Token ID，它是一种公开信息，使用这个Token ID就可以在集群中找到对应的Secret资源。 第二部分才是真正的Token Secret，只有这两部分都匹配上，才算是验证通过。
    3. 集群中的 Secret 令牌附带一个额外的身份组**system:bootstrappers:default-node-token**，用于将 Node 节点绑定到**system:node-bootstrapper**角色（上面 7.6.1章节的 yaml 文件中第二部分的ClusterRoleBinding），查看一下这个角色可以发现，这个角色仅拥有 CSR 证书相关的权限。 ```yaml [root@k8s-master01 ~]# kubectl get clusterrole system:node-bootstrapper -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: “true” creationTimestamp: “2022-04-10T02:28:47Z” labels: kubernetes.io/bootstrapping: rbac-defaults name: system:node-bootstrapper resourceVersion: “101” uid: 9cbbcca1-2e64-4484-96b4-52043cffb3e9 rules:
  • apiGroups:
    • certificates.k8s.io resources:
    • certificatesigningrequests verbs:
    • create
    • get
    • list
    • watch ```
  1. 验证通过后，kubelet 节点就获取到了**system:node-bootstrapper**角色的权限，以此来申请和创建 CSR 证书。
  2. controllerManager 如果配置了，就可以自动对 CSR 进行审核批复，如果没有配置，需要管理员手动使用命令进行批复。
    1. 配置 controllerManager 实际就是创建一个ClusterRoleBinding，将集群角色**system:certificates.k8s.io:certificatesigningrequests:nodeclient**绑定到 Node 节点的 Group **system:bootstrappers:default-node-token**上，查看一下这个角色可以发现，该角色拥有certificatesigningrequests/nodeclient的create权限。而这个certificatesigningrequests/nodeclient正是 Kubernetes 中专门为 Node 节点颁发证书的一种资源。 ```yaml [root@k8s-master01 ~]# kubectl get clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: “true” creationTimestamp: “2022-04-10T02:28:47Z” labels: kubernetes.io/bootstrapping: rbac-defaults name: system:certificates.k8s.io:certificatesigningrequests:nodeclient resourceVersion: “107” uid: b0ef0dae-cf2c-42b1-81bc-1b2e2c3541c6 rules:
  • apiGroups:
    • certificates.k8s.io resources:
    • certificatesigningrequests/nodeclient verbs:
    • create ```
  1. kubelet 所需要的证书被批复然后发放给 Node 节点。
  2. kubelet 拿到证书后会自行创建一个**kubeconfig**，里面包含了密钥和签名证书，并以此开始正常和 apiServier 交互。

注意：要想启用 TLS Bootstrapping 机制，需要在 kube-apiserver 服务中开启** --enable-bootstrap-token-auth=true**参数，上面部署过程中已经开启了此项参数，这里仅作说明。

8. 部署 Worker 节点组件

以下操作也需要在 Master 节点上上进行，即 Master Node 同时也作为 Worker Node，如果不想在 Master Node 上部署 Pod 的话，后续使用下面的命令为 Master Node 打上 taint 。

  1. kubectl taint nodes k8s-master01 node-role.kubernetes.io/master=:NoSchedule

8.1 部署 kubelet

8.1.1 创建 kubeconfig 文件

  1. KUBE_CONFIG="/opt/kubernetes/conf/bootstrap.kubeconfig"
  2. KUBE_APISERVER="https://10.0.0.99:6443"
  3. TOKEN="8e2e05.1bfcc0c4bb0dbf2c" # 与tls-bootstrapping.yaml里的令牌保持一致
  5. # 设置cluster信息
  6. kubectl config set-cluster kubernetes \
  7.   --certificate-authority=/opt/kubernetes/certs/k8s-ca.pem \
  8.   --embed-certs=true \
  9.   --server=${KUBE_APISERVER} \
  10.   --kubeconfig=${KUBE_CONFIG}
  12. # 设置用户信息
  13. kubectl config set-credentials kubelet-bootstrap \
  14.   --token=${TOKEN} \
  15.   --kubeconfig=${KUBE_CONFIG}
  17. # 设置上下文信息
  18. kubectl config set-context kubelet-bootstrap@kubernetes \
  19.   --cluster=kubernetes \
  20.   --user=kubelet-bootstrap \
  21.   --kubeconfig=${KUBE_CONFIG}
  23. # 选择要使用的上下文信息
  24. kubectl config use-context kubelet-bootstrap@kubernetes --kubeconfig=${KUBE_CONFIG}

8.1.2 创建配置参数文件

注意：**clusterDNS**默认为 service 子网第十个 ip，如果 service 子网有调整，请根据自己的设置进行修改。

  1. cat > /opt/kubernetes/conf/kubelet-config.yml << EOF
  2. apiVersion: kubelet.config.k8s.io/v1beta1
  3. kind: KubeletConfiguration
  4. address: 0.0.0.0
  5. port: 10250
  6. readOnlyPort: 10255
  7. authentication:
  8.   anonymous:
  9.     enabled: false
  10.   webhook:
  11.     cacheTTL: 2m0s
  12.     enabled: true
  13.   x509:
  14.     clientCAFile: /opt/kubernetes/certs/k8s-ca.pem
  15. authorization:
  16.   mode: Webhook
  17.   webhook:
  18.     cacheAuthorizedTTL: 5m0s
  19.     cacheUnauthorizedTTL: 30s
  20. cgroupDriver: systemd
  21. cgroupsPerQOS: true
  22. clusterDNS:
  23. - 192.168.0.10
  24. clusterDomain: cluster.local
  25. containerLogMaxFiles: 5
  26. containerLogMaxSize: 10Mi
  27. contentType: application/vnd.kubernetes.protobuf
  28. cpuCFSQuota: true
  29. cpuManagerPolicy: none
  30. cpuManagerReconcilePeriod: 10s
  31. enableControllerAttachDetach: true
  32. enableDebuggingHandlers: true
  33. enforceNodeAllocatable:
  34. - pods
  35. eventBurst: 10
  36. eventRecordQPS: 5
  37. evictionHard:
  38.   imagefs.available: 15%
  39.   memory.available: 100Mi
  40.   nodefs.available: 10%
  41.   nodefs.inodesFree: 5%
  42. evictionPressureTransitionPeriod: 5m0s
  43. failSwapOn: true
  44. fileCheckFrequency: 20s
  45. hairpinMode: promiscuous-bridge
  46. healthzBindAddress: 127.0.0.1
  47. healthzPort: 10248
  48. httpCheckFrequency: 20s
  49. imageGCHighThresholdPercent: 85
  50. imageGCLowThresholdPercent: 80
  51. imageMinimumGCAge: 2m0s
  52. iptablesDropBit: 15
  53. iptablesMasqueradeBit: 14
  54. kubeAPIBurst: 10
  55. kubeAPIQPS: 5
  56. makeIPTablesUtilChains: true
  57. maxOpenFiles: 1000000
  58. maxPods: 110
  59. nodeStatusUpdateFrequency: 10s
  60. oomScoreAdj: -999
  61. podPidsLimit: -1
  62. registryBurst: 10
  63. registryPullQPS: 5
  64. resolvConf: /etc/resolv.conf
  65. rotateCertificates: true
  66. runtimeRequestTimeout: 2m0s
  67. serializeImagePulls: true
  68. staticPodPath: /opt/kubernetes/manifests
  69. streamingConnectionIdleTimeout: 4h0m0s
  70. syncFrequency: 1m0s
  71. volumeStatsAggPeriod: 1m0s
  72. EOF

8.1.3 创建 service 文件

  1. cat > /usr/lib/systemd/system/kubelet.service << EOF
  2. [Unit]
  3. Description=Kubernetes Kubelet
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=docker.service
  6. Requires=docker.service
  8. [Service]
  9. ExecStart=/opt/kubernetes/bin/kubelet \\
  10.   --logtostderr=false \\
  11.   --v=2 \\
  12.   --log-file=/opt/kubernetes/logs/kubelet.log \\
  13.   --network-plugin=cni \\
  14.   --kubeconfig=/opt/kubernetes/conf/kubelet.kubeconfig \\
  15.   --bootstrap-kubeconfig=/opt/kubernetes/conf/bootstrap.kubeconfig \\
  16.   --config=/opt/kubernetes/conf/kubelet-config.yml \\
  17.   --cert-dir=/opt/kubernetes/certs \\
  18.   --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.5 \\
  19.   --tls-cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 \\
  20.   --image-pull-progress-deadline=30m
  21. Restart=always
  22. RestartSec=10
  23. LimitNOFILE=65536
  24. StartLimitInterval=0
  26. [Install]
  27. WantedBy=multi-user.target
  28. EOF

8.1.4 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now kubelet
  3. systemctl status kubelet

8.2 部署 kube-proxy

8.2.1 创建 kubeconfig 文件

  1. # 创建 kube-proxy sa用户
  2. kubectl -n kube-system create serviceaccount kube-proxy
  4. # 将kube-proxy sa用户绑定到system:node-proxier角色
  5. kubectl create clusterrolebinding system:kube-proxy --clusterrole system:node-proxier --serviceaccount kube-system:kube-proxy
  8. SECRET=$(kubectl -n kube-system get sa/kube-proxy --output=jsonpath='{.secrets[0].name}')
  9. TOKEN=$(kubectl -n kube-system get secret/$SECRET --output=jsonpath='{.data.token}' | base64 -d)
  10. KUBE_CONFIG="/opt/kubernetes/conf/kube-proxy.kubeconfig"
  11. KUBE_APISERVER="https://10.0.0.99:6443"
  13. # 设置集群信息
  14. kubectl config set-cluster kubernetes \
  15.   --certificate-authority=/opt/kubernetes/certs/k8s-ca.pem \
  16.   --embed-certs=true \
  17.   --server=${KUBE_APISERVER} \
  18.   --kubeconfig=${KUBE_CONFIG}
  20. # 设置用户信息
  21. kubectl config set-credentials kube-proxy \
  22.   --token=${TOKEN} \
  23.   --kubeconfig=${KUBE_CONFIG}
  25. # 设置上下文信息
  26. kubectl config set-context kube-proxy@kubernetes \
  27.   --cluster=kubernetes \
  28.   --user=kube-proxy \
  29.   --kubeconfig=${KUBE_CONFIG}
  31. # 选择要使用的上下文信息
  32. kubectl config use-context kube-proxy@kubernetes --kubeconfig=${KUBE_CONFIG}

8.2.2 创建 kube-proxy config 文件

注意：**kubeconfig**文件的路径和**clusterCIDR**pod 子网，如果和上面的操作不一样，请自行修改。

  1. cat > /opt/kubernetes/conf/kube-proxy-config.yml << EOF
  2. apiVersion: kubeproxy.config.k8s.io/v1alpha1
  3. bindAddress: 0.0.0.0
  4. clientConnection:
  5.   acceptContentTypes: ""
  6.   burst: 10
  7.   contentType: application/vnd.kubernetes.protobuf
  8.   kubeconfig: /opt/kubernetes/conf/kube-proxy.kubeconfig
  9.   qps: 5
  10. clusterCIDR: 172.16.0.0/12
  11. configSyncPeriod: 15m0s
  12. conntrack:
  13.   max: null
  14.   maxPerCore: 32768
  15.   min: 131072
  16.   tcpCloseWaitTimeout: 1h0m0s
  17.   tcpEstablishedTimeout: 24h0m0s
  18. enableProfiling: false
  19. healthzBindAddress: 0.0.0.0:10256
  20. hostnameOverride: ""
  21. iptables:
  22.   masqueradeAll: false
  23.   masqueradeBit: 14
  24.   minSyncPeriod: 0s
  25.   syncPeriod: 30s
  26. ipvs:
  27.   masqueradeAll: true
  28.   minSyncPeriod: 5s
  29.   scheduler: "rr"
  30.   syncPeriod: 30s
  31. kind: KubeProxyConfiguration
  32. metricsBindAddress: 127.0.0.1:10249
  33. mode: "ipvs"
  34. nodePortAddresses: null
  35. oomScoreAdj: -999
  36. portRange: ""
  37. udpIdleTimeout: 250ms
  38. EOF

8.2.3 创建 service 文件

  1. cat > /usr/lib/systemd/system/kube-proxy.service << EOF
  2. [Unit]
  3. Description=Kubernetes Proxy
  4. Documentation=https://github.com/kubernetes/kubernetes
  5. After=network.target
  7. [Service]
  8. ExecStart=/opt/kubernetes/bin/kube-proxy \\
  9.   --log-file=/opt/kubernetes/logs/kube-proxy.log \\
  10.   --v=2 \\
  11.   --config=/opt/kubernetes/conf/kube-proxy-config.yml
  12. Restart=always
  13. RestartSec=10s
  14. LimitNOFILE=65536
  16. [Install]
  17. WantedBy=multi-user.target
  18. EOF

8.2.4 加载配置并启动

  1. systemctl daemon-reload
  2. systemctl enable --now kube-proxy
  3. systemctl status kube-proxy

9. 部署其他节点

9.1 启动另外两个 Master 节点

9.1.1 拷贝文件

  1. scp -r /opt/kubernetes k8s-master02:/opt/
  2. scp -r /opt/kubernetes k8s-master03:/opt/
  3. scp /usr/lib/systemd/system/kube* k8s-master02:/usr/lib/systemd/system/
  4. scp /usr/lib/systemd/system/kube* k8s-master03:/usr/lib/systemd/system/
  6. # 清理日志目录文件
  7. rm -rf /opt/kubernetes/logs/*
  9. # 删除node节点证书
  10. rm -rf /opt/kubernetes/certs/kubelet-client*

9.1.2 修改service文件

只需要修改kube-apiserver.service文件即可。

  1. # 修改kube-apiserver.service
  2. [root@k8s-master02 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
  3. ...
  4.   --advertise-address=10.0.0.12 \
  5. ...
  7. [root@k8s-master03 ~]# vim /usr/lib/systemd/system/kube-apiserver.service
  8. ...
  9.   --advertise-address=10.0.0.13 \
  10. ...

9.1.3 启动服务

  1. systemctl daemon-reload
  2. systemctl enable --now kube-apiserver
  3. systemctl enable --now kube-controller-manager
  4. systemctl enable --now kube-scheduler
  5. systemctl enable --now kubelet
  6. systemctl enable --now kube-proxy

9.2 加入 Node 节点

9.2.1 拷贝文件

  1. scp -r /opt/kubernetes k8s-node01:/opt/
  2. scp -r /opt/kubernetes k8s-node02:/opt/
  3. scp /usr/lib/systemd/system/kube{let,-proxy}.service k8s-node01:/usr/lib/systemd/system/
  4. scp /usr/lib/systemd/system/kube{let,-proxy}.service k8s-node02:/usr/lib/systemd/system/
  7. # 清理日志目录文件
  8. rm -rf /opt/kubernetes/logs/*
  10. # 删除node节点证书
  11. rm -rf /opt/kubernetes/certs/kubelet-client*

9.2.2 启动服务

  1. systemctl daemon-reload
  2. systemctl enable --now kubelet
  3. systemctl enable --now kube-proxy

9.3 查看节点状态

  1. [root@k8s-master01 ~]# kubectl get nodes
  2. NAME           STATUS     ROLES    AGE     VERSION
  3. k8s-master01   NotReady   <none>   3m58s   v1.21.11
  4. k8s-master02   NotReady   <none>   54s     v1.21.11
  5. k8s-master03   NotReady   <none>   51s     v1.21.11
  6. k8s-node01     NotReady   <none>   7s      v1.21.11
  7. k8s-node02     NotReady   <none>   4s      v1.21.11

所有节点都加入集群成功，这里状态显示为NotReady是正常的，因为还没有部署网络插件。

10. 部署网络插件

这里使用 Calico 网络插件。

10.1 下载 yaml 文件

  1. curl https://projectcalico.docs.tigera.io/manifests/calico-etcd.yaml -O

10.2 修改 yaml 文件

  1. # 修改etcd后端集群地址
  2. sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379"#g' calico-etcd.yaml
  4. ETCD_CA=`cat /opt/etcd/certs/etcd-ca.pem | base64 -w 0`
  5. ETCD_CERT=`cat /opt/etcd/certs/etcd-server.pem | base64 -w 0`
  6. ETCD_KEY=`cat /opt/etcd/certs/etcd-server-key.pem | base64 -w 0`
  7. POD_SUBNET='172.16.0.0/12'
  9. # 添加etcd证书配置
  10. sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml
  12. # 修改Pod子网
  13. sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: "'${POD_SUBNET}'"@g' calico-etcd.yaml
  15. # 指定容器内证书路径
  16. sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
  19. # 替换镜像为国内镜像源
  20. sed -r -i '/calico\/cni:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-cni:v3.22.2#' calico-etcd.yaml
  21. sed -r -i '/calico\/pod2daemon-flexvol:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-pod2daemon-flexvol:v3.22.2#' calico-etcd.yaml
  22. sed -r -i '/calico\/node:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-node:v3.22.2#' calico-etcd.yaml
  23. sed -r -i '/calico\/kube-controllers:/s#(image: ).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-kube-controllers:v3.22.2#' calico-etcd.yaml

下面为修改后的 yaml 文件：（仅供参考，不可以直接拿来使用，需要按上面的步骤修改相应内容。）

  1. ---
  2. # Source: calico/templates/calico-etcd-secrets.yaml
  3. # The following contains k8s Secrets for use with a TLS enabled etcd cluster.
  4. # For information on populating Secrets, see http://kubernetes.io/docs/user-guide/secrets/
  5. apiVersion: v1
  6. kind: Secret
  7. type: Opaque
  8. metadata:
  9.   name: calico-etcd-secrets
  10.   namespace: kube-system
  11. data:
  12.   # Populate the following with etcd TLS configuration if desired, but leave blank if
  13.   # not using TLS for etcd.
  14.   # The keys below should be uncommented and the values populated with the base64
  15.   # encoded contents of each file that would be associated with the TLS data.
  16.   # Example command for encoding a file contents: cat <file> | base64 -w 0
  17.   etcd-key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBNFFoWUZ3M1dZeHdnWXFBdS9vZTJvS21QSTNJNStHcTlOQWh0TU1IaFQwQldId1YyCk5NTWl1dlJxQldaazJZSW5vMHZhdFZRVmkyalkxT3M1ZW9FTkh6aWJTbmdHNDRMWmd5TE5ib3llWFdabHN4OFIKKys0ck52Zm9ZZ1ZQaHFZUllLa0JJbXVpUkNMSk5rSzVzYm1YVy94Ky9icFRrTXNwbzNkMnkzd0lTa1EyVDdwTworOVpuR0xSNms5OE1OMWlUOUUzRSt1RzNnZE9PcGNrcTVOSGNCQWRvZzZVcnBLVGppT1ZnZW5RZ1BUQ1ptWGRTCjMrbHQ0ZDZrRDRYckVJRXpOVGJPb3RsK0lSOTk3b2ZYWU5Eb0FGY0hZZlVPZXBNNmhUSlpycm15MTVqUWdzMzMKb0Y0eWhUS3hvdGZEQWFUSGRKL2JubTlOSUl2NkR3YzI4ejRWb1FJREFRQUJBb0lCQUZWSkdrbXlmdHdWSHRrQwpLSTFpMWY1U0JPUEVPT0JEWSszQ2U4K015MTJEM3AvbkxBT1hLU1I5MldpdWcrNURlODVSdzFRUnlCQ0w3aEhVCkRIYTNUa2ZVNWxYMDQzZDZjZ0ZYUVZCeWp0cUxJbVFlbEd4TXVhZ2RrTDhGa0R4UjJDZ3dKVFBzWXhVN3NkS1UKdTFmMkpMMUNGc2NHeFFFTlhueUxuRVJHZjhYVkdtbVV3RlMzL0UxRlZ3elhHRithaitpQ3VTRnRuS1RacFJvWQpHTi9HUXg4a29uT3paazNBZVBUTGg5KzBWWkgraGJJdUZXeEpSVnc1ZDN2Nnphd2VHT255STVPcE9QKzdSaEVGCnIzbmpWK0RtNSttSjlOQmxkVDZIeklRQWllNWxralVYNUJEYkhmNHNpNFV4ODVLSHB6RmNhZ3pPOWhiM0hoSnMKa1hwaXNBRUNnWUVBNzF4RlVvaExrWk9RNnhzdmd0cHpibWVYdnJrUU5wZDR3UFlTdzV2UlZ6QzdXRFhQN2tNVQpqOHVTYTllOUZNQzd3dThDRmFkRFJNZy9wWDhLYUVveC9TMGwydTNpdFU4S2ZIRVlpb21wWldXSVBSdWNGTUFoCmk2WFJnWEdyQVUvckRhVDZiZUo3dW5vQm91MkNJTVdBbEU4MG9kQWpJa3MvMGFCTUdlUU1sY0VDZ1lFQThLMFgKUWRKTXR2L0FqTlMra216MnpBT0tzc3FLWXRDYStWay9MMmN3YjZsUFRkK1NqZ2U0eU1VMnltSzZkQjRHVDgzNgpPSzYzYXZIdWNTNTdpUGhIY3lFME5NUHhHK0l0cnE2a1ZYRFI4aFNWcnFDZG9yeEJXUlE0ZzRUTmFUVTZ4YjdkCkhJWDM2ejFjQ2plZUpQWVZkc1RUNHhKcU83ZHFmWXo4SzhTQ04rRUNnWUJ6VVVyOHBqbHlHMXdZU1BLczRqMzIKbE9ZZ3R0dyswTjhXWi9kR1VrMmE2aGw0T0FURFQ0SWFmUU0vL2FKVTJ6NGFmamZNNHRhZnpySUxNNWQ2QXpUdQpPdEFnRjdlL2ppTUpMUnVic3hMdmpPa0VOOGNNb0R3SklZdzhCdDFTTm9TU0FFZStwUHpqTEdDWmREL2JOZ3ZxClovWjlkdDlXYUZhSTBlZVlHYXlPZ1FLQmdRQ3VkcHE4VVFqaFF3VytVTjJRWVVLSjR5dGdyYWVoTjl5UFVoMlcKYkRQVlRhTTZMUTRjdlFZN0s1S3JKT1JLRmhXM05mWGpydXN3c0EyTHFBbE80Y05SZHNsVVVwRnVDQWZiYjZlYQpjLzBNZ2RWN1Q5elZRYWdSUGJJMXpFZm1zdnBXckNPMmNIcWw0TVhlZGpEQ1ZFeDgydk5KMEdpSHYwNkhrbktkCis0WDFvUUtCZ1FESk0wbE1oUUNBZFNpR3l6TjNLWjZLeEhvZTZnU0F5REVlcG43UE9tRkJ1VTBvL28zTTV2dEkKbTNxUjZMYnhqaWs4WkFDNUxSNXNvWW02dEJtOFZFdGpzQlo1Rm1DTytkN2tDWmVJWmFSOGZOQy9MMCtpM0JYeQpWTUdXUHIvSUdGYUNYSkJSNkpaYm9FMjNFaUlRZkNZemQ1ZnBQK2lya1ArbTQrTkgrcnpWM0E9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
  18.   etcd-cert: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVFRENDQXZpZ0F3SUJBZ0lVWEtOLzlwck9JZEpvcXJacFRVVGRQZE5hczlvd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2JERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0ZOb1lXNW5hR0ZwTVJFd0R3WURWUVFIRXdoVAphR0Z1WjJoaGFURU5NQXNHQTFVRUNoTUVaWFJqWkRFV01CUUdBMVVFQ3hNTlJYUmpaQ0JUWldOMWNtbDBlVEVRCk1BNEdBMVVFQXhNSFpYUmpaQ0JEUVRBZUZ3MHlNakEwTWpNd01UVTNNREJhRncwek1qQTBNakF3TVRVM01EQmEKTUVJeEN6QUpCZ05WQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hUYUdGdVoyaGhhVEVSTUE4R0ExVUVCeE1JVTJoaApibWRvWVdreERUQUxCZ05WQkFNVEJHVjBZMlF3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURoQ0ZnWERkWmpIQ0Jpb0M3K2g3YWdxWThqY2puNGFyMDBDRzB3d2VGUFFGWWZCWFkwd3lLNjlHb0YKWm1UWmdpZWpTOXExVkJXTGFOalU2emw2Z1EwZk9KdEtlQWJqZ3RtRElzMXVqSjVkWm1Xekh4SDc3aXMyOStoaQpCVStHcGhGZ3FRRWlhNkpFSXNrMlFybXh1WmRiL0g3OXVsT1F5eW1qZDNiTGZBaEtSRFpQdWs3NzFtY1l0SHFUCjN3dzNXSlAwVGNUNjRiZUIwNDZseVNyazBkd0VCMmlEcFN1a3BPT0k1V0I2ZENBOU1KbVpkMUxmNlczaDNxUVAKaGVzUWdUTTFOczZpMlg0aEgzM3VoOWRnME9nQVZ3ZGg5UTU2a3pxRk1sbXV1YkxYbU5DQ3pmZWdYaktGTXJHaQoxOE1CcE1kMG45dWViMDBnaS9vUEJ6YnpQaFdoQWdNQkFBR2pnZE13Z2RBd0RnWURWUjBQQVFIL0JBUURBZ1dnCk1CMEdBMVVkSlFRV01CUUdDQ3NHQVFVRkJ3TUJCZ2dyQmdFRkJRY0RBakFNQmdOVkhSTUJBZjhFQWpBQU1CMEcKQTFVZERnUVdCQlEvZVBzMitwRFo5NG82bXFiMVY2OWZHRkFLaFRBZkJnTlZIU01FR0RBV2dCVFNDYkEyVmRZawpUOWJGZmtLVWsxMFNNRWg2SmpCUkJnTlZIUkVFU2pCSWdneHJPSE10YldGemRHVnlNREdDREdzNGN5MXRZWE4wClpYSXdNb0lNYXpoekxXMWhjM1JsY2pBemh3UUtBQUFMaHdRS0FBQU1od1FLQUFBTmh3UUtBQUFPaHdRS0FBQVAKTUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFBYzdEOTJIV1Z2QlhQei9CelFuSm9BaVdUWjQ0aDFCdWxTVWhOdQpiVXRxQ2x4a1NJOHYxbUwya0hRMTFFU2l2UkxQQTM1L05pZ1pVSW5rbm1XSVdFeFVFbHVYNGlleVpnS3h4eWhsClNxVG9OSEdRYUFla0p3eHJqU0xqVEptUEhyRjVySTNPSW1qRmFLTk1RQ1l4VDFYdWVxenZJYmU5ak9ibDVCVWQKQkp4SkY1eEtOdjVuYkJCb2UvZHY1MEtGZWVxZ0IzN2NpOFlIMTFrRm9WWlNhQ0dPVWxFdmcvSW93Uy9QdkVqMQo2OXZYdmVLVFNrV0lRQWFlYjhEYUN4b0ZJeWpkV29RU0lmM05nQ2ZBU2dVNjJIczE3WGJLN1RFQjd3cUczUjNrCjVpeVFkYTBWYS9jS2sxOEdROWRmeWJSdkpWNE56SitBaVJ6Ukt6MndueXhzU29oNAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  19.   etcd-ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURxRENDQXBDZ0F3SUJBZ0lVRUM0R0tCWHA3UW9Ic0R3dnp4bVc2Z2NKTXJVd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2JERUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0ZOb1lXNW5hR0ZwTVJFd0R3WURWUVFIRXdoVAphR0Z1WjJoaGFURU5NQXNHQTFVRUNoTUVaWFJqWkRFV01CUUdBMVVFQ3hNTlJYUmpaQ0JUWldOMWNtbDBlVEVRCk1BNEdBMVVFQXhNSFpYUmpaQ0JEUVRBZUZ3MHlNakEwTWpNd01UVTNNREJhRncweU56QTBNakl3TVRVM01EQmEKTUd3eEN6QUpCZ05WQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hUYUdGdVoyaGhhVEVSTUE4R0ExVUVCeE1JVTJoaApibWRvWVdreERUQUxCZ05WQkFvVEJHVjBZMlF4RmpBVUJnTlZCQXNURFVWMFkyUWdVMlZqZFhKcGRIa3hFREFPCkJnTlZCQU1UQjJWMFkyUWdRMEV3Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRREEKQ3hmN2xIck5JdFlSeXhYbjM2cnovcDUyVnc3WHJST1JqUnlRQUZvZXU5SUgydHhQcjJ6WUxEdnN4Q0dRUU01OQpwWnQxOW9KTWcvUFpSLzR5VWg4Sm5jaXM2eWdzN2pPbEhjM2lSeStOdWFlV21qT2VxOEEzRnMvdlhFK3d0K0gyCk4zdlpCdXppQVNkL3UzMVRqMXhZcTlHdGJHNXRmd2swVmRibEhnV3JPQ2ZsTEtIdEFlQjRzOU1UZlhjaW1wbXkKNENUMmRTNTdxWitjeC9EdS9kcStsbk5tQldIcHp2TFUwZEFpQWtEdmZMK0ZFTjBIVG9Tc1lkQk5xd0RVSThITwppU1pVakxlTlg5aDkySEZGQ2x2N3hkOGlZNFhTWThyMkpVM0wzYjRqRFRLUkNyWHIwS3QrRlJYUkhOcG9sV0VNCnpyWUVrNmErMDJ4TnJ4MThYaU9WQWdNQkFBR2pRakJBTUE0R0ExVWREd0VCL3dRRUF3SUJCakFQQmdOVkhSTUIKQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJUU0NiQTJWZFlrVDliRmZrS1VrMTBTTUVoNkpqQU5CZ2txaGtpRwo5dzBCQVFzRkFBT0NBUUVBWjVScTRqOTVFV2VMdGsvY2toTk5NVTN2eDh5Yll0NEZ2SXFpRHZaRDVheFMwdG1QCnFxZitpUzY4d3grZXVLeGEwMEQyY3NOeVZVZUc0WWhyREp2aHhOemlmclpGMFp2cmtmTC9CMWtLSU9ZZk1aMWEKMkdNR1BkOUJmUXUvOHZPdXMrKys4dllnQVh1ZWVpUTZQQkRNYTl1R3hJd1h1RkJQYTV2cnlaUDE5RGNWazdjOApVcFlkSmtMRDRhWEVVMkcvTEdHNXdXa3IxOHdhTmFDbmRzbUVrWnJsWWJsdHNucTYraXQ4OVBTSmFVTlMrVXRtCkhxQys1em9JOHBmU1Nqa3Z1dklMSlpOZVR0MmV3U2UyYnBWSUVRVVF5aGZpVGk1NE1aM0NNdHhDVFR2WUQ0bWkKbVlnWXNCN3EyRnk4Z0ZLRnBDVWcyYlZLc053WTlPdlljNVlVMnc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
  20. ---
  21. # Source: calico/templates/calico-config.yaml
  22. # This ConfigMap is used to configure a self-hosted Calico installation.
  23. kind: ConfigMap
  24. apiVersion: v1
  25. metadata:
  26.   name: calico-config
  27.   namespace: kube-system
  28. data:
  29.   # Configure this with the location of your etcd cluster.
  30.   etcd_endpoints: "https://10.0.0.11:2379,https://10.0.0.12:2379,https://10.0.0.13:2379"
  31.   # If you're using TLS enabled etcd uncomment the following.
  32.   # You must also populate the Secret below with these files.
  33.   etcd_ca: "/calico-secrets/etcd-ca"   # "/calico-secrets/etcd-ca"
  34.   etcd_cert: "/calico-secrets/etcd-cert" # "/calico-secrets/etcd-cert"
  35.   etcd_key: "/calico-secrets/etcd-key"  # "/calico-secrets/etcd-key"
  36.   # Typha is disabled.
  37.   typha_service_name: "none"
  38.   # Configure the backend to use.
  39.   calico_backend: "bird"
  41.   # Configure the MTU to use for workload interfaces and tunnels.
  42.   # By default, MTU is auto-detected, and explicitly setting this field should not be required.
  43.   # You can override auto-detection by providing a non-zero value.
  44.   veth_mtu: "0"
  46.   # The CNI network configuration to install on each node. The special
  47.   # values in this config will be automatically populated.
  48.   cni_network_config: |-
  49.     {
  50.       "name": "k8s-pod-network",
  51.       "cniVersion": "0.3.1",
  52.       "plugins": [
  53.         {
  54.           "type": "calico",
  55.           "log_level": "info",
  56.           "log_file_path": "/var/log/calico/cni/cni.log",
  57.           "etcd_endpoints": "__ETCD_ENDPOINTS__",
  58.           "etcd_key_file": "__ETCD_KEY_FILE__",
  59.           "etcd_cert_file": "__ETCD_CERT_FILE__",
  60.           "etcd_ca_cert_file": "__ETCD_CA_CERT_FILE__",
  61.           "mtu": __CNI_MTU__,
  62.           "ipam": {
  63.               "type": "calico-ipam"
  64.           },
  65.           "policy": {
  66.               "type": "k8s"
  67.           },
  68.           "kubernetes": {
  69.               "kubeconfig": "__KUBECONFIG_FILEPATH__"
  70.           }
  71.         },
  72.         {
  73.           "type": "portmap",
  74.           "snat": true,
  75.           "capabilities": {"portMappings": true}
  76.         },
  77.         {
  78.           "type": "bandwidth",
  79.           "capabilities": {"bandwidth": true}
  80.         }
  81.       ]
  82.     }
  84. ---
  85. # Source: calico/templates/calico-kube-controllers-rbac.yaml
  87. # Include a clusterrole for the kube-controllers component,
  88. # and bind it to the calico-kube-controllers serviceaccount.
  89. kind: ClusterRole
  90. apiVersion: rbac.authorization.k8s.io/v1
  91. metadata:
  92.   name: calico-kube-controllers
  93. rules:
  94.   # Pods are monitored for changing labels.
  95.   # The node controller monitors Kubernetes nodes.
  96.   # Namespace and serviceaccount labels are used for policy.
  97.   - apiGroups: [""]
  98.     resources:
  99.       - pods
  100.       - nodes
  101.       - namespaces
  102.       - serviceaccounts
  103.     verbs:
  104.       - watch
  105.       - list
  106.       - get
  107.   # Watch for changes to Kubernetes NetworkPolicies.
  108.   - apiGroups: ["networking.k8s.io"]
  109.     resources:
  110.       - networkpolicies
  111.     verbs:
  112.       - watch
  113.       - list
  114. ---
  115. kind: ClusterRoleBinding
  116. apiVersion: rbac.authorization.k8s.io/v1
  117. metadata:
  118.   name: calico-kube-controllers
  119. roleRef:
  120.   apiGroup: rbac.authorization.k8s.io
  121.   kind: ClusterRole
  122.   name: calico-kube-controllers
  123. subjects:
  124. - kind: ServiceAccount
  125.   name: calico-kube-controllers
  126.   namespace: kube-system
  127. ---
  129. ---
  130. # Source: calico/templates/calico-node-rbac.yaml
  131. # Include a clusterrole for the calico-node DaemonSet,
  132. # and bind it to the calico-node serviceaccount.
  133. kind: ClusterRole
  134. apiVersion: rbac.authorization.k8s.io/v1
  135. metadata:
  136.   name: calico-node
  137. rules:
  138.   # The CNI plugin needs to get pods, nodes, and namespaces.
  139.   - apiGroups: [""]
  140.     resources:
  141.       - pods
  142.       - nodes
  143.       - namespaces
  144.     verbs:
  145.       - get
  146.   # EndpointSlices are used for Service-based network policy rule
  147.   # enforcement.
  148.   - apiGroups: ["discovery.k8s.io"]
  149.     resources:
  150.       - endpointslices
  151.     verbs:
  152.       - watch
  153.       - list
  154.   - apiGroups: [""]
  155.     resources:
  156.       - endpoints
  157.       - services
  158.     verbs:
  159.       # Used to discover service IPs for advertisement.
  160.       - watch
  161.       - list
  162.   # Pod CIDR auto-detection on kubeadm needs access to config maps.
  163.   - apiGroups: [""]
  164.     resources:
  165.       - configmaps
  166.     verbs:
  167.       - get
  168.   - apiGroups: [""]
  169.     resources:
  170.       - nodes/status
  171.     verbs:
  172.       # Needed for clearing NodeNetworkUnavailable flag.
  173.       - patch
  175. ---
  176. apiVersion: rbac.authorization.k8s.io/v1
  177. kind: ClusterRoleBinding
  178. metadata:
  179.   name: calico-node
  180. roleRef:
  181.   apiGroup: rbac.authorization.k8s.io
  182.   kind: ClusterRole
  183.   name: calico-node
  184. subjects:
  185. - kind: ServiceAccount
  186.   name: calico-node
  187.   namespace: kube-system
  189. ---
  190. # Source: calico/templates/calico-node.yaml
  191. # This manifest installs the calico-node container, as well
  192. # as the CNI plugins and network config on
  193. # each master and worker node in a Kubernetes cluster.
  194. kind: DaemonSet
  195. apiVersion: apps/v1
  196. metadata:
  197.   name: calico-node
  198.   namespace: kube-system
  199.   labels:
  200.     k8s-app: calico-node
  201. spec:
  202.   selector:
  203.     matchLabels:
  204.       k8s-app: calico-node
  205.   updateStrategy:
  206.     type: RollingUpdate
  207.     rollingUpdate:
  208.       maxUnavailable: 1
  209.   template:
  210.     metadata:
  211.       labels:
  212.         k8s-app: calico-node
  213.     spec:
  214.       nodeSelector:
  215.         kubernetes.io/os: linux
  216.       hostNetwork: true
  217.       tolerations:
  218.         # Make sure calico-node gets scheduled on all nodes.
  219.         - effect: NoSchedule
  220.           operator: Exists
  221.         # Mark the pod as a critical add-on for rescheduling.
  222.         - key: CriticalAddonsOnly
  223.           operator: Exists
  224.         - effect: NoExecute
  225.           operator: Exists
  226.       serviceAccountName: calico-node
  227.       # Minimize downtime during a rolling upgrade or deletion; tell Kubernetes to do a "force
  228.       # deletion": https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods.
  229.       terminationGracePeriodSeconds: 0
  230.       priorityClassName: system-node-critical
  231.       initContainers:
  232.         # This container installs the CNI binaries
  233.         # and CNI network config file on each node.
  234.         - name: install-cni
  235.           image: registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-cni:v3.22.2
  236.           command: ["/opt/cni/bin/install"]
  237.           envFrom:
  238.           - configMapRef:
  239.               # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
  240.               name: kubernetes-services-endpoint
  241.               optional: true
  242.           env:
  243.             # Name of the CNI config file to create.
  244.             - name: CNI_CONF_NAME
  245.               value: "10-calico.conflist"
  246.             # The CNI network config to install on each node.
  247.             - name: CNI_NETWORK_CONFIG
  248.               valueFrom:
  249.                 configMapKeyRef:
  250.                   name: calico-config
  251.                   key: cni_network_config
  252.             # The location of the etcd cluster.
  253.             - name: ETCD_ENDPOINTS
  254.               valueFrom:
  255.                 configMapKeyRef:
  256.                   name: calico-config
  257.                   key: etcd_endpoints
  258.             # CNI MTU Config variable
  259.             - name: CNI_MTU
  260.               valueFrom:
  261.                 configMapKeyRef:
  262.                   name: calico-config
  263.                   key: veth_mtu
  264.             # Prevents the container from sleeping forever.
  265.             - name: SLEEP
  266.               value: "false"
  267.           volumeMounts:
  268.             - mountPath: /host/opt/cni/bin
  269.               name: cni-bin-dir
  270.             - mountPath: /host/etc/cni/net.d
  271.               name: cni-net-dir
  272.             - mountPath: /calico-secrets
  273.               name: etcd-certs
  274.           securityContext:
  275.             privileged: true
  276.         # Adds a Flex Volume Driver that creates a per-pod Unix Domain Socket to allow Dikastes
  277.         # to communicate with Felix over the Policy Sync API.
  278.         - name: flexvol-driver
  279.           image: registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-pod2daemon-flexvol:v3.22.2
  280.           volumeMounts:
  281.           - name: flexvol-driver-host
  282.             mountPath: /host/driver
  283.           securityContext:
  284.             privileged: true
  285.       containers:
  286.         # Runs calico-node container on each Kubernetes node. This
  287.         # container programs network policy and routes on each
  288.         # host.
  289.         - name: calico-node
  290.           image: registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-node:v3.22.2
  291.           envFrom:
  292.           - configMapRef:
  293.               # Allow KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT to be overridden for eBPF mode.
  294.               name: kubernetes-services-endpoint
  295.               optional: true
  296.           env:
  297.             # The location of the etcd cluster.
  298.             - name: ETCD_ENDPOINTS
  299.               valueFrom:
  300.                 configMapKeyRef:
  301.                   name: calico-config
  302.                   key: etcd_endpoints
  303.             # Location of the CA certificate for etcd.
  304.             - name: ETCD_CA_CERT_FILE
  305.               valueFrom:
  306.                 configMapKeyRef:
  307.                   name: calico-config
  308.                   key: etcd_ca
  309.             # Location of the client key for etcd.
  310.             - name: ETCD_KEY_FILE
  311.               valueFrom:
  312.                 configMapKeyRef:
  313.                   name: calico-config
  314.                   key: etcd_key
  315.             # Location of the client certificate for etcd.
  316.             - name: ETCD_CERT_FILE
  317.               valueFrom:
  318.                 configMapKeyRef:
  319.                   name: calico-config
  320.                   key: etcd_cert
  321.             # Set noderef for node controller.
  322.             - name: CALICO_K8S_NODE_REF
  323.               valueFrom:
  324.                 fieldRef:
  325.                   fieldPath: spec.nodeName
  326.             # Choose the backend to use.
  327.             - name: CALICO_NETWORKING_BACKEND
  328.               valueFrom:
  329.                 configMapKeyRef:
  330.                   name: calico-config
  331.                   key: calico_backend
  332.             # Cluster type to identify the deployment type
  333.             - name: CLUSTER_TYPE
  334.               value: "k8s,bgp"
  335.             # Auto-detect the BGP IP address.
  336.             - name: IP
  337.               value: "autodetect"
  338.             # Enable IPIP
  339.             - name: CALICO_IPV4POOL_IPIP
  340.               value: "Always"
  341.             # Enable or Disable VXLAN on the default IP pool.
  342.             - name: CALICO_IPV4POOL_VXLAN
  343.               value: "Never"
  344.             # Set MTU for tunnel device used if ipip is enabled
  345.             - name: FELIX_IPINIPMTU
  346.               valueFrom:
  347.                 configMapKeyRef:
  348.                   name: calico-config
  349.                   key: veth_mtu
  350.             # Set MTU for the VXLAN tunnel device.
  351.             - name: FELIX_VXLANMTU
  352.               valueFrom:
  353.                 configMapKeyRef:
  354.                   name: calico-config
  355.                   key: veth_mtu
  356.             # Set MTU for the Wireguard tunnel device.
  357.             - name: FELIX_WIREGUARDMTU
  358.               valueFrom:
  359.                 configMapKeyRef:
  360.                   name: calico-config
  361.                   key: veth_mtu
  362.             # The default IPv4 pool to create on startup if none exists. Pod IPs will be
  363.             # chosen from this range. Changing this value after installation will have
  364.             # no effect. This should fall within `--cluster-cidr`.
  365.             - name: CALICO_IPV4POOL_CIDR
  366.               value: "172.16.0.0/12"
  367.             # Disable file logging so `kubectl logs` works.
  368.             - name: CALICO_DISABLE_FILE_LOGGING
  369.               value: "true"
  370.             # Set Felix endpoint to host default action to ACCEPT.
  371.             - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
  372.               value: "ACCEPT"
  373.             # Disable IPv6 on Kubernetes.
  374.             - name: FELIX_IPV6SUPPORT
  375.               value: "false"
  376.             - name: FELIX_HEALTHENABLED
  377.               value: "true"
  378.           securityContext:
  379.             privileged: true
  380.           resources:
  381.             requests:
  382.               cpu: 250m
  383.           lifecycle:
  384.             preStop:
  385.               exec:
  386.                 command:
  387.                 - /bin/calico-node
  388.                 - -shutdown
  389.           livenessProbe:
  390.             exec:
  391.               command:
  392.               - /bin/calico-node
  393.               - -felix-live
  394.               - -bird-live
  395.             periodSeconds: 10
  396.             initialDelaySeconds: 10
  397.             failureThreshold: 6
  398.             timeoutSeconds: 10
  399.           readinessProbe:
  400.             exec:
  401.               command:
  402.               - /bin/calico-node
  403.               - -felix-ready
  404.               - -bird-ready
  405.             periodSeconds: 10
  406.             timeoutSeconds: 10
  407.           volumeMounts:
  408.             # For maintaining CNI plugin API credentials.
  409.             - mountPath: /host/etc/cni/net.d
  410.               name: cni-net-dir
  411.               readOnly: false
  412.             - mountPath: /lib/modules
  413.               name: lib-modules
  414.               readOnly: true
  415.             - mountPath: /run/xtables.lock
  416.               name: xtables-lock
  417.               readOnly: false
  418.             - mountPath: /var/run/calico
  419.               name: var-run-calico
  420.               readOnly: false
  421.             - mountPath: /var/lib/calico
  422.               name: var-lib-calico
  423.               readOnly: false
  424.             - mountPath: /calico-secrets
  425.               name: etcd-certs
  426.             - name: policysync
  427.               mountPath: /var/run/nodeagent
  428.             # For eBPF mode, we need to be able to mount the BPF filesystem at /sys/fs/bpf so we mount in the
  429.             # parent directory.
  430.             - name: sysfs
  431.               mountPath: /sys/fs/
  432.               # Bidirectional means that, if we mount the BPF filesystem at /sys/fs/bpf it will propagate to the host.
  433.               # If the host is known to mount that filesystem already then Bidirectional can be omitted.
  434.               mountPropagation: Bidirectional
  435.             - name: cni-log-dir
  436.               mountPath: /var/log/calico/cni
  437.               readOnly: true
  438.       volumes:
  439.         # Used by calico-node.
  440.         - name: lib-modules
  441.           hostPath:
  442.             path: /lib/modules
  443.         - name: var-run-calico
  444.           hostPath:
  445.             path: /var/run/calico
  446.         - name: var-lib-calico
  447.           hostPath:
  448.             path: /var/lib/calico
  449.         - name: xtables-lock
  450.           hostPath:
  451.             path: /run/xtables.lock
  452.             type: FileOrCreate
  453.         - name: sysfs
  454.           hostPath:
  455.             path: /sys/fs/
  456.             type: DirectoryOrCreate
  457.         # Used to install CNI.
  458.         - name: cni-bin-dir
  459.           hostPath:
  460.             path: /opt/cni/bin
  461.         - name: cni-net-dir
  462.           hostPath:
  463.             path: /etc/cni/net.d
  464.         # Used to access CNI logs.
  465.         - name: cni-log-dir
  466.           hostPath:
  467.             path: /var/log/calico/cni
  468.         # Mount in the etcd TLS secrets with mode 400.
  469.         # See https://kubernetes.io/docs/concepts/configuration/secret/
  470.         - name: etcd-certs
  471.           secret:
  472.             secretName: calico-etcd-secrets
  473.             defaultMode: 0400
  474.         # Used to create per-pod Unix Domain Sockets
  475.         - name: policysync
  476.           hostPath:
  477.             type: DirectoryOrCreate
  478.             path: /var/run/nodeagent
  479.         # Used to install Flex Volume Driver
  480.         - name: flexvol-driver-host
  481.           hostPath:
  482.             type: DirectoryOrCreate
  483.             path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
  484. ---
  486. apiVersion: v1
  487. kind: ServiceAccount
  488. metadata:
  489.   name: calico-node
  490.   namespace: kube-system
  492. ---
  493. # Source: calico/templates/calico-kube-controllers.yaml
  494. # See https://github.com/projectcalico/kube-controllers
  495. apiVersion: apps/v1
  496. kind: Deployment
  497. metadata:
  498.   name: calico-kube-controllers
  499.   namespace: kube-system
  500.   labels:
  501.     k8s-app: calico-kube-controllers
  502. spec:
  503.   # The controllers can only have a single active instance.
  504.   replicas: 1
  505.   selector:
  506.     matchLabels:
  507.       k8s-app: calico-kube-controllers
  508.   strategy:
  509.     type: Recreate
  510.   template:
  511.     metadata:
  512.       name: calico-kube-controllers
  513.       namespace: kube-system
  514.       labels:
  515.         k8s-app: calico-kube-controllers
  516.     spec:
  517.       nodeSelector:
  518.         kubernetes.io/os: linux
  519.       tolerations:
  520.         # Mark the pod as a critical add-on for rescheduling.
  521.         - key: CriticalAddonsOnly
  522.           operator: Exists
  523.         - key: node-role.kubernetes.io/master
  524.           effect: NoSchedule
  525.       serviceAccountName: calico-kube-controllers
  526.       priorityClassName: system-cluster-critical
  527.       # The controllers must run in the host network namespace so that
  528.       # it isn't governed by policy that would prevent it from working.
  529.       hostNetwork: true
  530.       containers:
  531.         - name: calico-kube-controllers
  532.           image: registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/calico-kube-controllers:v3.22.2
  533.           env:
  534.             # The location of the etcd cluster.
  535.             - name: ETCD_ENDPOINTS
  536.               valueFrom:
  537.                 configMapKeyRef:
  538.                   name: calico-config
  539.                   key: etcd_endpoints
  540.             # Location of the CA certificate for etcd.
  541.             - name: ETCD_CA_CERT_FILE
  542.               valueFrom:
  543.                 configMapKeyRef:
  544.                   name: calico-config
  545.                   key: etcd_ca
  546.             # Location of the client key for etcd.
  547.             - name: ETCD_KEY_FILE
  548.               valueFrom:
  549.                 configMapKeyRef:
  550.                   name: calico-config
  551.                   key: etcd_key
  552.             # Location of the client certificate for etcd.
  553.             - name: ETCD_CERT_FILE
  554.               valueFrom:
  555.                 configMapKeyRef:
  556.                   name: calico-config
  557.                   key: etcd_cert
  558.             # Choose which controllers to run.
  559.             - name: ENABLED_CONTROLLERS
  560.               value: policy,namespace,serviceaccount,workloadendpoint,node
  561.           volumeMounts:
  562.             # Mount in the etcd TLS secrets.
  563.             - mountPath: /calico-secrets
  564.               name: etcd-certs
  565.           livenessProbe:
  566.             exec:
  567.               command:
  568.               - /usr/bin/check-status
  569.               - -l
  570.             periodSeconds: 10
  571.             initialDelaySeconds: 10
  572.             failureThreshold: 6
  573.             timeoutSeconds: 10
  574.           readinessProbe:
  575.             exec:
  576.               command:
  577.               - /usr/bin/check-status
  578.               - -r
  579.             periodSeconds: 10
  580.       volumes:
  581.         # Mount in the etcd TLS secrets with mode 400.
  582.         # See https://kubernetes.io/docs/concepts/configuration/secret/
  583.         - name: etcd-certs
  584.           secret:
  585.             secretName: calico-etcd-secrets
  586.             defaultMode: 0440
  588. ---
  590. apiVersion: v1
  591. kind: ServiceAccount
  592. metadata:
  593.   name: calico-kube-controllers
  594.   namespace: kube-system
  596. ---
  598. # This manifest creates a Pod Disruption Budget for Controller to allow K8s Cluster Autoscaler to evict
  600. apiVersion: policy/v1beta1
  601. kind: PodDisruptionBudget
  602. metadata:
  603.   name: calico-kube-controllers
  604.   namespace: kube-system
  605.   labels:
  606.     k8s-app: calico-kube-controllers
  607. spec:
  608.   maxUnavailable: 1
  609.   selector:
  610.     matchLabels:
  611.       k8s-app: calico-kube-controllers
  613. ---
  614. # Source: calico/templates/calico-typha.yaml
  616. ---
  617. # Source: calico/templates/configure-canal.yaml
  619. ---
  620. # Source: calico/templates/kdd-crds.yaml

10.3 应用 yaml 文件

  1. kubectl apply -f calico-etcd.yaml

10.4 查看集群

  1. # 等待calico pod全部正常运行
  2. [root@k8s-master01 ~]# kubectl get pod -n kube-system
  3. NAME                                       READY   STATUS    RESTARTS   AGE
  4. calico-kube-controllers-66b655f8f5-js6sd   1/1     Running   0          3h48m
  5. calico-node-2qmlw                          1/1     Running   0          3h48m
  6. calico-node-ffk6b                          1/1     Running   0          3h48m
  7. calico-node-lzkbd                          1/1     Running   0          3h48m
  8. calico-node-p4jdj                          1/1     Running   0          3h48m
  9. calico-node-s28pq                          1/1     Running   0          3h48m
  12. # 节点都变为Ready状态
  13. [root@k8s-master01 ~]# kubectl get nodes
  14. NAME           STATUS   ROLES    AGE     VERSION
  15. k8s-master01   Ready    <none>   7m56s   v1.21.11
  16. k8s-master02   Ready    <none>   4m52s   v1.21.11
  17. k8s-master03   Ready    <none>   4m49s   v1.21.11
  18. k8s-node01     Ready    <none>   4m5s    v1.21.11
  19. k8s-node02     Ready    <none>   4m2s    v1.21.11

11. 部署 CoreDNS

11.1 下载 yaml 文件

  1. curl https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/dns/coredns/coredns.yaml.sed -o coredns.yaml

11.2 修改 yaml 文件

  1. # 修改domain
  2. sed -i 's#$DNS_DOMAIN#cluster.local#g' coredns.yaml
  4. # 修改内存限制
  5. sed -i 's#$DNS_MEMORY_LIMIT#170Mi#g' coredns.yaml
  7. # 修改service ip
  8. sed -i 's#$DNS_SERVER_IP#192.168.0.10#g' coredns.yaml
  10. # 替换国内镜像
  11. sed -ri 's#(image:).*#\1 registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/coredns:1.8.6#g' coredns.yaml

下面为修改后的 yaml 文件：（注意 service ip，如果一样的话，可以直接使用此 yaml 文件）

  1. # Warning: This is a file generated from the base underscore template file: coredns.yaml.base
  3. apiVersion: v1
  4. kind: ServiceAccount
  5. metadata:
  6.   name: coredns
  7.   namespace: kube-system
  8.   labels:
  9.       kubernetes.io/cluster-service: "true"
  10.       addonmanager.kubernetes.io/mode: Reconcile
  11. ---
  12. apiVersion: rbac.authorization.k8s.io/v1
  13. kind: ClusterRole
  14. metadata:
  15.   labels:
  16.     kubernetes.io/bootstrapping: rbac-defaults
  17.     addonmanager.kubernetes.io/mode: Reconcile
  18.   name: system:coredns
  19. rules:
  20. - apiGroups:
  21.   - ""
  22.   resources:
  23.   - endpoints
  24.   - services
  25.   - pods
  26.   - namespaces
  27.   verbs:
  28.   - list
  29.   - watch
  30. - apiGroups:
  31.   - ""
  32.   resources:
  33.   - nodes
  34.   verbs:
  35.   - get
  36. - apiGroups:
  37.   - discovery.k8s.io
  38.   resources:
  39.   - endpointslices
  40.   verbs:
  41.   - list
  42.   - watch
  43. ---
  44. apiVersion: rbac.authorization.k8s.io/v1
  45. kind: ClusterRoleBinding
  46. metadata:
  47.   annotations:
  48.     rbac.authorization.kubernetes.io/autoupdate: "true"
  49.   labels:
  50.     kubernetes.io/bootstrapping: rbac-defaults
  51.     addonmanager.kubernetes.io/mode: EnsureExists
  52.   name: system:coredns
  53. roleRef:
  54.   apiGroup: rbac.authorization.k8s.io
  55.   kind: ClusterRole
  56.   name: system:coredns
  57. subjects:
  58. - kind: ServiceAccount
  59.   name: coredns
  60.   namespace: kube-system
  61. ---
  62. apiVersion: v1
  63. kind: ConfigMap
  64. metadata:
  65.   name: coredns
  66.   namespace: kube-system
  67.   labels:
  68.       addonmanager.kubernetes.io/mode: EnsureExists
  69. data:
  70.   Corefile: |
  71.     .:53 {
  72.         errors
  73.         health {
  74.             lameduck 5s
  75.         }
  76.         ready
  77.         kubernetes cluster.local in-addr.arpa ip6.arpa {
  78.             pods insecure
  79.             fallthrough in-addr.arpa ip6.arpa
  80.             ttl 30
  81.         }
  82.         prometheus :9153
  83.         forward . /etc/resolv.conf {
  84.             max_concurrent 1000
  85.         }
  86.         cache 30
  87.         loop
  88.         reload
  89.         loadbalance
  90.     }
  91. ---
  92. apiVersion: apps/v1
  93. kind: Deployment
  94. metadata:
  95.   name: coredns
  96.   namespace: kube-system
  97.   labels:
  98.     k8s-app: kube-dns
  99.     kubernetes.io/cluster-service: "true"
  100.     addonmanager.kubernetes.io/mode: Reconcile
  101.     kubernetes.io/name: "CoreDNS"
  102. spec:
  103.   # replicas: not specified here:
  104.   # 1. In order to make Addon Manager do not reconcile this replicas parameter.
  105.   # 2. Default is 1.
  106.   # 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.
  107.   strategy:
  108.     type: RollingUpdate
  109.     rollingUpdate:
  110.       maxUnavailable: 1
  111.   selector:
  112.     matchLabels:
  113.       k8s-app: kube-dns
  114.   template:
  115.     metadata:
  116.       labels:
  117.         k8s-app: kube-dns
  118.     spec:
  119.       securityContext:
  120.         seccompProfile:
  121.           type: RuntimeDefault
  122.       priorityClassName: system-cluster-critical
  123.       serviceAccountName: coredns
  124.       affinity:
  125.         podAntiAffinity:
  126.           preferredDuringSchedulingIgnoredDuringExecution:
  127.           - weight: 100
  128.             podAffinityTerm:
  129.               labelSelector:
  130.                 matchExpressions:
  131.                   - key: k8s-app
  132.                     operator: In
  133.                     values: ["kube-dns"]
  134.               topologyKey: kubernetes.io/hostname
  135.       tolerations:
  136.         - key: "CriticalAddonsOnly"
  137.           operator: "Exists"
  138.       nodeSelector:
  139.         kubernetes.io/os: linux
  140.       containers:
  141.       - name: coredns
  142.         image: registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/coredns:1.8.6
  143.         imagePullPolicy: IfNotPresent
  144.         resources:
  145.           limits:
  146.             memory: 170Mi
  147.           requests:
  148.             cpu: 100m
  149.             memory: 70Mi
  150.         args: [ "-conf", "/etc/coredns/Corefile" ]
  151.         volumeMounts:
  152.         - name: config-volume
  153.           mountPath: /etc/coredns
  154.           readOnly: true
  155.         ports:
  156.         - containerPort: 53
  157.           name: dns
  158.           protocol: UDP
  159.         - containerPort: 53
  160.           name: dns-tcp
  161.           protocol: TCP
  162.         - containerPort: 9153
  163.           name: metrics
  164.           protocol: TCP
  165.         livenessProbe:
  166.           httpGet:
  167.             path: /health
  168.             port: 8080
  169.             scheme: HTTP
  170.           initialDelaySeconds: 60
  171.           timeoutSeconds: 5
  172.           successThreshold: 1
  173.           failureThreshold: 5
  174.         readinessProbe:
  175.           httpGet:
  176.             path: /ready
  177.             port: 8181
  178.             scheme: HTTP
  179.         securityContext:
  180.           allowPrivilegeEscalation: false
  181.           capabilities:
  182.             add:
  183.             - NET_BIND_SERVICE
  184.             drop:
  185.             - all
  186.           readOnlyRootFilesystem: true
  187.       dnsPolicy: Default
  188.       volumes:
  189.         - name: config-volume
  190.           configMap:
  191.             name: coredns
  192.             items:
  193.             - key: Corefile
  194.               path: Corefile
  195. ---
  196. apiVersion: v1
  197. kind: Service
  198. metadata:
  199.   name: kube-dns
  200.   namespace: kube-system
  201.   annotations:
  202.     prometheus.io/port: "9153"
  203.     prometheus.io/scrape: "true"
  204.   labels:
  205.     k8s-app: kube-dns
  206.     kubernetes.io/cluster-service: "true"
  207.     addonmanager.kubernetes.io/mode: Reconcile
  208.     kubernetes.io/name: "CoreDNS"
  209. spec:
  210.   selector:
  211.     k8s-app: kube-dns
  212.   clusterIP: 192.168.0.10
  213.   ports:
  214.   - name: dns
  215.     port: 53
  216.     protocol: UDP
  217.   - name: dns-tcp
  218.     port: 53
  219.     protocol: TCP
  220.   - name: metrics
  221.     port: 9153
  222.     protocol: TCP

11.2 应用 yaml 文件

  1. [root@k8s-master01 ~]# kubectl apply -f coredns.yaml
  3. # 查看状态
  4. [root@k8s-master01 ~]# kubectl get pod -n kube-system -l k8s-app=kube-dns
  5. NAME                      READY   STATUS    RESTARTS   AGE
  6. coredns-cc6b9d95d-dm6sr   1/1     Running   0          11s

12. 部署 Metrics Server

Github项目地址

12.1 下载 yaml 文件

  1. wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

12.1 修改 yaml 文件

  1. # 修改为国内镜像源。
  2. sed -i -e '/image:/s#\(image: \).*#\1registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/metrics-server:v0.5.2#' components.yaml
  4. # 添加api聚合证书配置
  5. sed -i '/args:/a\        - --kubelet-insecure-tls' components.yaml
  6. sed -i '/args:/a\        - --requestheader-client-ca-file=/opt/kubernetes/certs/front-proxy-ca.pem' components.yaml
  7. sed -i '/args:/a\        - --requestheader-username-headers=X-Remote-User' components.yaml
  8. sed -i '/args:/a\        - --requestheader-group-headers=X-Remote-Group' components.yaml
  9. sed -i '/args:/a\        - --requestheader-extra-headers-prefix=X-Remote-Extra-' components.yaml
  11. # 将证书目录挂载到容器中
  12. sed -i '/volumes:/a\      - name: ca-ssl\n        hostPath:\n          path: /opt/kubernetes/certs' components.yaml
  13. sed -i '/volumeMounts:/a\        - name: ca-ssl\n          mountPath: /opt/kubernetes/certs' components.yaml

下面为修改后的 yaml 文件：

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   labels:
  5.     k8s-app: metrics-server
  6.   name: metrics-server
  7.   namespace: kube-system
  8. ---
  9. apiVersion: rbac.authorization.k8s.io/v1
  10. kind: ClusterRole
  11. metadata:
  12.   labels:
  13.     k8s-app: metrics-server
  14.     rbac.authorization.k8s.io/aggregate-to-admin: "true"
  15.     rbac.authorization.k8s.io/aggregate-to-edit: "true"
  16.     rbac.authorization.k8s.io/aggregate-to-view: "true"
  17.   name: system:aggregated-metrics-reader
  18. rules:
  19. - apiGroups:
  20.   - metrics.k8s.io
  21.   resources:
  22.   - pods
  23.   - nodes
  24.   verbs:
  25.   - get
  26.   - list
  27.   - watch
  28. ---
  29. apiVersion: rbac.authorization.k8s.io/v1
  30. kind: ClusterRole
  31. metadata:
  32.   labels:
  33.     k8s-app: metrics-server
  34.   name: system:metrics-server
  35. rules:
  36. - apiGroups:
  37.   - ""
  38.   resources:
  39.   - pods
  40.   - nodes
  41.   - nodes/stats
  42.   - namespaces
  43.   - configmaps
  44.   verbs:
  45.   - get
  46.   - list
  47.   - watch
  48. ---
  49. apiVersion: rbac.authorization.k8s.io/v1
  50. kind: RoleBinding
  51. metadata:
  52.   labels:
  53.     k8s-app: metrics-server
  54.   name: metrics-server-auth-reader
  55.   namespace: kube-system
  56. roleRef:
  57.   apiGroup: rbac.authorization.k8s.io
  58.   kind: Role
  59.   name: extension-apiserver-authentication-reader
  60. subjects:
  61. - kind: ServiceAccount
  62.   name: metrics-server
  63.   namespace: kube-system
  64. ---
  65. apiVersion: rbac.authorization.k8s.io/v1
  66. kind: ClusterRoleBinding
  67. metadata:
  68.   labels:
  69.     k8s-app: metrics-server
  70.   name: metrics-server:system:auth-delegator
  71. roleRef:
  72.   apiGroup: rbac.authorization.k8s.io
  73.   kind: ClusterRole
  74.   name: system:auth-delegator
  75. subjects:
  76.   - kind: ServiceAccount
  77.     name: metrics-server
  78.     namespace: kube-system
  79. ---
  80. apiVersion: rbac.authorization.k8s.io/v1
  81. kind: ClusterRoleBinding
  82. metadata:
  83.   labels:
  84.     k8s-app: metrics-server
  85.   name: system:metrics-server
  86. roleRef:
  87.   apiGroup: rbac.authorization.k8s.io
  88.   kind: ClusterRole
  89.   name: system:metrics-server
  90. subjects:
  91.   - kind: ServiceAccount
  92.     name: metrics-server
  93.     namespace: kube-system
  94. ---
  95. apiVersion: v1
  96. kind: Service
  97. metadata:
  98.   labels:
  99.     k8s-app: metrics-server
  100.   name: metrics-server
  101.   namespace: kube-system
  102. spec:
  103.   ports:
  104.     - name: https
  105.       port: 443
  106.       protocol: TCP
  107.       targetPort: https
  108.   selector:
  109.     k8s-app: metrics-server
  110. ---
  111. apiVersion: apps/v1
  112. kind: Deployment
  113. metadata:
  114.   labels:
  115.     k8s-app: metrics-server
  116.   name: metrics-server
  117.   namespace: kube-system
  118. spec:
  119.   selector:
  120.     matchLabels:
  121.       k8s-app: metrics-server
  122.   strategy:
  123.     rollingUpdate:
  124.       maxUnavailable: 0
  125.   template:
  126.     metadata:
  127.       labels:
  128.         k8s-app: metrics-server
  129.     spec:
  130.       containers:
  131.         - args:
  132.           - --cert-dir=/tmp
  133.           - --secure-port=4443
  134.           - --metric-resolution=30s
  135.           - --kubelet-insecure-tls
  136.           - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  137.           - --requestheader-client-ca-file=/opt/kubernetes/certs/front-proxy-ca.pem
  138.           - --requestheader-username-headers=X-Remote-User
  139.           - --requestheader-group-headers=X-Remote-Group
  140.           - --requestheader-extra-headers-prefix=X-Remote-Extra-
  141.           image: registry.cn-shanghai.aliyuncs.com/wuvikr-k8s/metrics-server:v0.5.2
  142.           imagePullPolicy: IfNotPresent
  143.           livenessProbe:
  144.             failureThreshold: 3
  145.             httpGet:
  146.               path: /livez
  147.               port: https
  148.               scheme: HTTPS
  149.             periodSeconds: 10
  150.           name: metrics-server
  151.           ports:
  152.           - containerPort: 4443
  153.             name: https
  154.             protocol: TCP
  155.           readinessProbe:
  156.             failureThreshold: 3
  157.             httpGet:
  158.               path: /readyz
  159.               port: https
  160.               scheme: HTTPS
  161.             periodSeconds: 10
  162.           securityContext:
  163.             readOnlyRootFilesystem: true
  164.             runAsNonRoot: true
  165.             runAsUser: 1000
  166.           volumeMounts:
  167.           - mountPath: /tmp
  168.             name: tmp-dir
  169.           - name: ca-ssl
  170.             mountPath: /opt/kubernetes/certs
  171.       nodeSelector:
  172.         kubernetes.io/os: linux
  173.       priorityClassName: system-cluster-critical
  174.       serviceAccountName: metrics-server
  175.       volumes:
  176.       - emptyDir: {}
  177.         name: tmp-dir
  178.       - name: ca-ssl
  179.           hostPath:
  180.             path: /opt/kubernetes/certs
  181. ---
  182. apiVersion: apiregistration.k8s.io/v1
  183. kind: APIService
  184. metadata:
  185.   labels:
  186.     k8s-app: metrics-server
  187.   name: v1beta1.metrics.k8s.io
  188. spec:
  189.   group: metrics.k8s.io
  190.   groupPriorityMinimum: 100
  191.   insecureSkipTLSVerify: true
  192.   service:
  193.     name: metrics-server
  194.     namespace: kube-system
  195.   version: v1beta1
  196.   versionPriority: 100

12.3 应用 yaml 文件

  1. [root@k8s-master01 ~]# kubectl apply -f components.yaml
  3. # 查看pod状态
  4. [root@k8s-master01 ~]# kubectl get pod -n kube-system -l k8s-app=metrics-server
  5. NAME                              READY   STATUS    RESTARTS   AGE
  6. metrics-server-6f687df468-8p5vr   1/1     Running   0          5m

13. 集群验证

集群搭建完成后，必须验证集群是否可用，一般有下面几个验证步骤：

  1. Pod 必须能解析 Service
  2. Pod 必须能解析跨 namespace 的 Service
  3. 每个节点都必须要能访问 Kubernetes 的 kubernetes svc 443 和 kube-dns 的 service 53 端口
  4. Pod 和 Pod 之前能直接通信
    1. 同namespace能通信
    2. 跨namespace能通信
    3. 跨节点能通信 ```bash

      运行busybox容器

      [root@k8s-master01 ~]# kubectl run busybox —image=busybox:1.28 — sleep 24h

解析kubernetes

[root@k8s-master01 ~]# kubectl exec busybox — nslookup kubernetes Server: 192.168.0.10 Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name: kubernetes Address 1: 192.168.0.1 kubernetes.default.svc.cluster.local

解析kube-dns.kube-system

[root@k8s-master01 ~]# kubectl exec buxybox — nslookup kube-dns.kube-system Server: 192.168.0.10 Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

Name: kube-dns.kube-system Address 1: 192.168.0.10 kube-dns.kube-system.svc.cluster.local

  1.     创建一个nginx deployment 服务：
  2. ```bash
  3. kubectl create deployment nginx --image=nginx
  5. kubectl expose deployment nginx --port=80 --type=NodePort
  7. kubectl get pod,svc
  9. curl <pod-ip>:80