1、安装要求
2、准备环境
3、操作系统初始化配置
4、部署Etcd集群
5、安装Docker
6、部署Master Node
7、部署Worker Node
8、测试kubernetes集群
9、附件

1、安装要求

在开始之前，部署 Kubernetes 集群机器需要满足以下几个条件：

一台或多台机器，操作系统 CentOS7.x-86_x64，安装：
硬件配置：2GB 或更多 RAM，2 个 CPU 或更多 CPU，硬盘 30GB 或更多
集群中所有机器之间网络互通
可以访问外网，需要拉取镜像，如果服务器不能上网，需要提前下载镜像并导入节点
禁止 swap 分区

2、准备环境

软件环境 | 软件 | 版本 | | —- | —- | | 操作系统 | CentOS-7-x86_64-DVD-2003.iso | | Docker | 19-ce | | Kubernetes | 1.18.3 |

服务器规划 | 角色 | IP | 组件 | | —- | —- | —- | | k8s-master | 192.168.127.200 | kube-apiserver，kube-controller-manager，kube -scheduler，docker，etcd | | k8s-node01 | 192.168.127.201 | kubelet，kube-proxy，docker，etcd | | k8s-node02 | 192.168.127.202 | kubelet，kube-proxy，docker，etcd |

3、操作系统初始化配置

在每台服务器上均需执行，如有特殊则特殊说明

#安装wget以便获取数据包
yum install wget
#关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
#关闭 selinux：
sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久
setenforce 0 # 临时
#关闭 swap：
sed -ri 's/.*swap.*/#&/' /etc/fstab # 永久
swapoff -a # 临时
#主机名（分别在对应服务器执行），后续用主机名代替ip地址说明：
hostnamectl set-hostname k8s-master #192.168.127.200
hostnamectl set-hostname k8s-node01 #192.168.127.201
hostnamectl set-hostname k8s-node02 #192.168.127.202
#在 k8s-master 添加 hosts：
cat >> /etc/hosts << EOF
192.168.127.200 k8s-master
192.168.127.201 k8s-node01
192.168.127.202 k8s-node02
EOF
#将桥接的 IPv4 流量传递到 iptables 的链：
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
# 生效
sysctl --system
#时间同步：
yum install ntpdate -y
ntpdate time.windows.com

4、部署Etcd集群

     Etcd 是一个分布式键值存储系统，Kubernetes 使用 Etcd 进行数据存储，所以先准备 一个 Etcd 数据库，为解决 Etcd 单点故障，应采用集群方式部署，这里使用 3 台组建集 群，可容忍 1 台机器故障，当然，也可以使用 5 台组建集群，可容忍 2 台机器故障。

节点名称	IP
etct-1	192.168.127.200
etct-2	192.168.127.201
etct-3	192.168.127.202

注：为了节省机器，这里与 K8s 节点机器复用。也可以独立于 k8s 集群之外部署，只要 apiserver 能连接到就行。

4.1 准备 cfssl 证书生成工具

cfssl 是一个开源的证书管理工具，使用 json 文件生成证书，相比 openssl 更方便使用。找任意一台服务器操作，这里用 Master 节点。

[root@k8s-master ~]# https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
[root@k8s-master ~]# https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
[root@k8s-master ~]# https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
[root@k8s-master ~]# chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
[root@k8s-master ~]# mv cfssl_linux-amd64 /usr/local/bin/cfssl
[root@k8s-master ~]# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
[root@k8s-master ~]# mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

4.2 生成Etcd证书

4.2.1 自签证书颁发机构（CA）

[root@k8s-master ~]# mkdir -p ~/TLS/{etcd,k8s}
[root@k8s-master ~]# cd TLS/etcd/

自签CA

[root@k8s-master etcd]# cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

[root@k8s-master etcd]# cat > ca-csr.json<< EOF 
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成证书

[root@k8s-master etcd]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/12/06 15:23:36 [INFO] generating a new CA key and certificate from CSR
2021/12/06 15:23:36 [INFO] generate received request
2021/12/06 15:23:36 [INFO] received CSR
2021/12/06 15:23:36 [INFO] generating key: rsa-2048
2021/12/06 15:23:37 [INFO] encoded CSR
2021/12/06 15:23:37 [INFO] signed certificate with serial number 182545974887289596052662432759196280377550212480

[root@k8s-master etcd]# ls ca*pem
ca-key.pem  ca.pem

4.2.2 使用自签CA签发Etcd HTTPS证书

创建证书申请文件

[root@k8s-master etcd]# cat > server-csr.json << EOF
{
  "CN": "etcd",
  "hosts": [
    "192.168.127.200",      
    "192.168.127.201",
    "192.168.127.202"
  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing"
    }
  ]
}
EOF

上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP，一个都不能少！为了方便后期扩容可以多写几个预留的 IP。

生成证书

[root@k8s-master etcd]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/12/06 15:27:39 [INFO] generate received request
2021/12/06 15:27:39 [INFO] received CSR
2021/12/06 15:27:39 [INFO] generating key: rsa-2048
2021/12/06 15:27:39 [INFO] encoded CSR
2021/12/06 15:27:39 [INFO] signed certificate with serial number 265604037498542186840673064278810513522697138227
2021/12/06 15:27:39 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

这里会有warning，不需要处理

[root@k8s-master etcd]# ls server*pem 
server-key.pem  server.pem

4.3 下载Etcd二进制文件

下载地址V3.4.9

版本列表

4.4 部署Etcd集群

4.4.1 创建工作目录并解压二进制包

[root@k8s-master etcd]# cd ~
[root@k8s-master ~]# mkdir /opt/etcd/{bin,cfg,ssl} -p
[root@k8s-master ~]# tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
[root@k8s-master ~]# mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

4.4.2 创建Etcd配置文件

[root@k8s-master ~]# cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.127.200:2380" 
ETCD_LISTEN_CLIENT_URLS="https://192.168.127.200:2379" 
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.127.200:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.127.200:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.127.200:2380,etcd-2=https://192.168.127.201:2380,etcd-3=https://192.168.127.202:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" 
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通信监听地址（当前服务器地址）

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址（当前服务器地址）

ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址（当前服务器地址）

ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址（当前服务器地址）

ETCD_INITIAL_CLUSTER：集群节点地址（所有集群节点）

ETCD_INITIAL_CLUSTER_TOKEN：集群 Token

ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new 是新集群，existing 表示加入已有集群

4.4.3 systemd管理Etcd

[root@k8s-master ~]# cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
    --cert-file=/opt/etcd/ssl/server.pem \
    --key-file=/opt/etcd/ssl/server-key.pem \
    --peer-cert-file=/opt/etcd/ssl/server.pem \
    --peer-key-file=/opt/etcd/ssl/server-key.pem \
    --trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
    --logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

4.4.4 拷贝刚刚生成的证书

[root@k8s-master ~]# cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl/

4.4.5 将上面节点1所有生成的文件拷贝到节点2和节点3

# 节点2
[root@k8s-master ~]# scp -r /opt/etcd/ root@k8s-node01:/opt/
[root@k8s-master ~]# scp /usr/lib/systemd/system/etcd.service root@k8s-node01:/usr/lib/systemd/system/
# 节点3
[root@k8s-master ~]# scp -r /opt/etcd/ root@k8s-node02:/opt/
[root@k8s-master ~]# scp /usr/lib/systemd/system/etcd.service root@k8s-node02:/usr/lib/systemd/system/

在节点 2 和节点 3 分别修改 etcd.conf 配置文件中的节点名称和当前服务器 IP（节点3参照节点2）：

[root@k8s-node01 ~]# vi /opt/etcd/cfg/etcd.conf 
#[Member]
ETCD_NAME="etcd-2" # 修改此处，节点 2 改为 etcd-2，节点 3 改为 etcd-3
ETCD_DATA_DIR="/var/lib/etcd/default.etcd" # 修改此处为当前服务器 IP
ETCD_LISTEN_PEER_URLS="https://192.168.127.201:2380" # 修改此处为当前服务器 IP
ETCD_LISTEN_CLIENT_URLS="https://192.168.127.201:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.127.201:2380" # 修改此处为当前服务器 IP
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.127.201:2379" # 修改此处为当前服务器 IP
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.127.200:2380,etcd-2=https://192.168.127.201:2380,etcd-3=https://192.168.127.202:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

4.4.6 启动Etcd服务并设置开机启动

首先启动节点2、节点3的Etcd服务（节点3参照节点2）

[root@k8s-node01 ~]# systemctl daemon-reload 
[root@k8s-node01 ~]# systemctl start etcd
[root@k8s-node01 ~]# systemctl enable etcd
Created symlink from /etc/systemd/system/multi-user.target.wants/etcd.service to /usr/lib/systemd/system/etcd.service.

查看启动状态

[root@k8s-node01 ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 15:51:52 CST; 8s ago
 Main PID: 9427 (etcd)
   CGroup: /system.slice/etcd.service
           └─9427 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/serv...

12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.023+0800","caller":"raft/raft.go:700","msg":"53cdd4e3f357...erm 497"}
12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.023+0800","caller":"raft/raft.go:960","msg":"53cdd4e3f357...erm 497"}
12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.029+0800","caller":"raft/node.go:325","msg":"raft.node: 5...erm 497"}
12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.093+0800","caller":"membership/cluster.go:558","msg":"set...n":"3.0"}
12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.093+0800","caller":"api/capability.go:76","msg":"enabled ...n":"3.0"}
12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.094+0800","caller":"etcdserver/server.go:2036","msg":"published lo...
12月 06 15:51:52 k8s-node01 systemd[1]: Started Etcd Server.
12月 06 15:51:52 k8s-node01 etcd[9427]: {"level":"info","ts":"2021-12-06T15:51:52.095+0800","caller":"embed/serve.go:191","msg":"serving cl...01:2379"}
12月 06 15:51:56 k8s-node01 etcd[9427]: {"level":"warn","ts":"2021-12-06T15:51:56.368+0800","caller":"rafthttp/probing_status.go:70","msg":...refused"}
12月 06 15:51:56 k8s-node01 etcd[9427]: {"level":"warn","ts":"2021-12-06T15:51:56.369+0800","caller":"rafthttp/probing_status.go:70","msg":"prober d...
Hint: Some lines were ellipsized, use -l to show in full.

回到节点1，启动Etcd服务

[root@k8s-master ~]# systemctl daemon-reload 
[root@k8s-master ~]# systemctl start etcd 
[root@k8s-master ~]# systemctl enable etcd

查看启动状态

[root@k8s-master ~]# systemctl status etcd
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 15:51:52 CST; 5min ago
 Main PID: 9514 (etcd)
   CGroup: /system.slice/etcd.service
           └─9514 /opt/etcd/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/serv...

12月 06 15:55:54 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:54.511+0800","caller":"rafthttp/stream.go:425","msg":"establ...4f23134"}
12月 06 15:55:54 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:54.511+0800","caller":"rafthttp/stream.go:425","msg":"establ...4f23134"}
12月 06 15:55:54 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:54.613+0800","caller":"rafthttp/stream.go:250","msg":"set me...gApp v2"}
12月 06 15:55:54 k8s-master etcd[9514]: {"level":"warn","ts":"2021-12-06T15:55:54.613+0800","caller":"rafthttp/stream.go:277","msg":"establ...4f23134"}
12月 06 15:55:54 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:54.635+0800","caller":"rafthttp/stream.go:250","msg":"set me...Message"}
12月 06 15:55:54 k8s-master etcd[9514]: {"level":"warn","ts":"2021-12-06T15:55:54.635+0800","caller":"rafthttp/stream.go:277","msg":"establ...4f23134"}
12月 06 15:55:56 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:56.417+0800","caller":"etcdserver/server.go:2536","msg":"upd...o":"3.4"}
12月 06 15:55:56 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:56.419+0800","caller":"membership/cluster.go:546","msg":"upd...m":"3.4"}
12月 06 15:55:56 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:56.419+0800","caller":"api/capability.go:76","msg":"enabled ...n":"3.4"}
12月 06 15:55:56 k8s-master etcd[9514]: {"level":"info","ts":"2021-12-06T15:55:56.419+0800","caller":"etcdserver/server.go:2559","msg":"clu...n":"3.4"}
Hint: Some lines were ellipsized, use -l to show in full.

这里建议先启动节点2、节点3，否则节点1无法启动

4.4.7 查看集群状态

在节点1上面查看集群状态

[root@k8s-master ~]# ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.127.200:2379,https://192.168.127.201:2379,https://192.168.127.202:2379" endpoint health
https://192.168.127.201:2379 is healthy: successfully committed proposal: took = 27.196098ms
https://192.168.127.200:2379 is healthy: successfully committed proposal: took = 27.339095ms
https://192.168.127.202:2379 is healthy: successfully committed proposal: took = 30.948846ms

如果输出上面信息，就说明集群部署成功。如果有问题第一步先看日志： /var/log/message 或 journalctl -u etcd

5、安装Docker

5.1 下载Docker二进制文件

下载地址V19

版本列表

5.2 解压二进制包

[root@k8s-master ~]# cd ~
[root@k8s-master ~]# tar zxvf docker-19.03.9.tgz
[root@k8s-master ~]# cp docker/* /usr/bin

5.3 systemd管理Docker

[root@k8s-master ~]# cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
EOF

5.4 创建配置文件

[root@k8s-master ~]# mkdir /etc/docker
[root@k8s-master ~]# cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://0dhl3431.mirror.aliyuncs.com"]
}
EOF

registry-mirrors 阿里云镜像加速器

5.5 启动并设置开机启动

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start docker
[root@k8s-master ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

查看服务启动状态

[root@k8s-master ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 16:09:23 CST; 31s ago
     Docs: https://docs.docker.com
 Main PID: 9623 (dockerd)
   CGroup: /system.slice/docker.service
           ├─9623 /usr/bin/dockerd
           └─9632 containerd --config /var/run/docker/containerd/containerd.toml --log-level info

12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.134959912+08:00" level=info msg="scheme \"unix\" not registered, fallba...ule=grpc
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.134992357+08:00" level=info msg="ccResolverWrapper: sending update to c...ule=grpc
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.135010000+08:00" level=info msg="ClientConn switching balancer to \"pic...ule=grpc
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.274383347+08:00" level=info msg="Loading containers: start."
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.673871567+08:00" level=info msg="Default bridge (docker0) is assigned w...address"
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.778964030+08:00" level=info msg="Loading containers: done."
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.826317651+08:00" level=info msg="Docker daemon" commit=9d988398e7 graph...=19.03.9
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.826524086+08:00" level=info msg="Daemon has completed initialization"
12月 06 16:09:23 k8s-master dockerd[9623]: time="2021-12-06T16:09:23.863387301+08:00" level=info msg="API listen on /var/run/docker.sock"
12月 06 16:09:23 k8s-master systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.

5.6 安装Docker至Worker Node节点

拷贝Docker文件至Node节点

# k8s-node01
[root@k8s-master ~]# cd ~
[root@k8s-master ~]# scp docker/* root@k8s-node01:/usr/bin/
[root@k8s-master ~]# scp /usr/lib/systemd/system/docker.service root@k8s-node01:/usr/lib/systemd/system/
# k8s-node02
[root@k8s-master ~]# scp docker/* root@k8s-node02:/usr/bin/
[root@k8s-master ~]# scp /usr/lib/systemd/system/docker.service root@k8s-node02:/usr/lib/systemd/system/

创建配置文件

分别在k8s-node01、k8s-node02创建配置文件

[root@k8s-node01 ~]# mkdir /etc/docker
[root@k8s-node01 ~]# cat > /etc/docker/daemon.json << EOF
{
  "registry-mirrors": ["https://0dhl3431.mirror.aliyuncs.com"]
}
EOF

启动并设置开机启动

[root@k8s-node01 ~]# systemctl daemon-reload
[root@k8s-node01 ~]# systemctl start docker
[root@k8s-node01 ~]# systemctl enable docker
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

查看服务启动状态

[root@k8s-node01 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 16:19:18 CST; 1min 8s ago
     Docs: https://docs.docker.com
 Main PID: 9541 (dockerd)
   CGroup: /system.slice/docker.service
           ├─9541 /usr/bin/dockerd
           └─9550 containerd --config /var/run/docker/containerd/containerd.toml --log-level info

12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.199709550+08:00" level=info msg="scheme \"unix\" not registered, fallba...ule=grpc
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.199753682+08:00" level=info msg="ccResolverWrapper: sending update to c...ule=grpc
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.199782633+08:00" level=info msg="ClientConn switching balancer to \"pic...ule=grpc
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.318568572+08:00" level=info msg="Loading containers: start."
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.684687056+08:00" level=info msg="Default bridge (docker0) is assigned w...address"
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.803371643+08:00" level=info msg="Loading containers: done."
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.897258896+08:00" level=info msg="Docker daemon" commit=9d988398e7 graph...=19.03.9
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.897475376+08:00" level=info msg="Daemon has completed initialization"
12月 06 16:19:18 k8s-node01 dockerd[9541]: time="2021-12-06T16:19:18.941927810+08:00" level=info msg="API listen on /var/run/docker.sock"
12月 06 16:19:18 k8s-node01 systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.

6、部署Master Node

6.1 生成自签证书

6.1.1 自签证书颁发机构

# 切换工作目录
[root@k8s-master ~]# cd ~/TLS/k8s/

[root@k8s-master k8s]# cat > ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

[root@k8s-master k8s]# cat > ca-csr.json<< EOF 
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

6.1.2 生成自签证书

[root@k8s-master k8s]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/12/06 16:25:53 [INFO] generating a new CA key and certificate from CSR
2021/12/06 16:25:53 [INFO] generate received request
2021/12/06 16:25:53 [INFO] received CSR
2021/12/06 16:25:53 [INFO] generating key: rsa-2048
2021/12/06 16:25:53 [INFO] encoded CSR
2021/12/06 16:25:53 [INFO] signed certificate with serial number 192747439607729933538570162256306583934679978757

查看已生成证书

[root@k8s-master k8s]# ls *pem
ca-key.pem  ca.pem

6.2 使用自签CA签发kube-apiserver HTTPS证书

6.2.1 创建证书申请文件

[root@k8s-master k8s]# cat > server-csr.json << EOF
{
    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "192.168.127.200",
      "192.168.127.201",
      "192.168.127.202",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing",
            "O": "k8s",
            "OU": "System"
        }
    ]
}
EOF

hosts中的地址为集群所有服务器IP地址

6.2.2 生成证书

[root@k8s-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server
2021/12/06 16:33:53 [INFO] generate received request
2021/12/06 16:33:53 [INFO] received CSR
2021/12/06 16:33:53 [INFO] generating key: rsa-2048
2021/12/06 16:33:54 [INFO] encoded CSR
2021/12/06 16:33:54 [INFO] signed certificate with serial number 318318384200786667079070726590902069820039717884
2021/12/06 16:33:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

查看已生成证书

[root@k8s-master k8s]# ls server*pem
server-key.pem  server.pem

6.3 下载Kubernetes二进制文件

下载地址V1.18.3

版本列表

只下载Server binaries即可，里面包含了Master/Worker Node需要的所有二进制文件：kube-apiserver，kube-controller-manager，kube -scheduler，kubelet，kube-proxy

6.4 解压二进制包

[root@k8s-master k8s]# cd ~
[root@k8s-master ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
[root@k8s-master ~]# tar zxvf kubernetes-server-linux-amd64.tar.gz
[root@k8s-master ~]# cd kubernetes/server/bin
[root@k8s-master bin]# cp kube-apiserver kube-scheduler kube-controller-manager /opt/kubernetes/bin
[root@k8s-master bin]# cp kubectl /usr/bin/

6.5 部署kube-apiserver

6.5.1 创建配置文件

[root@k8s-master bin]# cd ~
[root@k8s-master ~]# cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--etcd-servers=https://192.168.127.200:2379,https://192.168.127.201:2379,https://192.168.127.202:2379 \
--bind-address=192.168.127.200 \
--secure-port=6443 \
--advertise-address=192.168.127.200 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth=true \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--service-node-port-range=30000-32767 \
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem  \
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \
--tls-cert-file=/opt/kubernetes/ssl/server.pem \
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

–logtostderr：启用日志

-v：日志等级

–log-dir：日志目录

–etcd-servers：etcd 集群地址

–bind-address：监听地址（当前服务器地址）

–secure-port：https 安全端口

–advertise-address：集群通告地址（当前服务器地址）

–allow-privileged：启用授权

–service-cluster-ip-range：Service 虚拟 IP 地址段

–enable-admission-plugins：准入控制模块

–authorization-mode：认证授权，启用 RBAC 授权和节点自管理

–enable-bootstrap-token-auth：启用 TLS bootstrap 机制

–token-auth-file：bootstrap token 文件

–service-node-port-range：Service nodeport 类型默认分配端口范围

–kubelet-client-xxx：apiserver 访问 kubelet 客户端证书

–tls-xxx-file：apiserver https 证书

–etcd-xxxfile：连接 Etcd 集群证书

–audit-log-xxx：审计日志

6.5.2 拷贝刚才生成的证书

[root@k8s-master ~]# cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

6.5.3 启用TLS BootStrpping机制

TLS Bootstraping：Master apiserver 启用 TLS 认证后，Node 节点 kubelet 和 kube- proxy 要与 kube-apiserver 进行通信，必须使用 CA 签发的有效证书才可以，当 Node 节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。为了简化流程，Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书，kubelet 会以一个低权限用户自动向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署。所以强烈建议在 Node 上使用这种方式，目前主要用于 kubelet，kube-proxy 还是由我们统一颁发一个证书。

创建上述配置文件中的token文件

[root@k8s-master ~]# cat > /opt/kubernetes/cfg/token.csv << EOF
40463628941fb0c42ba104df325dc83e,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

格式：token，用户名，UID，用户组 token 也可自行生成替换：head -c 16 /dev/urandom | od -An -t x | tr -d ‘ ‘

6.5.4 systemd管理api-server

[root@k8s-master ~]# cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS 
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF

6.5.5 启动并设置开机启动

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kube-apiserver
[root@k8s-master ~]# systemctl enable kube-apiserver
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-apiserver.service to /usr/lib/systemd/system/kube-apiserver.service.

查看服务启动状态

[root@k8s-master ~]# systemctl status kube-apiserver
● kube-apiserver.service - Kubernetes API Server
   Loaded: loaded (/usr/lib/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 16:58:43 CST; 34s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 9874 (kube-apiserver)
   CGroup: /system.slice/kube-apiserver.service
           └─9874 /opt/kubernetes/bin/kube-apiserver --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --etcd-servers=https://192.168.127.200:...

12月 06 16:58:43 k8s-master systemd[1]: Started Kubernetes API Server.
12月 06 16:58:46 k8s-master kube-apiserver[9874]: E1206 16:58:46.589882    9874 controller.go:152] Unable to remove old endpoints from kuber...rrorMsg:
Hint: Some lines were ellipsized, use -l to show in full.

一定要验证一下，启动失败则后续无法执行；

错误日志可以通过：/opt/kubernetes/logs查看

6.5.6 授权kubelet-bootstrap用户允许请求证书

[root@k8s-master ~]# kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap

6.6 部署kube-controller-manager

6.6.1 创建配置文件

[root@k8s-master ~]# cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect=true \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1 \
--allocate-node-cidrs=true \
--cluster-cidr=10.244.0.0/16 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"
EOF

–master：通过本地非安全本地端口 8080 连接 apiserver。

–leader-elect：当该组件启动多个时，自动选举（HA）

–cluster-signing-cert-file/–cluster-signing-key-file：自动为 kubelet 颁发证书的 CA，与 apiserver 保持一致

6.6.2 systemd管理controller-manager

[root@k8s-master ~]# cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

6.6.3 启动并设置开机启动

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kube-controller-manager
[root@k8s-master ~]# systemctl enable kube-controller-manager
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-controller-manager.service to /usr/lib/systemd/system/kube-controller-manager.service.

查看服务启动状态

[root@k8s-master ~]# systemctl status kube-controller-manager 
● kube-controller-manager.service - Kubernetes Controller Manager
   Loaded: loaded (/usr/lib/systemd/system/kube-controller-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 17:06:06 CST; 24s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 9955 (kube-controller)
   CGroup: /system.slice/kube-controller-manager.service
           └─9955 /opt/kubernetes/bin/kube-controller-manager --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect=true --master=1...

12月 06 17:06:06 k8s-master systemd[1]: Started Kubernetes Controller Manager.
12月 06 17:06:07 k8s-master kube-controller-manager[9955]: E1206 17:06:07.621468    9955 core.go:89] Failed to start service controller: WARN...ll fail
12月 06 17:06:07 k8s-master kube-controller-manager[9955]: E1206 17:06:07.623625    9955 core.go:229] failed to start cloud node lifecycle co...rovided
12月 06 17:06:17 k8s-master kube-controller-manager[9955]: E1206 17:06:17.938024    9955 clusterroleaggregation_controller.go:181] edit faile...y again
Hint: Some lines were ellipsized, use -l to show in full.

6.7 部署kube-scheduler

6.7.1 创建配置文件

[root@k8s-master ~]# cat > /opt/kubernetes/cfg/kube-scheduler.conf << EOF 
KUBE_SCHEDULER_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--leader-elect \
--master=127.0.0.1:8080 \
--bind-address=127.0.0.1"
EOF

6.7.2 systemd管理scheduler

[root@k8s-master ~]# cat > /usr/lib/systemd/system/kube-scheduler.service << EOF 
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-scheduler.conf
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS 
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

6.7.3 启动并设置开机启动

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kube-scheduler
[root@k8s-master ~]# systemctl enable kube-scheduler
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-scheduler.service to /usr/lib/systemd/system/kube-scheduler.service.

查看服务启动状态

[root@k8s-master ~]# systemctl status kube-scheduler
● kube-scheduler.service - Kubernetes Scheduler
   Loaded: loaded (/usr/lib/systemd/system/kube-scheduler.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 17:11:01 CST; 29s ago
     Docs: https://github.com/kubernetes/kubernetes
 Main PID: 10010 (kube-scheduler)
   CGroup: /system.slice/kube-scheduler.service
           └─10010 /opt/kubernetes/bin/kube-scheduler --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --leader-elect --master=127.0.0.1:8080...

12月 06 17:11:01 k8s-master systemd[1]: Started Kubernetes Scheduler.
12月 06 17:11:01 k8s-master kube-scheduler[10010]: I1206 17:11:01.443568   10010 registry.go:150] Registering EvenPodsSpread predicate and pr...unction
12月 06 17:11:01 k8s-master kube-scheduler[10010]: I1206 17:11:01.443710   10010 registry.go:150] Registering EvenPodsSpread predicate and pr...unction
Hint: Some lines were ellipsized, use -l to show in full.

6.8 查看集群状态

[root@k8s-master ~]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-2               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"}   
etcd-0               Healthy   {"health":"true"}

如上输出说明 Master 节点组件运行正常。

7、部署Worker Node

下面仍旧在Master Node上操作，及Master节点同时作为Worker Node

7.1 创建工作目录并包括二进制文件

[root@k8s-master ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
[root@k8s-master ~]# cd kubernetes/server/bin
[root@k8s-master bin]# cp kubelet kube-proxy /opt/kubernetes/bin

7.2 生成bootstrap.kubeconfig文件

[root@k8s-master bin]# cd ~/TLS/k8s
[root@k8s-master k8s]# KUBE_APISERVER="https://192.168.127.200:6443"
[root@k8s-master k8s]# TOKEN=40463628941fb0c42ba104df325dc83e #这个和上面创建token文件的一致
[root@k8s-master k8s]# kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=bootstrap.kubeconfig

[root@k8s-master k8s]# kubectl config set-credentials "kubelet-bootstrap" \
  --token=${TOKEN} \
  --kubeconfig=bootstrap.kubeconfig

[root@k8s-master k8s]# kubectl config set-context default \
  --cluster=kubernetes \
  --user=kubelet-bootstrap \
  --kubeconfig=bootstrap.kubeconfig

[root@k8s-master k8s]# kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

[root@k8s-master k8s]# ls bootstrap*
bootstrap.kubeconfig

拷贝到kubernetes配置文件路径

[root@k8s-master k8s]# cp bootstrap.kubeconfig /opt/kubernetes/cfg

7.3 部署kubelet

7.3.1 创建配置文件

[root@k8s-master k8s]# cd ~
[root@k8s-master k8s]# cat > /opt/kubernetes/cfg/kubelet.conf << EOF
KUBELET_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--hostname-override=k8s-master \
--network-plugin=cni \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet-config.yml \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF

–hostname-override：显示名称，集群中唯一
–network-plugin：启用 CNI
–kubeconfig：空路径，会自动生成，后面用于连接 apiserver
–bootstrap-kubeconfig：首次启动向 apiserver 申请证书
–config：配置参数文件
–cert-dir：kubelet 证书生成目录
–pod-infra-container-image：管理 Pod 网络容器的镜像

7.3.2 配置参数文件kubelet-config.yml

[root@k8s-master k8s]# cat > /opt/kubernetes/cfg/kubelet-config.yml << EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
  anonymous:
    enabled: false 
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /opt/kubernetes/ssl/ca.pem 
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 5m0s
    cacheUnauthorizedTTL: 30s 
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
maxOpenFiles: 1000000 
maxPods: 110
EOF

7.3.3 systemd管理kubelet

[root@k8s-master k8s]# cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet.conf
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

7.3.4 启动并设置开机启动

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kubelet 
[root@k8s-master ~]# systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

查看服务启动状态

[root@k8s-master ~]# systemctl status kubelet 
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 17:31:00 CST; 31s ago
 Main PID: 10170 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─10170 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=k8s-master --network-plug...

12月 06 17:31:00 k8s-master systemd[1]: Started Kubernetes Kubelet.

7.4 批准kubelet证书申请并加入集群

# 查看 kubelet 证书请求
[root@k8s-master ~]# kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-Vc3SnwdGfd0sSP641u7Ejkp5GfBfFbdjs-2N5FCsMGI   86s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 批准申请
[root@k8s-master ~]# kubectl certificate approve node-csr-Vc3SnwdGfd0sSP641u7Ejkp5GfBfFbdjs-2N5FCsMGI
certificatesigningrequest.certificates.k8s.io/node-csr-Vc3SnwdGfd0sSP641u7Ejkp5GfBfFbdjs-2N5FCsMGI approved

kubectl certificate approve 批准申请后面的即为请求的name

# 查看节点
[root@k8s-master ~]# kubectl get node
NAME         STATUS     ROLES    AGE   VERSION
k8s-master   NotReady   <none>   56s   v1.18.3

由于网络插件还没有部署，节点会没有准备就绪 NotReady

7.5 部署kube-proxy

7.5.1 生成kube-proxy证书

# 切换工作目录
[root@k8s-master ~]# cd ~/TLS/k8s
# 创建证书请求文件
[root@k8s-master k8s]# cat > kube-proxy-csr.json<< EOF
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}
EOF
# 生成证书
[root@k8s-master k8s]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2021/12/06 17:47:19 [INFO] generate received request
2021/12/06 17:47:19 [INFO] received CSR
2021/12/06 17:47:19 [INFO] generating key: rsa-2048
2021/12/06 17:47:20 [INFO] encoded CSR
2021/12/06 17:47:20 [INFO] signed certificate with serial number 19182532311011123094671496940595247309896730816
2021/12/06 17:47:20 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

# 查看已生成证书
[root@k8s-master k8s]# ls kube-proxy*pem
kube-proxy-key.pem  kube-proxy.pem

证书生成过程中的warning无需处理

7.5.1 生成kube-proxy.kubeconfig文件

[root@k8s-master k8s]# kubectl config set-cluster kubernetes \
  --certificate-authority=/opt/kubernetes/ssl/ca.pem \
  --embed-certs=true \
  --server=${KUBE_APISERVER} \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master k8s]# kubectl config set-credentials kube-proxy \
  --client-certificate=./kube-proxy.pem \
  --client-key=./kube-proxy-key.pem \
  --embed-certs=true \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master k8s]# kubectl config set-context default \
  --cluster=kubernetes \
  --user=kube-proxy \
  --kubeconfig=kube-proxy.kubeconfig

[root@k8s-master k8s]# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

# 拷贝kube-proxy.kubeconfig至k8s配置文件目录
[root@k8s-master k8s]# cp kube-proxy.kubeconfig /opt/kubernetes/cfg

7.5.2 创建配置文件

[root@k8s-master k8s]# cd ~
[root@k8s-master ~]# cat > /opt/kubernetes/cfg/kube-proxy.conf << EOF
KUBE_PROXY_OPTS="--logtostderr=false \
--v=2 \
--log-dir=/opt/kubernetes/logs \
--config=/opt/kubernetes/cfg/kube-proxy-config.yml" 
EOF

7.5.3 创建配置参数文件

[root@k8s-master ~]# cat > /opt/kubernetes/cfg/kube-proxy-config.yml << EOF 
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1 
bindAddress: 0.0.0.0
metricsBindAddress: 0.0.0.0:10249
clientConnection:
  kubeconfig: /opt/kubernetes/cfg/kube-proxy.kubeconfig 
hostnameOverride: k8s-master
clusterCIDR: 10.0.0.0/24
EOF

7.5.4 systemd管理kube-proxy

[root@k8s-master ~]# cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Proxy
After=network.target
[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-proxy.conf 
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS 
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

7.5.5 启动并设置开机启动

[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl start kube-proxy 
[root@k8s-master ~]# systemctl enable kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

查看服务启动状态

[root@k8s-master ~]# systemctl status kube-proxy 
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2021-12-06 17:55:15 CST; 15s ago
 Main PID: 14603 (kube-proxy)
   CGroup: /system.slice/kube-proxy.service
           └─14603 /opt/kubernetes/bin/kube-proxy --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/cfg/kube-proxy-co...

12月 06 17:55:15 k8s-master systemd[1]: Started Kubernetes Proxy.

7.6 部署CNI网络

下载地址

7.6.1 解压二进制文件并移动默认工作目录

[root@k8s-master ~]# mkdir /opt/cni/bin -p
[root@k8s-master ~]# tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

7.6.2 部署CNI网络

# 下载配置文件
[root@k8s-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

# 默认镜像地址无法访问，修改为 docker hub 镜像仓库。
[root@k8s-master ~]# sed -i -r "s#quay.io/coreos/flannel:.*-amd64#lizhenliang/flannel:v0.12.0-amd64#g" kube-flannel.yml

# 应用修改内容
[root@k8s-master ~]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

7.6.3 查看pods状态

[root@k8s-master ~]# kubectl get pods -n kube-system
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-flszz   1/1     Running   0          40s

当看到状态为Running，说明网络插件部署成功，初始化需要一些时间，等一会即可。

7.6.4 查看node状态

[root@k8s-master ~]# kubectl get node
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    <none>   31m   v1.18.3

这时候可以看到，node状态已经是Ready

7.7 授权apiserver访问kubelet

# 创建授权文件
[root@k8s-master ~]# cd /opt/kubernetes/cfg
[root@k8s-master cfg]# cat > apiserver-to-kubelet-rbac.yaml << EOF 
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:kube-apiserver-to-kubelet
rules:
  - apiGroups:
      - ""
    resources:
      - nodes/proxy
      - nodes/stats
      - nodes/log
      - nodes/spec
      - nodes/metrics
      - pods/log
    verbs:
      - "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:kube-apiserver
  namespace: ""
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:kube-apiserver-to-kubelet
subjects:
  - apiGroup: rbac.authorization.k8s.io
    kind: User
    name: kubernetes
EOF

# 部署
[root@k8s-master cfg]# kubectl apply -f apiserver-to-kubelet-rbac.yaml
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

# 查看是否创建
[root@k8s-master cfg]# kubectl get clusterrole,clusterrolebinding | grep system:kube-apiserver
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet                                       2021-12-07T08:53:34Z
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver                                  ClusterRole/system:kube-apiserver-to-kubelet                       5m46s

7.8 新增加Worker Node

7.8.1 创建Worker Node工作目录

# k8s-node01
[root@k8s-node01 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
# k8s-node02
[root@k8s-node02 ~]# mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}

7.8.2 拷贝已部署好的文件至新节点

# k8s-master copy to k8s-node01

# 切换工作目录
[root@k8s-master cfg]# cd ~

# kubelet、kube-proxy二进制文件
[root@k8s-master ~]# scp ~/kubernetes/server/bin/kubelet ~/kubernetes/server/bin/kube-proxy root@k8s-node01:/opt/kubernetes/bin

# kubelet、kube-proxy配置文件
[root@k8s-master ~]# cd /opt/kubernetes/cfg
[root@k8s-master cfg]# scp bootstrap.kubeconfig kube-proxy.kubeconfig root@k8s-node01:/opt/kubernetes/cfg
[root@k8s-master cfg]# scp kubelet.conf kubelet-config.yml kube-proxy.conf kube-proxy-config.yml root@k8s-node01:/opt/kubernetes/cfg

# kubelet、kube-proxy自动服务
[root@k8s-master cfg]# cd ~
[root@k8s-master ~]# scp -r /usr/lib/systemd/system/{kubelet,kube-proxy}.service root@k8s-node01:/usr/lib/systemd/system

# 自签ca
[root@k8s-master ~]# scp /opt/kubernetes/ssl/ca.pem root@k8s-node01:/opt/kubernetes/ssl

# CNI网络
[root@k8s-master ~]# scp -r /opt/cni/ root@k8s-node01:/opt/

k8s-node02参照k8s-node01进行

7.8.3 修改Worker Node节点主机名

# k8s-node01节点
[root@k8s-node01 ~]# vi /opt/kubernetes/cfg/kubelet.conf
--hostname-override=k8s-node01

[root@k8s-node01 ~]# vi /opt/kubernetes/cfg/kube-proxy-config.yml
hostnameOverride: k8s-node01

k8s-node02参照k8s-node01进行

7.8.4 启动并设置开机启动

# k8s-node01节点
[root@k8s-node01 ~]# systemctl daemon-reload
[root@k8s-node01 ~]# systemctl start kubelet
[root@k8s-node01 ~]# systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.
[root@k8s-node01 ~]# systemctl start kube-proxy
[root@k8s-node01 ~]# systemctl enable kube-proxy
Created symlink from /etc/systemd/system/multi-user.target.wants/kube-proxy.service to /usr/lib/systemd/system/kube-proxy.service.

查看服务启动状态

# kubelet
[root@k8s-node01 ~]# systemctl status kubelet
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
   Active: active (running) since 三 2021-12-08 08:33:59 CST; 38s ago
 Main PID: 1737 (kubelet)
   CGroup: /system.slice/kubelet.service
           └─1737 /opt/kubernetes/bin/kubelet --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --hostname-override=k8s-node01 --network-plugin...

12月 08 08:33:59 k8s-node01 systemd[1]: Started Kubernetes Kubelet.

#kube-proxy
[root@k8s-node01 ~]# systemctl status kube-proxy
● kube-proxy.service - Kubernetes Proxy
   Loaded: loaded (/usr/lib/systemd/system/kube-proxy.service; enabled; vendor preset: disabled)
   Active: active (running) since 三 2021-12-08 08:34:06 CST; 35s ago
 Main PID: 1772 (kube-proxy)
   CGroup: /system.slice/kube-proxy.service
           └─1772 /opt/kubernetes/bin/kube-proxy --logtostderr=false --v=2 --log-dir=/opt/kubernetes/logs --config=/opt/kubernetes/cfg/kube-proxy-conf...

12月 08 08:34:06 k8s-node01 systemd[1]: Started Kubernetes Proxy.
12月 08 08:34:06 k8s-node01 kube-proxy[1772]: E1208 08:34:06.917435    1772 node.go:125] Failed to retrieve node info: nodes "k8s-node01" not found
12月 08 08:34:08 k8s-node01 kube-proxy[1772]: E1208 08:34:08.075556    1772 node.go:125] Failed to retrieve node info: nodes "k8s-node01" not found
12月 08 08:34:10 k8s-node01 kube-proxy[1772]: E1208 08:34:10.405456    1772 node.go:125] Failed to retrieve node info: nodes "k8s-node01" not found
12月 08 08:34:15 k8s-node01 kube-proxy[1772]: E1208 08:34:15.091961    1772 node.go:125] Failed to retrieve node info: nodes "k8s-node01" not found
12月 08 08:34:24 k8s-node01 kube-proxy[1772]: E1208 08:34:24.209478    1772 node.go:125] Failed to retrieve node info: nodes "k8s-node01" not found

k8s-node02参照k8s-node01进行

7.8.5 在Master上批准新Node kubelet证书申请

[root@k8s-master ~]# kubectl get csr
NAME                                                   AGE     SIGNERNAME                                    REQUESTOR           CONDITION
node-csr-PBvjikAhiKqHCq-l0o9duoRPFlTpHBi0GFFokd2eKSo   31s     kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending
node-csr-wtXliSh8Yxg5I6kovAr_S0mt7caak3OhgwklBuq9luE   6m46s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   Pending

# 审批证书申请
[root@k8s-master ~]# kubectl certificate approve node-csr-wtXliSh8Yxg5I6kovAr_S0mt7caak3OhgwklBuq9luE
certificatesigningrequest.certificates.k8s.io/node-csr-wtXliSh8Yxg5I6kovAr_S0mt7caak3OhgwklBuq9luE approved
[root@k8s-master ~]# kubectl certificate approve node-csr-PBvjikAhiKqHCq-l0o9duoRPFlTpHBi0GFFokd2eKSo
certificatesigningrequest.certificates.k8s.io/node-csr-PBvjikAhiKqHCq-l0o9duoRPFlTpHBi0GFFokd2eKSo approved

7.8.6 查看状态

# 稍等一会，等pods状态为Running
[root@k8s-master ~]# kubectl get pods -n kube-system
NAME                    READY   STATUS     RESTARTS   AGE
kube-flannel-ds-brhjg   0/1     Init:1/2   0          38s
kube-flannel-ds-flszz   1/1     Running    2          38h
kube-flannel-ds-rp77b   0/1     Init:1/2   0          27s

[root@k8s-master ~]# kubectl get pods -n kube-system
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-brhjg   1/1     Running   0          115s
kube-flannel-ds-flszz   1/1     Running   2          38h
kube-flannel-ds-rp77b   1/1     Running   0          104s

# 查看nodes状态
[root@k8s-master ~]# kubectl get nodes
NAME         STATUS   ROLES    AGE    VERSION
k8s-master   Ready    <none>   39h    v1.18.3
k8s-node01   Ready    <none>   2m3s   v1.18.3
k8s-node02   Ready    <none>   112s   v1.18.3

8、测试kubernetes集群

在Kubernetes集群中创建一个pod，验证是否正常运行：

# k8s-master节点执行
[root@k8s-master ~]# kubectl create deployment nginx --image=nginx
[root@k8s-master ~]# kubectl expose deployment nginx --port=80 --type=NodePort
[root@k8s-master ~]# kubectl get pod,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-f89759699-tnv2c   1/1     Running   0          55s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        20m
service/nginx        NodePort    10.103.63.117   <none>        80:31252/TCP   50s

浏览器访问地址：http://NodeIP:Port（这里是：http://192.168.127.201:31252）

9、附件

提取码：a6pt

3、kubernetes 集群搭建(二进制方式)

1、安装要求

2、准备环境

3、操作系统初始化配置

4、部署Etcd集群

4.1 准备 cfssl 证书生成工具

4.2 生成Etcd证书

4.2.1 自签证书颁发机构（CA）

4.2.2 使用自签CA签发Etcd HTTPS证书

4.3 下载Etcd二进制文件

4.4 部署Etcd集群

4.4.1 创建工作目录并解压二进制包

4.4.2 创建Etcd配置文件

4.4.3 systemd管理Etcd

4.4.4 拷贝刚刚生成的证书

4.4.5 将上面节点1所有生成的文件拷贝到节点2和节点3

4.4.6 启动Etcd服务并设置开机启动

4.4.7 查看集群状态

5、安装Docker

5.1 下载Docker二进制文件

5.2 解压二进制包

5.3 systemd管理Docker

5.4 创建配置文件

5.5 启动并设置开机启动

5.6 安装Docker至Worker Node节点

6、部署Master Node

6.1 生成自签证书

6.1.1 自签证书颁发机构

6.1.2 生成自签证书

6.2 使用自签CA签发kube-apiserver HTTPS证书

6.2.1 创建证书申请文件

6.2.2 生成证书

6.3 下载Kubernetes二进制文件

6.4 解压二进制包

6.5 部署kube-apiserver

6.5.1 创建配置文件

6.5.2 拷贝刚才生成的证书

6.5.3 启用TLS BootStrpping机制

6.5.4 systemd管理api-server

6.5.5 启动并设置开机启动

6.5.6 授权kubelet-bootstrap用户允许请求证书

6.6 部署kube-controller-manager

6.6.1 创建配置文件

6.6.2 systemd管理controller-manager

6.6.3 启动并设置开机启动

6.7 部署kube-scheduler

6.7.1 创建配置文件

6.7.2 systemd管理scheduler

6.7.3 启动并设置开机启动

6.8 查看集群状态

7、部署Worker Node

7.1 创建工作目录并包括二进制文件

7.2 生成bootstrap.kubeconfig文件

7.3 部署kubelet

7.3.1 创建配置文件

7.3.2 配置参数文件kubelet-config.yml

7.3.3 systemd管理kubelet

7.3.4 启动并设置开机启动

7.4 批准kubelet证书申请并加入集群

7.5 部署kube-proxy

7.5.1 生成kube-proxy证书

7.5.1 生成kube-proxy.kubeconfig文件

7.5.2 创建配置文件

7.5.3 创建配置参数文件

7.5.4 systemd管理kube-proxy

7.5.5 启动并设置开机启动

7.6 部署CNI网络

7.6.1 解压二进制文件并移动默认工作目录

7.6.2 部署CNI网络

7.6.3 查看pods状态

7.6.4 查看node状态

7.7 授权apiserver访问kubelet

7.8 新增加Worker Node

7.8.1 创建Worker Node工作目录

7.8.2 拷贝已部署好的文件至新节点

7.8.3 修改Worker Node节点主机名

7.8.4 启动并设置开机启动

7.8.5 在Master上批准新Node kubelet证书申请

7.8.6 查看状态

8、测试kubernetes集群