Kubernetes 1.8.x 全手动安装教程

版本信息

Kubernetes v1.8.2
Etcd v3.2.9
Calico v2.6.2
Docker v17.10.0-ce

预先准备信息

服务器信息

简要说明

 master 为主要控制节点也是部署节点，node 为应用程序工作节点。

安装准备—所有节点都需要操作

确认关闭防火墙和selinux
$ systemctl stop firewalld && systemctl disable firewalld
$ setenforce 0
$ vim /etc/selinux/config
SELINUX=disabled
所有节点设置hosts
[root@test3 ~]# cat /etc/hosts
192.168.170.170 test-xlx
192.168.170.171 test2
192.168.170.172 test3
所有节点需要安装Docker或rtk引擎。这边采用Docker来当作容器引擎，安装方式如下：
 curl -fsSL "https://get.docker.com/" | sh 【自动启动】
 systemctl enable docker && systemctl start docker
编辑/lib/systemd/system/docker.service，添加
[root@test-xlx ~]# vim /lib/systemd/system/docker.service
ExecStartPost=/sbin/iptables -I FORWARD -s 0.0.0.0/0 -j ACCEPT
[root@test-xlx ~]# systemctl daemon-reload 
[root@test-xlx ~]# systemctl restart docker.service 
需要设定/etc/sysctl.d/k8s.conf的系统参数，将桥接的IPv4流量传递到iptables的链：
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

安装CFSSL工具

这将会用来建立 TLS certificates

在master节点操作
export CFSSL_URL="https://pkg.cfssl.org/R1.2"
wget "${CFSSL_URL}/cfssl_linux-amd64" -O /usr/local/bin/cfssl
wget "${CFSSL_URL}/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

提示：在开始安装 Kubernetes 之前，需要先将一些必要系统创建完成，其中 Etcd 就是 Kubernetes 最重要的一环，Kubernetes 会将大部分信息储存于 Etcd 上，来提供给其他节点索取，以确保整个集群运作与沟通正常。

创建集群 CA 与 Certificates

在这部分，将会需要产生 client 与 server 的各组件 certificates，并且替 Kubernetes admin user 产生 client 证书。

建立/etc/etcd/ssl文件夹，然后进入目录完成以下操作
mkdir -p /etc/etcd/ssl && cd /etc/etcd/ssl
export PKI_URL="https://kairen.github.io/files/manual-v1.8/pki"
下载ca-config.json与etcd-ca-csr.json文件，并产生 CA 密钥：
wget "${PKI_URL}/ca-config.json" "${PKI_URL}/etcd-ca-csr.json"【可能会无法下载】
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
代替方案生成ca-config.json文件 etcd-ca-csr.json文件
cat <<EOF > ca-config.json
{
    "signing": {
        "default": {
            "expiry": "87600h"
        },
        "profiles": {
            "kubernetes": {
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ],
                "expiry": "87600h"
            }
        }
    }
}
EOF
cat <<EOF > etcd-ca-csr.json
{
    "CN": "etcd",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "SC",
            "ST": "ChengDu",
            "L": "ChengDu",
            "O": "etcd",
            "OU": "Etcd Security"
        }
    ]
}
EOF
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
生成证书
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca
------------------------------------------------------------------------------------------------------------------------------
[root@test-xlx ssl]# cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare etcd-ca
2021/03/14 00:59:33 [INFO] generating a new CA key and certificate from CSR
2021/03/14 00:59:33 [INFO] generate received request
2021/03/14 00:59:33 [INFO] received CSR
2021/03/14 00:59:33 [INFO] generating key: rsa-2048
2021/03/14 00:59:33 [INFO] encoded CSR
2021/03/14 00:59:33 [INFO] signed certificate with serial number 3757025824159786925659790574978323027931637352
------------------------------------------------------------------------------------------------------------------------------
[root@test-xlx ssl]# ls /etc/etcd/ssl/
ca-config.json  etcd-ca.csr  etcd-ca-csr.json  etcd-ca-key.pem  etcd-ca.pem
下载etcd-csr.json文件，并产生 kube-apiserver certificate 证书：
wget "${PKI_URL}/etcd-csr.json"【可能无法下载】
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
代替方案生成etcd-csr.json文件
cat <<EOF > etcd-csr.json
{
    "CN": "etcd",
    "hosts": [
        "127.0.0.1",
        "192.168.170.171",
        "192.168.170.172"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "SC",
            "ST": "ChengDu",
            "L": "ChengDu",
            "O": "etcd",
            "OU": "Etcd Security"
        }
    ]
}
EOF
注意hosts更换成自己的IP
生成证书
生成证书
cfssl gencert \
  -ca=etcd-ca.pem \
  -ca-key=etcd-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  etcd-csr.json | cfssljson -bare etcd
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[root@test-xlx ssl]# cfssl gencert   -ca=etcd-ca.pem   -ca-key=etcd-ca-key.pem   -config=ca-config.json   -profile=kubernetes   etcd-csr.json | cfssljson -bare etcd
2021/03/14 01:15:17 [INFO] generate received request
2021/03/14 01:15:17 [INFO] received CSR
2021/03/14 01:15:17 [INFO] generating key: rsa-2048
2021/03/14 01:15:18 [INFO] encoded CSR
2021/03/14 01:15:18 [INFO] signed certificate with serial number 457273308119455569510143983837366203361410694843
2021/03/14 01:15:18 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
可以删除不必要的json文件 也可以保留

Etcd 安装与设定—在master节点执行

首先在master1节点下载 Etcd，并解压缩放到 /opt 底下与安装：
export ETCD_URL="https://github.com/coreos/etcd/releases/download"
wget "${ETCD_URL}/v3.2.9/etcd-v3.2.9-linux-amd64.tar.gz"
tar -zxvf  etcd-v3.2.9-linux-amd64.tar.gz
mv etcd-v3.2.9-linux-amd64/etcd* /usr/local/bin/ && rm -rf etcd-v3.2.9-linux-amd64
完成后新建 Etcd Group 与 User，并建立 Etcd 配置文件目录：
groupadd etcd && useradd -c "Etcd user" -g etcd -s /sbin/nologin -r etcd
下载etcd相关文件，我们将来管理 Etcd：
export ETCD_CONF_URL="https://kairen.github.io/files/manual-v1.8/master"【可能无法下载】
wget "${ETCD_CONF_URL}/etcd.conf" -O /etc/etcd/etcd.conf
 wget "${ETCD_CONF_URL}/etcd.service" -O /lib/systemd/system/etcd.service
------------------------------------------------------------------------------------------------------
生成etcd.conf文件
# cat /etc/etcd/etcd.conf 
# [member]
ETCD_NAME=node170
ETCD_DATA_DIR=/var/lib/etcd
ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
ETCD_PROXY=off
# [cluster]
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.170.170:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.170.170:2380
ETCD_INITIAL_CLUSTER=node162=https://192.168.170.170:2380
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_INITIAL_CLUSTER_TOKEN=etcd-k8s-cluster
# [security]
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_CLIENT_CERT_AUTH="true"
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_AUTO_TLS="true"
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"
ETCD_PEER_CLIENT_CERT_AUTH="true"
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"
ETCD_PEER_AUTO_TLS="true"
生成etcd.service
cat <<EOF > /lib/systemd/system/etcd.service
[Unit]
Description=Etcd Service
After=network.target
[Service]
Environment=ETCD_DATA_DIR=/var/lib/etcd/default
EnvironmentFile=-/etc/etcd/etcd.conf
Type=notify
User=etcd
PermissionsStartOnly=true
ExecStart=/usr/local/bin/etcd
Restart=on-failure
RestartSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
------------------------------------------------------------------------------------------------------
建立 var 存放信息，然后启动 Etcd 服务:
mkdir -p /var/lib/etcd && chown etcd:etcd -R /var/lib/etcd /etc/etcd
systemctl enable etcd.service && systemctl start etcd.service
通过简单指令验证：
ETCDCTL_API=3 etcdctl \
    --cacert=${CA}/etcd-ca.pem \
    --cert=${CA}/etcd.pem \
    --key=${CA}/etcd-key.pem \
    --endpoints="https://127.0.0.1:2379" \
    endpoint health
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 919.691µs

遗留问题

etcd通过简单指令验证：
ETCDCTL_API=3 etcdctl \
    --cacert=${CA}/etcd-ca.pem \
    --cert=${CA}/etcd.pem \
    --key=${CA}/etcd-key.pem \
    --endpoints="https://127.0.0.1:2379" \
    endpoint health
【本机地址不可以】
https://192.168.170.170:2379 is unhealthy: failed to connect: grpc: timed out when dialing
Error:  unhealthy cluster

Kubernetes Master

Master 是 Kubernetes 的大总管，主要创建apiserver、Controller manager与Scheduler来组件管理所有 Node。本步骤将下载 Kubernetes 并安装至 master1上，然后产生相关 TLS Cert 与 CA 密钥，提供给集群组件认证使用。

下载 Kubernetes 组件

首先通过网络取得所有需要的执行文件：
# Download Kubernetes
export KUBE_URL="https://storage.googleapis.com/kubernetes-release/release/v1.8.2/bin/linux/amd64"
wget "${KUBE_URL}/kubelet" -O /usr/local/bin/kubelet
wget "${KUBE_URL}/kubectl" -O /usr/local/bin/kubectl
chmod +x /usr/local/bin/kubelet /usr/local/bin/kubectl
# Download CNI
mkdir -p /opt/cni/bin && cd /opt/cni/bin
export CNI_URL="https://github.com/containernetworking/plugins/releases/download"
wget -qO  --show-progress "${CNI_URL}/v0.6.0/cni-plugins-amd64-v0.6.0.tgz" | tar -zx
【wget https://github.com/containernetworking/plugins/releases/download/v0.6.0/cni-plugins-amd64-v0.6.0.tgz】
[root@test-xlx bin]# tar -zxvf cni-plugins-amd64-v0.6.0.tgz 
[root@test-xlx bin]# ls
bridge  cni-plugins-amd64-v0.6.0.tgz  dhcp  flannel  host-local  ipvlan  loopback  macvlan  portmap  ptp  sample  tuning  vlan

创建集群 CA 与 Certificates

在这部分，将会需要生成 client 与 server 的各组件 certificates，并且替 Kubernetes admin user 生成 client 证书。
创建pki文件夹，然后进入目录完成以下操作。

mkdir -p /etc/kubernetes/pki && cd /etc/kubernetes/pki
export PKI_URL="https://kairen.github.io/files/manual-v1.8/pki"
export KUBE_APISERVER="https://192.168.170.170:6443"
下载ca-config.json与ca-csr.json文件，并生成 CA 密钥：
wget "${PKI_URL}/ca-config.json" "${PKI_URL}/ca-csr.json"【可能无法下载】
ls ca*.pem
ca-key.pem  ca.pem
———————————————————————————————————————————————————————————————————————————————————————————————————————
生成ca-config.json文件 
CA 配置文件用于配置根证书的使用场景 (profile) 和具体参数 (usage，过期时间、服务端认证、客户端认证、加密等)：
cat > ca-config.json <<EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "kubernetes": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "876000h"
      }
    }
  }
}
EOF
######################################################################################################
signing：表示该证书可用于签名其它证书（生成的ca.pem证书中CA=TRUE）；
server auth：表示 client 可以用该证书对 server 提供的证书进行验证；
client auth：表示 server 可以用该证书对 client 提供的证书进行验证；
expiry:876000h：证书有效期设置为 100 年。
######################################################################################################
生成 ca-csr.json 创建证书签名请求文件
cat > ca-csr.json <<EOF
{
  "CN": "kubernetes-ca",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "zhaoyixin"
    }
  ],
  "ca": {
    "expiry": "876000h"
 }
}
EOF
[root@test-xlx pki]# ls /etc/kubernetes/pki/
ca-config.json  ca-csr.json
 cfssl gencert -initca ca-csr.json | cfssljson -bare ca  【生成 CA 密钥】
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[root@test-xlx pki]#   cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2021/03/16 02:32:01 [INFO] generating a new CA key and certificate from CSR
2021/03/16 02:32:01 [INFO] generate received request
2021/03/16 02:32:01 [INFO] received CSR
2021/03/16 02:32:01 [INFO] generating key: rsa-2048
2021/03/16 02:32:02 [INFO] encoded CSR
2021/03/16 02:32:02 [INFO] signed certificate with serial number 37557834931298748572512819073885813630524656387
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
———————————————————————————————————————————————————————————————————————————————————————————————————————
API server certificate
下载apiserver-csr.json文件，并生成 kube-apiserver certificate 证书：
$ wget "${PKI_URL}/apiserver-csr.json"  【可能无法下载】
$ cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=10.96.0.1,172.16.35.12,127.0.0.1,kubernetes.default \
  -profile=kubernetes \
  apiserver-csr.json | cfssljson -bare apiserver
$ ls apiserver*.pem
apiserver-key.pem  apiserver.pe
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
生成apiserver-csr.json
[root@test-xlx pki]# cat apiserver-csr.json 
{
    "CN": "k8s-apiserver",
    "hosts": [
        "127.0.0.1",
        "192.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "192.168.170.170",
        "192.168.170.171",
        "192.168.170.172"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "ShangHai",
            "L": "ShangHai",
            "O": "xy",
            "OU": "ops"
        }
    ]
}
##################################################################################################################
[root@test-xlx pki]#  cfssl gencert \
>   -ca=ca.pem \
>   -ca-key=ca-key.pem \
>   -config=ca-config.json \
>   -hostname=10.96.0.1,172.16.35.12,127.0.0.1,kubernetes.default \
>   -profile=kubernetes \
>   apiserver-csr.json | cfssljson -bare apiserver
2021/03/16 02:50:05 [INFO] generate received request
2021/03/16 02:50:05 [INFO] received CSR
2021/03/16 02:50:05 [INFO] generating key: rsa-2048
2021/03/16 02:50:05 [INFO] encoded CSR
2021/03/16 02:50:05 [INFO] signed certificate with serial number 719572894668747476648756995718566282319980397122
##################################################################################################################
[root@test-xlx pki]# ls apiserver*.pem
apiserver-key.pem  apiserver.pem
[root@test-xlx pki]# ls
apiserver.csr  apiserver-csr.json  apiserver-key.pem  apiserver.pem  ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
Front proxy certificate
下载front-proxy-ca-csr.json文件，并生成 Front proxy CA 密钥，Front proxy 主要是用在 API aggregator 上:【可能无法下载】
wget "${PKI_URL}/front-proxy-ca-csr.json"
cfssl gencert \
  -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca
ls front-proxy-ca*.pem
front-proxy-ca-key.pem  front-proxy-ca.pem
  下载front-proxy-client-csr.json文件，并生成 front-proxy-client 证书：
$ wget "${PKI_URL}/front-proxy-client-csr.json"
$ cfssl gencert \
  -ca=front-proxy-ca.pem \
  -ca-key=front-proxy-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  front-proxy-client-csr.json | cfssljson -bare front-proxy-client
$ ls front-proxy-client*.pem
front-proxy-client-key.pem  front-proxy-client.pem下载front-proxy-client-csr.json文件，并生成 front-proxy-client 证书：【可能无法下载】
$ wget "${PKI_URL}/front-proxy-client-csr.json" 【可能无法下载】
$ cfssl gencert \
  -ca=front-proxy-ca.pem \
  -ca-key=front-proxy-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  front-proxy-client-csr.json | cfssljson -bare front-proxy-client
$ ls front-proxy-client*.pem
front-proxy-client-key.pem  front-proxy-client.pem

Bootstrap Token

http://docs.kubernetes.org.cn/713.html
由于通过手动创建 CA 方式太过繁杂，只适合少量机器，因为每次签证时都需要绑定 Node IP，随机器增加会带来很多困扰，因此这边使用 TLS Bootstrapping 方式进行授权，由 apiserver 自动给符合条件的 Node 发送证书来授权加入集群。
主要做法是 kubelet 启动时，向 kube-apiserver 传送 TLS Bootstrapping 请求，而 kube-apiserver 验证 kubelet 请求的 token 是否与设定的一样，若一样就自动产生 kubelet 证书与密钥。具体作法可以参考 TLS bootstrapping。

首先建立一个变量来产生BOOTSTRAP_TOKEN，并建立 bootstrap.conf 的 kubeconfig 文件：
$ export BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ')
$ cat <<EOF > /etc/kubernetes/token.csv
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
# bootstrap set-cluster
cd /etc/kubernetes/pki
kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=../bootstrap.conf
——————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=../bootstrap.conf
Cluster "kubernetes" set.
——————————————————————————————————————————————————————————————————————————————————————————
# bootstrap set-credentials
$ kubectl config set-credentials kubelet-bootstrap \
    --token=${BOOTSTRAP_TOKEN} \
    --kubeconfig=../bootstrap.conf
___________________________________________________________________________________________
[root@test-xlx pki]# kubectl config set-credentials kubelet-bootstrap \
>     --token=${BOOTSTRAP_TOKEN} \
>     --kubeconfig=../bootstrap.conf
User "kubelet-bootstrap" set.
___________________________________________________________________________________________
# bootstrap set-context
$ kubectl config set-context default \
    --cluster=kubernetes \
    --user=kubelet-bootstrap \
   --kubeconfig=../bootstrap.conf
___________________________________________________________________________________________
[root@test-xlx pki]#  kubectl config set-context default \
>     --cluster=kubernetes \
>     --user=kubelet-bootstrap \
>    --kubeconfig=../bootstrap.conf
Context "default" created.
___________________________________________________________________________________________
# bootstrap set default context
$ kubectl config use-context default --kubeconfig=../bootstrap.conf
___________________________________________________________________________________________
[root@test-xlx pki]#  kubectl config use-context default --kubeconfig=../bootstrap.conf
Switched to context "default".
___________________________________________________________________________________________

Admin certificate

下载admin-csr.json文件，并生成 admin certificate 证书：
$ wget "${PKI_URL}/admin-csr.json" 【可能无法下载】
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
cat >> admin-csr.json << EOF
{
    "CN": "kube-admin",
    "hosts": [
        "10.10.175.3"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shanghai",
            "L": "Shanghai",
            "O": "system:masters",
            "OU": "System"
        }
    ]
}
EOF
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$ cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare admin
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# cfssl gencert \
>   -ca=ca.pem \
>   -ca-key=ca-key.pem \
>   -config=ca-config.json \
>   -profile=kubernetes \
>   admin-csr.json | cfssljson -bare admin
>   admin-csr.json | cfssljson -bare admin
2021/03/17 09:59:31 [INFO] generate received request
2021/03/17 09:59:31 [INFO] received CSR
2021/03/17 09:59:31 [INFO] generating key: rsa-2048
2021/03/17 09:59:31 [INFO] encoded CSR
2021/03/17 09:59:31 [INFO] signed certificate with serial number 632498633726938545834618549292840159760184158107
2021/03/17 09:59:31 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements"). 
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# ls admin*.pem
admin-key.pem  admin.pem
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# ls
admin.csr       admin-key.pem  apiserver.csr       apiserver-key.pem  ca-config.json  ca-csr.json  ca.pem
admin-csr.json  admin.pem      apiserver-csr.json  apiserver.pem      ca.csr          ca-key.pem
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
接着通过以下指令生成名称为 admin.conf 的 kubeconfig 文件：
# admin set-cluster
$ kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=../admin.conf
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-cluster kubernetes \
>     --certificate-authority=ca.pem \
>     --embed-certs=true \
>     --server=${KUBE_APISERVER} \
>     --kubeconfig=../admin.conf
Cluster "kubernetes" set.
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
# admin set-credentials
$ kubectl config set-credentials kubernetes-admin \
    --client-certificate=admin.pem \
    --client-key=admin-key.pem \
    --embed-certs=true \
    --kubeconfig=../admin.conf
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-credentials kubernetes-admin \
>     --client-certificate=admin.pem \
>     --client-key=admin-key.pem \
>     --embed-certs=true \
>     --kubeconfig=../admin.conf
User "kubernetes-admin" set.
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
# admin set-context
$ kubectl config set-context kubernetes-admin@kubernetes \
    --cluster=kubernetes \
    --user=kubernetes-admin \
    --kubeconfig=../admin.conf
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-context kubernetes-admin@kubernetes \
>     --cluster=kubernetes \
>     --user=kubernetes-admin \
>     --kubeconfig=../admin.conf
Context "kubernetes-admin@kubernetes" created.
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
# admin set default context
$ kubectl config use-context kubernetes-admin@kubernetes \
    --kubeconfig=../admin.conf
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config use-context kubernetes-admin@kubernetes \
>     --kubeconfig=../admin.conf
Switched to context "kubernetes-admin@kubernetes".
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————

Controller manager certificate

下载manager-csr.json文件，并生成 kube-controller-manager certificate 证书：
$ wget "${PKI_URL}/manager-csr.json" 【可能无法下载】
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
[root@test-xlx pki]# cat  manager-csr.json
{
    "CN": "system:kube-controller-manager",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.170.170"
    ],
    "names": [
      {
        "C": "CN",
        "ST": "Hubei",
        "L": "Wuhan",
        "O": "system:kube-controller-manager",
        "OU": "system"
      }
    ]
}
若节点 IP 不同，需要修改manager-csr.json的hosts。
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
$ cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  manager-csr.json | cfssljson -bare controller-manager
——————————————————————————————————————————————————————————————————————————————————————————————————————————-——————————————————
[root@test-xlx pki]#  cfssl gencert \
>   -ca=ca.pem \
>   -ca-key=ca-key.pem \
>   -config=ca-config.json \
>   -profile=kubernetes \
>   manager-csr.json | cfssljson -bare controller-manager
2021/03/17 15:39:34 [INFO] generate received request
2021/03/17 15:39:34 [INFO] received CSR
2021/03/17 15:39:34 [INFO] generating key: rsa-2048
2021/03/17 15:39:35 [INFO] encoded CSR
2021/03/17 15:39:35 [INFO] signed certificate with serial number 261293080311829436818998124701626478945424524174
2021/03/17 15:39:35 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
——————————————————————————————————————————————————————————————————————————————————————————————————————————-——————————————————
[root@test-xlx pki]# ls controller-manager*.pem
controller-manager-key.pem  controller-manager.pem
接着通过以下指令生成名称为controller-manager.conf的 kubeconfig 文件：
# controller-manager set-cluster
$ kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=../controller-manager.conf
# controller-manager set-credentials
$ kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=controller-manager.pem \
    --client-key=controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=../controller-manager.conf
# controller-manager set-context
$ kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=../controller-manager.conf
# controller-manager set default context
$ kubectl config use-context system:kube-controller-manager@kubernetes \
    --kubeconfig=../controller-manager.conf
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-cluster kubernetes \
>     --certificate-authority=ca.pem \
>     --embed-certs=true \
>     --server=${KUBE_APISERVER} \
>     --kubeconfig=../controller-manager.conf
Cluster "kubernetes" set.
[root@test-xlx pki]# kubectl config set-credentials system:kube-controller-manager \
>     --client-certificate=controller-manager.pem \
>     --client-key=controller-manager-key.pem \
>     --embed-certs=true \
>     --kubeconfig=../controller-manager.conf
User "system:kube-controller-manager" set.
[root@test-xlx pki]# kubectl config set-context system:kube-controller-manager@kubernetes \
>     --cluster=kubernetes \
>     --user=system:kube-controller-manager \
>     --kubeconfig=../controller-manager.conf
Context "system:kube-controller-manager@kubernetes" created.
[root@test-xlx pki]# kubectl config use-context system:kube-controller-manager@kubernetes \
>     --kubeconfig=../controller-manager.conf
Switched to context "system:kube-controller-manager@kubernetes".
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

Scheduler certificate

下载scheduler-csr.json文件，并生成 kube-scheduler certificate 证书：
wget "${PKI_URL}/scheduler-csr.json"【可能无法下载】
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
生成scheduler-csr.json配置文件
[root@test-xlx pki]# pwd
/etc/kubernetes/pki
cat >> scheduler-csr.json << EOF
{
    "CN": "system:kube-scheduler",
    "hosts": [
        "192.168.170.170",
        "192.168.170.171",
        "192.168.170.172"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "Shanghai",
            "L": "Shanghai",
            "O": "system:kube-scheduler",
            "OU": "System"
        }
    ]
}
EOF
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  scheduler-csr.json | cfssljson -bare scheduler
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# cfssl gencert   -ca=ca.pem   -ca-key=ca-key.pem   -config=ca-config.json   -profile=kubernetes   scheduler-csr.json | cfssljson -bare scheduler
2021/03/26 07:40:56 [INFO] generate received request
2021/03/26 07:40:56 [INFO] received CSR
2021/03/26 07:40:56 [INFO] generating key: rsa-2048
2021/03/26 07:40:56 [INFO] encoded CSR
2021/03/26 07:40:56 [INFO] signed certificate with serial number 715503307064271558981204673099714115916622820615
2021/03/26 07:40:56 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
【若节点 IP 不同，需要修改scheduler-csr.json的hosts。】
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# ls scheduler*.pem
scheduler-key.pem  scheduler.pem

接着通过以下指令生成名称为 scheduler.conf 的 kubeconfig 文件

# scheduler set-cluster
$ kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=../scheduler.conf
# scheduler set-credentials
$ kubectl config set-credentials system:kube-scheduler \
    --client-certificate=scheduler.pem \
    --client-key=scheduler-key.pem \
    --embed-certs=true \
    --kubeconfig=../scheduler.conf
# scheduler set-context
$ kubectl config set-context system:kube-scheduler@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-scheduler \
    --kubeconfig=../scheduler.conf
# scheduler set default context
$ kubectl config use-context system:kube-scheduler@kubernetes \
    --kubeconfig=../scheduler.conf
————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-cluster kubernetes \
>     --certificate-authority=ca.pem \
>     --embed-certs=true \
>     --server=${KUBE_APISERVER} \
>     --kubeconfig=../scheduler.conf
Cluster "kubernetes" set.
[root@test-xlx pki]# kubectl config set-credentials system:kube-scheduler \
>     --client-certificate=scheduler.pem \
>     --client-key=scheduler-key.pem \
>     --embed-certs=true \
>     --kubeconfig=../scheduler.conf
User "system:kube-scheduler" set.
[root@test-xlx pki]# kubectl config set-context system:kube-scheduler@kubernetes \
>     --cluster=kubernetes \
>     --user=system:kube-scheduler \
>     --kubeconfig=../scheduler.conf
Context "system:kube-scheduler@kubernetes" created.
[root@test-xlx pki]# kubectl config use-context system:kube-scheduler@kubernetes \
>     --kubeconfig=../scheduler.conf
Switched to context "system:kube-scheduler@kubernetes".
————————————————————————————————————————————————————————————————————————————————————————————————————————

Kubelet master certificate

下载kubelet-csr.json文件，并生成 master node certificate 证书：
wget "${PKI_URL}/kubelet-csr.json"【可能无法下载】
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
cat >> kubelet-csr.json << EOF
{
    "CN": "system:node:master01",
    "hosts": [
        "test-xlx",
        "192.168.170.170"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Shanghai",
            "ST": "Shanghai",
            "O": "system:nodes",
            "OU": "Kubernetes-manual"
        }
    ]
}
EOF
【sed -i 's/$NODE/test-xlx/g' kubelet-csr.json】
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
cfssl gencert \
  -ca=ca.pem \
  -ca-key=ca-key.pem \
  -config=ca-config.json \
  -hostname=test-xlx,192.168.170.170,192.168.170.170 \
  -profile=kubernetes \
  kubelet-csr.json | cfssljson -bare kubelet
【这边$NODE需要随节点名称不同而改变。】
______________________________________________________________________________________________________________________
[root@test-xlx pki]# cfssl gencert \
>   -ca=ca.pem \
>   -ca-key=ca-key.pem \
>   -config=ca-config.json \
>   -hostname=test-xlx,192.168.170.170,192.168.170.170 \
>   -profile=kubernetes \
>   kubelet-csr.json | cfssljson -bare kubelet
2021/03/26 07:49:20 [INFO] generate received request
2021/03/26 07:49:20 [INFO] received CSR
2021/03/26 07:49:20 [INFO] generating key: rsa-2048
2021/03/26 07:49:20 [INFO] encoded CSR
2021/03/26 07:49:20 [INFO] signed certificate with serial number 21765872373764965684739476885736354706459574089
______________________________________________________________________________________________________________________
[root@test-xlx pki]# ls kubelet*.pem
kubelet-key.pem  kubelet.pem
接着通过以下指令生成名称为 kubelet.conf 的 kubeconfig 文件：
$ kubectl config set-cluster kubernetes \
    --certificate-authority=ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=../kubelet.conf
# kubelet set-credentials
$ kubectl config set-credentials system:node:test-xlx \
    --client-certificate=kubelet.pem \
    --client-key=kubelet-key.pem \
    --embed-certs=true \
    --kubeconfig=../kubelet.conf
# kubelet set-context
$ kubectl config set-context system:node:test-xlx@kubernetes \
    --cluster=kubernetes \
    --user=system:node:test-xlx \
    --kubeconfig=../kubelet.conf
# kubelet set default context
$ kubectl config use-context system:node:test-xlx@kubernetes \
    --kubeconfig=../kubelet.conf
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
[root@test-xlx pki]# kubectl config set-cluster kubernetes \
>     --certificate-authority=ca.pem \
>     --embed-certs=true \
>     --server=${KUBE_APISERVER} \
>     --kubeconfig=../kubelet.conf
[root@test-xlx pki]# kubectl config set-credentials system:node:test-xlx \
>     --client-certificate=kubelet.pem \
>     --client-key=kubelet-key.pem \
>     --embed-certs=true \
>     --kubeconfig=../kubelet.conf
User "system:node:test-xlx" set.
[root@test-xlx pki]#  kubectl config set-context system:node:test-xlx@kubernetes \
>     --cluster=kubernetes \
>     --user=system:node:test-xlx \
>     --kubeconfig=../kubelet.conf
Context "system:node:test-xlx@kubernetes" created.
[root@test-xlx pki]#  kubectl config use-context system:node:test-xlx@kubernetes \
>     --kubeconfig=../kubelet.conf
Switched to context "system:node:test-xlx@kubernetes".
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

Service account key

Service account 不是通过 CA 进行认证，因此不要通过 CA 来做 Service account key 的检查，这边建立一组 Private 与 Public 密钥提供给 Service account key 使用：
[root@test-xlx pki]#  openssl genrsa -out sa.key 2048
[root@test-xlx pki]# openssl rsa -in sa.key -pubout -out sa.pub
__________________________________________________________________________________________________________________________________________________________
[root@test-xlx pki]#  openssl genrsa -out sa.key 2048
Generating RSA private key, 2048 bit long modulus
...........+++
.........+++
e is 65537 (0x10001)
[root@test-xlx pki]# openssl rsa -in sa.key -pubout -out sa.pub
writing RSA key
__________________________________________________________________________________________________________________________________________________________
[root@test-xlx pki]#  ls sa.*
sa.key  sa.pub