从k8s到PaaS：k8s-02-核心组件

k8s-centos8u2-集群部署02-核心组件：kube-apiserver、kube-controller-manager、kube-schedule、kubelet、kube-proxy

本实验环境每次关机后重启需要检查：

• keepalived是否工作（systemctl status keepalived），vip是否正常。（ip addr查看192.168.26.10是否存在）
• harbaor启动是否正常：在启动目录下docker-compose ps查看是否正常。
• supervisorctl status：查看各进程启动状态。
• 检查docker和k8s集群。

kube-apiserver

1 集群规划

注意：这里192.168.26.11和192.168.26.12使用nginx做4层负载均衡器，用keepalived跑一个vip：192.168.26.10，代理两个kube-apiserver，实现高可用。

这里部署文档以vms21.cos.com主机为例，另外一台运算节点安装部署方法类似。

2 下载软件，解压，做软连接

在vms21.cos.com上：

下载地址：https://github.com/kubernetes/kubernetes/releases

• 选择下载版本Downloads for v1.18.5
• 从Server binaries列表中选择：

右键复制下载地址：https://dl.k8s.io/v1.18.5/kubernetes-server-linux-amd64.tar.gz

[root@vms21 ~]# mkdir /opt/src
[root@vms21 ~]# cd /opt/src
[root@vms21 src]# wget https://dl.k8s.io/v1.18.5/kubernetes-server-linux-amd64.tar.gz
[root@vms21 src]# tar xf kubernetes-server-linux-amd64.tar.gz -C /opt
[root@vms21 src]# mv /opt/kubernetes /opt/kubernetes-v1.18.5
[root@vms21 src]# ln -s /opt/kubernetes-v1.18.5 /opt/kubernetes
[root@vms21 src]# mkdir /opt/kubernetes/server/bin/{cert,conf}
[root@vms21 src]# ls -l /opt|grep kubernetes
lrwxrwxrwx 1 root root  23 Jul 15 10:46 kubernetes -> /opt/kubernetes-v1.18.5
drwxr-xr-x 4 root root  79 Jun 26 12:26 kubernetes-v1.18.5

删除一些不必要的包和文件，只保留如下：

[root@vms21 src]# ls -l /opt/kubernetes/server/bin
total 458360
drwxr-xr-x 2 root root         6 Jul 15 10:47 cert
drwxr-xr-x 2 root root         6 Jul 15 10:47 conf
-rwxr-xr-x 1 root root 120659968 Jun 26 12:26 kube-apiserver
-rwxr-xr-x 1 root root 110059520 Jun 26 12:26 kube-controller-manager
-rwxr-xr-x 1 root root  44027904 Jun 26 12:26 kubectl
-rwxr-xr-x 1 root root 113283800 Jun 26 12:26 kubelet
-rwxr-xr-x 1 root root  38379520 Jun 26 12:26 kube-proxy
-rwxr-xr-x 1 root root  42946560 Jun 26 12:26 kube-scheduler

3 签发client证书

apiserver与etc通信用的证书。apiserver是客户端，etcd是服务端

运维主机vms200.cos.com上：

• 创建生成证书签名请求（csr）的JSON配置文件

[root@vms200 certs]# vi /opt/certs/client-csr.json

{
    "CN": "k8s-node",
    "hosts": [
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "op",
            "OU": "ops"
        }
    ]
}

• 生成client证书和私钥

[root@vms200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json |cfssl-json -bare client
2020/07/16 11:20:54 [INFO] generate received request
2020/07/16 11:20:54 [INFO] received CSR
2020/07/16 11:20:54 [INFO] generating key: rsa-2048
2020/07/16 11:20:54 [INFO] encoded CSR
2020/07/16 11:20:54 [INFO] signed certificate with serial number 384811578816106154028165762700148655012765287575
2020/07/16 11:20:54 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@vms200 certs]# ls -l client*
-rw-r--r-- 1 root root  993 Jul 16 11:20 client.csr
-rw-r--r-- 1 root root  280 Jul 16 11:14 client-csr.json
-rw------- 1 root root 1675 Jul 16 11:20 client-key.pem
-rw-r--r-- 1 root root 1363 Jul 16 11:20 client.pem

4 签发server证书

apiserver和其它k8s组件通信使用

运维主机vms200.cos.com上：

• 创建生成证书签名请求（csr）的JSON配置文件

[root@vms200 certs]# vi /opt/certs/apiserver-csr.json

{
    "CN": "k8s-apiserver",
    "hosts": [
        "127.0.0.1",
        "10.168.0.1",
        "kubernetes.default",
        "kubernetes.default.svc",
        "kubernetes.default.svc.cluster",
        "kubernetes.default.svc.cluster.local",
        "192.168.26.10",
        "192.168.26.21",
        "192.168.26.22",
        "192.168.26.23"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "op",
            "OU": "ops"
        }
    ]
}

• 生成kube-apiserver证书和私钥

[root@vms200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server apiserver-csr.json |cfssl-json -bare apiserver
2020/07/16 14:04:16 [INFO] generate received request
2020/07/16 14:04:16 [INFO] received CSR
2020/07/16 14:04:16 [INFO] generating key: rsa-2048
2020/07/16 14:04:17 [INFO] encoded CSR
2020/07/16 14:04:17 [INFO] signed certificate with serial number 462064728215999941525650774614395532190617463799
2020/07/16 14:04:17 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
[root@vms200 certs]# ls -l apiserver*
-rw-r--r-- 1 root root 1249 Jul 16 14:04 apiserver.csr
-rw-r--r-- 1 root root  581 Jul 16 11:33 apiserver-csr.json
-rw------- 1 root root 1679 Jul 16 14:04 apiserver-key.pem
-rw-r--r-- 1 root root 1594 Jul 16 14:04 apiserver.pem

5 拷贝证书

在vms21.cos.com上：

[root@vms21 cert]# cd /opt/kubernetes/server/bin/cert/
[root@vms21 cert]# scp vms200:/opt/certs/ca.pem .
[root@vms21 cert]# scp vms200:/opt/certs/ca-key.pem .
[root@vms21 cert]# scp vms200:/opt/certs/client.pem .
[root@vms21 cert]# scp vms200:/opt/certs/client-key.pem .
[root@vms21 cert]# scp vms200:/opt/certs/apiserver.pem .
[root@vms21 cert]# scp vms200:/opt/certs/apiserver-key.pem .
[root@vms21 cert]# ls -l
total 24
-rw------- 1 root root 1679 Jul 16 14:14 apiserver-key.pem
-rw-r--r-- 1 root root 1594 Jul 16 14:14 apiserver.pem
-rw------- 1 root root 1679 Jul 16 14:13 ca-key.pem
-rw-r--r-- 1 root root 1338 Jul 16 14:12 ca.pem
-rw------- 1 root root 1675 Jul 16 14:13 client-key.pem
-rw-r--r-- 1 root root 1363 Jul 16 14:13 client.pem

6 创建配置

在vms21.cos.com上：

[root@vms21 bin]# cd /opt/kubernetes/server/bin/conf/
[root@vms21 conf]# vi audit.yaml

apiVersion: audit.k8s.io/v1beta1 # This is required.
kind: Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages:
  - "RequestReceived"
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
    - group: ""
      # Resource "pods" doesn't match requests to any subresource of pods,
      # which is consistent with the RBAC policy.
      resources: ["pods"]
  # Log "pods/log", "pods/status" at Metadata level
  - level: Metadata
    resources:
    - group: ""
      resources: ["pods/log", "pods/status"]
  # Don't log requests to a configmap called "controller-leader"
  - level: None
    resources:
    - group: ""
      resources: ["configmaps"]
      resourceNames: ["controller-leader"]
  # Don't log watch requests by the "system:kube-proxy" on endpoints or services
  - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    resources:
    - group: "" # core API group
      resources: ["endpoints", "services"]
  # Don't log authenticated requests to certain non-resource URL paths.
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
    - "/api*" # Wildcard matching.
    - "/version"
  # Log the request body of configmap changes in kube-system.
  - level: Request
    resources:
    - group: "" # core API group
      resources: ["configmaps"]
    # This rule only applies to resources in the "kube-system" namespace.
    # The empty string "" can be used to select non-namespaced resources.
    namespaces: ["kube-system"]
  # Log configmap and secret changes in all other namespaces at the Metadata level.
  - level: Metadata
    resources:
    - group: "" # core API group
      resources: ["secrets", "configmaps"]
  # Log all other resources in core and extensions at the Request level.
  - level: Request
    resources:
    - group: "" # core API group
    - group: "extensions" # Version of group should NOT be included.
  # A catch-all rule to log all other requests at the Metadata level.
  - level: Metadata
    # Long-running requests like watches that fall under this rule will not
    # generate an audit event in RequestReceived.
    omitStages:
      - "RequestReceived"

7 创建启动脚本

在vms21.cos.com上：

[root@vms21 bin]# vi /opt/kubernetes/server/bin/kube-apiserver.sh

#!/bin/bash
./kube-apiserver \
  --apiserver-count 2 \
  --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log \
  --audit-policy-file ./conf/audit.yaml \
  --authorization-mode RBAC \
  --client-ca-file ./cert/ca.pem \
  --requestheader-client-ca-file ./cert/ca.pem \
  --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota \
  --etcd-cafile ./cert/ca.pem \
  --etcd-certfile ./cert/client.pem \
  --etcd-keyfile ./cert/client-key.pem \
  --etcd-servers https://192.168.26.12:2379,https://192.168.26.21:2379,https://192.168.26.22:2379 \
  --service-account-key-file ./cert/ca-key.pem \
  --service-cluster-ip-range 10.168.0.0/16 \
  --service-node-port-range 3000-29999 \
  --target-ram-mb=1024 \
  --kubelet-client-certificate ./cert/client.pem \
  --kubelet-client-key ./cert/client-key.pem \
  --log-dir  /data/logs/kubernetes/kube-apiserver \
  --tls-cert-file ./cert/apiserver.pem \
  --tls-private-key-file ./cert/apiserver-key.pem \
  --v 2 \
  --enable-aggregator-routing=true \
  --requestheader-client-ca-file=./cert/ca.pem \
  --requestheader-allowed-names=aggregator,metrics-server \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=./cert/metrics-server.pem \
  --proxy-client-key-file=./cert/metrics-server-key.pem

• apiserver-count 2：apiserver的数量
• 修改etcd-servers、service-cluster-ip-range
• 查看帮助命令，查看每行参数的意思：

[root@vms21 bin]# ./kube-apiserver --help|grep -A 5 target-ram-mb
      --target-ram-mb int     Memory limit for apiserver in MB (used to configure sizes of caches, etc.)

8 调整权限和目录

在vms21.cos.com上：

[root@vms21 bin]# chmod +x /opt/kubernetes/server/bin/kube-apiserver.sh
[root@vms21 bin]# mkdir -p /data/logs/kubernetes/kube-apiserver

9 创建supervisor配置

在vms21.cos.com上：

[root@vms21 bin]# vi /etc/supervisord.d/kube-apiserver.ini

[program:kube-apiserver-26-21]                    # 21根据实际IP地址更改
command=/opt/kubernetes/server/bin/kube-apiserver.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                            ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log        ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true

10 启动服务并检查

在vms21.cos.com上：

[root@vms21 bin]# supervisorctl update
kube-apiserver-26-21: added process group
[root@vms21 bin]# supervisorctl status kube-apiserver-26-21
kube-apiserver-26-21              RUNNING   pid 2016, uptime 0:00:57
[root@vms21 bin]# tail /data/logs/kubernetes/kube-apiserver/apiserver.stdout.log  #查看日志
[root@vms21 bin]# netstat -luntp | grep kube-api
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      2017/./kube-apiserv
tcp6       0      0 :::6443                 :::*                    LISTEN      2017/./kube-apiserv
[root@vms21 bin]# ps uax|grep kube-apiserver|grep -v grep
root        2016  0.0  0.1 234904  3528 ?        S    14:54   0:00 /bin/bash /opt/kubernetes/server/bin/kube-apiserver.sh
root        2017 21.7 18.9 496864 380332 ?       Sl   14:54   0:46 ./kube-apiserver --apiserver-count 2 --audit-log-path /data/logs/kubernetes/kube-apiserver/audit-log --audit-policy-file ./conf/audit.yaml --authorization-mode RBAC --client-ca-file ./cert/ca.pem --requestheader-client-ca-file ./cert/ca.pem --enable-admission-plugins NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --etcd-cafile ./cert/ca.pem --etcd-certfile ./cert/client.pem --etcd-keyfile ./cert/client-key.pem --etcd-servers https://192.168.26.12:2379,https://192.168.26.21:2379,https://192.168.26.22:2379 --service-account-key-file ./cert/ca-key.pem --service-cluster-ip-range 10.168.0.0/16 --service-node-port-range 3000-29999 --target-ram-mb=1024 --kubelet-client-certificate ./cert/client.pem --kubelet-client-key ./cert/client-key.pem --log-dir /data/logs/kubernetes/kube-apiserver --tls-cert-file ./cert/apiserver.pem --tls-private-key-file ./cert/apiserver-key.pem --v 2

启停apiserver的命令：

# supervisorctl start kube-apiserver-26-21
# supervisorctl stop kube-apiserver-26-21
# supervisorctl restart kube-apiserver-26-21
# supervisorctl status kube-apiserver-26-21

11 快速部署到vms22

在vms22.cos.com上：

• 复制软件、证书、创建目录

[root@vms22 opt]# scp -r vms21:/opt/kubernetes-v1.18.5/ /opt
[root@vms22 opt]# ln -s /opt/kubernetes-v1.18.5 /opt/kubernetes
[root@vms22 opt]# ls -l /opt|grep kubernetes
lrwxrwxrwx 1 root root  23 Jul 16 15:22 kubernetes -> /opt/kubernetes-v1.18.5
drwxr-xr-x 4 root root  79 Jul 16 15:20 kubernetes-v1.18.5
[root@vms22 opt]# ls -l /opt/kubernetes/server/bin
total 458364
drwxr-xr-x 2 root root       124 Jul 16 15:20 cert
drwxr-xr-x 2 root root        43 Jul 16 15:20 conf
-rwxr-xr-x 1 root root 120659968 Jul 16 15:20 kube-apiserver
-rwxr-xr-x 1 root root      1089 Jul 16 15:20 kube-apiserver.sh
-rwxr-xr-x 1 root root 110059520 Jul 16 15:20 kube-controller-manager
-rwxr-xr-x 1 root root  44027904 Jul 16 15:20 kubectl
-rwxr-xr-x 1 root root 113283800 Jul 16 15:20 kubelet
-rwxr-xr-x 1 root root  38379520 Jul 16 15:20 kube-proxy
-rwxr-xr-x 1 root root  42946560 Jul 16 15:20 kube-scheduler
[root@vms22 opt]# mkdir -p /data/logs/kubernetes/kube-apiserver

• 创建supervisor配置kube-apiserver-26-22

[root@vms22 opt]# vi /etc/supervisord.d/kube-apiserver.ini

[program:kube-apiserver-26-22]
command=/opt/kubernetes/server/bin/kube-apiserver.sh            ; the program (relative uses PATH, can take args)
numprocs=1                                                      ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                            ; directory to cwd to before exec (def no cwd)
autostart=true                                                  ; start at supervisord start (default: true)
autorestart=true                                                ; retstart at unexpected quit (default: true)
startsecs=30                                                    ; number of secs prog must stay running (def. 1)
startretries=3                                                  ; max # of serial start failures (default 3)
exitcodes=0,2                                                   ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                 ; signal used to kill process (default TERM)
stopwaitsecs=10                                                 ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                       ; setuid to this UNIX account to run the program
redirect_stderr=true                                            ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-apiserver/apiserver.stdout.log        ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                    ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                        ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                     ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                     ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true

• 启动服务并检查。同上

12 配4层反向代理nginx + keepalived

vms11.cos.com、vms12.cos.com上：

安装nginx和keepalived

nginx安装

[root@vms11 ~]# rpm -qa nginx
[root@vms11 ~]# yum install -y nginx
...
[root@vms11 ~]# rpm -qa nginx
nginx-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64

keepalived安装

[root@vms11 ~]# rpm -qa  keepalived
[root@vms11 ~]# yum install keepalived -y
...
[root@vms11 ~]# rpm -qa  keepalived
keepalived-2.0.10-10.el8.x86_64

nginx配置

[root@vms11 ~]# vi /etc/nginx/nginx.conf

• 末尾加上以下内容，stream 只能加在 main 中
• 此处只是简单配置下nginx，实际生产中，建议进行更合理的配置

stream {
    upstream kube-apiserver {
        server 192.168.26.21:6443     max_fails=3 fail_timeout=30s;
        server 192.168.26.22:6443     max_fails=3 fail_timeout=30s;
    }
    server {
        listen 8443;
        proxy_connect_timeout 2s;
        proxy_timeout 900s;
        proxy_pass kube-apiserver;
    }
}

检查配置、启动、测试

[root@vms11 ~]# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
[root@vms11 ~]# systemctl enable nginx
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/systemd/system/nginx.service.
[root@vms11 ~]# systemctl start nginx
[root@vms11 ~]# curl 127.0.0.1:8443  #测试几次
Client sent an HTTP request to an HTTPS server.

keepalived配置

参考链接：https://www.cnblogs.com/zeq912/p/11065199.html

[root@vms11 ~]# vi /etc/keepalived/check_port.sh #vms12也要配置

#!/bin/bash
#keepalived 监控端口脚本
#使用方法：
#在keepalived的配置文件中
#vrrp_script check_port {#创建一个vrrp_script脚本,检查配置
#    script "/etc/keepalived/check_port.sh 6379" #配置监听的端口
#    interval 2 #检查脚本的频率,单位（秒）
#}
CHK_PORT=$1
if [ -n "$CHK_PORT" ];then
        PORT_PROCESS=`ss -lnt|grep $CHK_PORT|wc -l`
        if [ $PORT_PROCESS -eq 0 ];then
                echo "Port $CHK_PORT Is Not Used,End."
                #systemctl stop keepalived
                exit 1
        fi
else
        echo "Check Port Cant Be Empty!"
fi

注释systemctl stop keepalived后主备自动切换，VIP自动漂移。否则，VIP不会自动切换和漂移，需要人工确认及操作

测试：

[root@vms11 ~]# ss -lnt|grep 8443|wc -l
1
[root@vms12 ~]# ss -lnt|grep 8443|wc -l
1

参考脚本：

#!/bin/bash
if [ $# -eq 1 ] && [[ $1 =~ ^[0-9]+ ]];then
    [ $(netstat -lntp|grep ":$1 " |wc -l) -eq 0 ] && echo "[ERROR] nginx may be not running!" && exit 1 || exit 0
else
    echo "[ERROR] need one port!"
    exit 1
fi

chmod +x /etc/keepalived/check_port.sh

配置主节点（vms11）：[root@vms11 ~]# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
   router_id 192.168.26.11
}
vrrp_script chk_nginx {
    script "/etc/keepalived/check_port.sh 8443"
    interval 2
    weight -20
}
vrrp_instance VI_1 {
    state MASTER
    interface ens160
    virtual_router_id 251
    priority 100
    advert_int 1
    mcast_src_ip 192.168.26.11
    nopreempt
    authentication {
        auth_type PASS
        auth_pass 11111111
    }
    track_script {
         chk_nginx
    }
    virtual_ipaddress {
        192.168.26.10
    }
}

• interface # 根据实际网卡更改，用ifconfig查看
• 主节点中，必须加上 nopreempt
  • 因为一旦因为网络抖动导致VIP漂移，不能让它自动飘回来，必须要分析原因后手动迁移VIP到主节点！如主节点确认正常后，重启备节点的keepalive，让VIP飘到主节点。
• keepalived 的日志输出配置此处省略，生产中需要进行处理。

配置备节点（vms12）：[root@vms12 ~]# vi /etc/keepalived/keepalived.conf

! Configuration File for keepalived
global_defs {
        router_id 192.168.26.12
        script_user root
        enable_script_security
}
vrrp_script chk_nginx {
        script "/etc/keepalived/check_port.sh 8443"
        interval 2
        weight -20
}
vrrp_instance VI_1 {
        state BACKUP
        interface ens160
        virtual_router_id 251
        mcast_src_ip 192.168.26.12
        priority 90
        advert_int 1
        authentication {
                auth_type PASS
                auth_pass 11111111
        }
        track_script {
                chk_nginx
        }
        virtual_ipaddress {
                192.168.26.10
        }
}

启动keepalived，并检查VIP（192.168.26.10）是否出现

[root@vms11 ~]# systemctl start keepalived ; systemctl enable keepalived
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.
[root@vms12 ~]# systemctl start keepalived ; systemctl enable keepalived
Created symlink /etc/systemd/system/multi-user.target.wants/keepalived.service → /usr/lib/systemd/system/keepalived.service.
[root@vms11 ~]# netstat -luntp | grep 8443
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      24891/nginx: master
[root@vms12 ~]# netstat -luntp | grep 8443
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      930/nginx: master p
[root@vms11 ~]# ip addr show ens160
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:ce:4b:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.11/24 brd 192.168.26.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.26.10/32 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fece:4bd4/64 scope link
       valid_lft forever preferred_lft forever

测试主备（自动）切换、VIP漂移

• 在vms11主：停nginx，VIP（192.168.26.10）漂移走

[root@vms11 ~]# netstat -luntp | grep 8443
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      20526/nginx: master
[root@vms11 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:ce:4b:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.11/24 brd 192.168.26.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.26.10/32 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fece:4bd4/64 scope link
       valid_lft forever preferred_lft forever
[root@vms11 ~]# nginx -s stop
[root@vms11 ~]# netstat -luntp | grep 8443
[root@vms11 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:ce:4b:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.11/24 brd 192.168.26.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fece:4bd4/64 scope link
       valid_lft forever preferred_lft forever

• 在vms12从：VIP漂移到备上

[root@vms12 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:40:05:87 brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.12/24 brd 192.168.26.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.26.10/32 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe40:587/64 scope link
       valid_lft forever preferred_lft forever

• 在vms11主：启动nginx，过一会VIP自动漂移回来

[root@vms11 ~]# nginx
[root@vms11 ~]# netstat -luntp | grep 8443
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      21045/nginx: master
[root@vms11 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:ce:4b:d4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.11/24 brd 192.168.26.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet 192.168.26.10/32 scope global ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fece:4bd4/64 scope link
       valid_lft forever preferred_lft forever

在vms12从：VIP已经自动漂移到主了

[root@vms12 ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:40:05:87 brd ff:ff:ff:ff:ff:ff
    inet 192.168.26.12/24 brd 192.168.26.255 scope global noprefixroute ens160
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe40:587/64 scope link
       valid_lft forever preferred_lft forever

主的上面nginx杀掉后，vip漂移到备上能正常访问，主的nginx起来后，vip又飘回到主上。

实际生产情况一般要求：Keepalived设置master故障恢复后不重新抢回VIP，需人工确认好后才能切换和漂移

如果vip出现变动，主keepalived恢复后，一定要确认主keepalived端口起来， 服务搞好后，再重启keepalived（主、备），让vip变回主keepalived。

此时需要修改检查脚本。参考脚本：/etc/keepalived/check_keepalived.sh

#!/bin/bash
NGINX_SBIN=`which nginx`
NGINX_PORT=$1
function check_nginx(){
    NGINX_STATUS=`nmap localhost -p ${NGINX_PORT} | grep "8443/tcp open" | awk '{print $2}'`
    NGINX_PROCESS=`ps -ef | grep nginx|grep -v grep|wc -l`
}
check_nginx
if [ "$NGINX_STATUS" != "open"  -o  $NGINX_PROCESS -lt 2 ]
then
    ${NGINX_SBIN} -s stop
    ${NGINX_SBIN}
    sleep 3
    check_nginx
    if [ "$NGINX_STATUS" != "open"  -o  $NGINX_PROCESS -lt 2 ];then
        systemctl stop keepalived
    fi
fi

kube-controller-manager

1 集群规划

注意：这里部署文档以vms21.cos.com主机为例，另外一台运算节点安装部署方法类似。

controller-manager 设置为只调用当前机器的 apiserver，走127.0.0.1网卡，因此不配置SSL证书。

2 创建启动脚本

vms21.cos.com上：

[root@vms21 bin]# vi /opt/kubernetes/server/bin/kube-controller-manager.sh

#!/bin/sh
./kube-controller-manager \
  --cluster-cidr 172.26.0.0/16 \
  --leader-elect true \
  --log-dir /data/logs/kubernetes/kube-controller-manager \
  --master http://127.0.0.1:8080 \
  --service-account-private-key-file ./cert/ca-key.pem \
  --service-cluster-ip-range 10.168.0.0/16 \
  --root-ca-file ./cert/ca.pem \
  --v 2

[root@vms21 bin]# mkdir -p /data/logs/kubernetes/kube-controller-manager
[root@vms21 bin]# chmod +x /opt/kubernetes/server/bin/kube-controller-manager.sh

3 创建supervisor配置

[root@vms21 bin]# vi /etc/supervisord.d/kube-conntroller-manager.ini

[program:kube-controller-manager-26-21]
command=/opt/kubernetes/server/bin/kube-controller-manager.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                                        ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                              ; directory to cwd to before exec (def no cwd)
autostart=true                                                                    ; start at supervisord start (default: true)
autorestart=true                                                                  ; retstart at unexpected quit (default: true)
startsecs=30                                                                      ; number of secs prog must stay running (def. 1)
startretries=3                                                                    ; max # of serial start failures (default 3)
exitcodes=0,2                                                                     ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                                   ; signal used to kill process (default TERM)
stopwaitsecs=10                                                                   ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                         ; setuid to this UNIX account to run the program
redirect_stderr=true                                                              ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-controller-manager/controller.stdout.log  ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                                      ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                          ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                                       ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                                       ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true

注意：vms22上修改为[program:kube-controller-manager-26-22]

4 启动服务并检查

[root@vms21 bin]# supervisorctl update
kube-controller-manager-26-21: added process group
[root@vms21 bin]# supervisorctl status
etcd-server-26-21                RUNNING   pid 1030, uptime 6:15:58
kube-apiserver-26-21              RUNNING   pid 1031, uptime 6:15:58
kube-controller-manager-26-21     RUNNING   pid 1659, uptime 0:00:35

[root@vms22 bin]# supervisorctl update
kube-controller-manager-26-22: added process group
[root@vms22 bin]# supervisorctl status
etcd-server-26-22                RUNNING   pid 1045, uptime 6:20:57
kube-apiserver-26-22              RUNNING   pid 1046, uptime 6:20:57
kube-controller-manager-26-22     RUNNING   pid 1660, uptime 0:00:37

kube-scheduler

1 集群规划

注意：这里部署文档以vms21.cos.com主机为例，另外一台运算节点安装部署方法类似。

kube-scheduler 设置为只调用当前机器的 apiserver，走127.0.0.1网卡，因此不配置SSL证书。

2 创建启动脚本

vms21.cos.com上：

[root@vms21 bin]# vi /opt/kubernetes/server/bin/kube-scheduler.sh

#!/bin/sh
./kube-scheduler \
  --leader-elect  \
  --log-dir /data/logs/kubernetes/kube-scheduler \
  --master http://127.0.0.1:8080 \
  --v 2

[root@vms21 bin]# chmod +x /opt/kubernetes/server/bin/kube-scheduler.sh
[root@vms21 bin]# mkdir -p /data/logs/kubernetes/kube-scheduler

3 创建supervisor配置

[root@vms21 bin]# vi /etc/supervisord.d/kube-scheduler.ini

[program:kube-scheduler-26-21]
command=/opt/kubernetes/server/bin/kube-scheduler.sh                     ; the program (relative uses PATH, can take args)
numprocs=1                                                               ; number of processes copies to start (def 1)
directory=/opt/kubernetes/server/bin                                     ; directory to cwd to before exec (def no cwd)
autostart=true                                                           ; start at supervisord start (default: true)
autorestart=true                                                         ; retstart at unexpected quit (default: true)
startsecs=30                                                             ; number of secs prog must stay running (def. 1)
startretries=3                                                           ; max # of serial start failures (default 3)
exitcodes=0,2                                                            ; 'expected' exit codes for process (default 0,2)
stopsignal=QUIT                                                          ; signal used to kill process (default TERM)
stopwaitsecs=10                                                          ; max num secs to wait b4 SIGKILL (default 10)
user=root                                                                ; setuid to this UNIX account to run the program
redirect_stderr=true                                                     ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/kubernetes/kube-scheduler/scheduler.stdout.log ; stderr log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB                                             ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4                                                 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB                                              ; number of bytes in 'capturemode' (default 0)
stdout_events_enabled=false                                              ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true

注意：vms22上修改为[program:kube-scheduler-26-22]

4 启动服务并检查

[root@vms21 bin]# supervisorctl update
kube-scheduler-26-21: added process group
[root@vms21 bin]# supervisorctl status
etcd-server-26-21                RUNNING   pid 1030, uptime 6:36:44
kube-apiserver-26-21              RUNNING   pid 1031, uptime 6:36:44
kube-controller-manager-26-21     RUNNING   pid 1659, uptime 0:21:21
kube-scheduler-26-21              RUNNING   pid 1726, uptime 0:00:32

[root@vms22 bin]# supervisorctl update
kube-scheduler-26-22: added process group
[root@vms22 bin]# supervisorctl status
etcd-server-26-22                RUNNING   pid 1045, uptime 6:38:26
kube-apiserver-26-22              RUNNING   pid 1046, uptime 6:38:26
kube-controller-manager-26-22     RUNNING   pid 1660, uptime 0:18:06
kube-scheduler-26-22              RUNNING   pid 1688, uptime 0:00:50

5 检查主控节点状态

至此，主控节点组件已经部署完成！

[root@vms21 bin]# ln -s /opt/kubernetes/server/bin/kubectl /usr/bin/kubectl  #给kubectl创建软连接
[root@vms21 bin]# kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}
etcd-1               Healthy   {"health":"true"}
etcd-2               Healthy   {"health":"true"}

kubelet

1 集群规划

注意：这里部署文档以vms21.cos.com主机为例，其它运算节点安装部署方法类似

2 签发kubelet证书

运维主机vms200.cos.com上：

创建生成证书签名请求（csr）的JSON配置文件

/opt/certs：将所有可能的kubelet机器IP添加到hosts中 [root@vms200 certs]# vi kubelet-csr.json

{
    "CN": "k8s-kubelet",
    "hosts": [
    "127.0.0.1",
    "192.168.26.10",
    "192.168.26.21",
    "192.168.26.22",
    "192.168.26.23",
    "192.168.26.24",
    "192.168.26.25",
    "192.168.26.26"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "ST": "beijing",
            "L": "beijing",
            "O": "op",
            "OU": "ops"
        }
    ]
}

生成kubelet证书和私钥

[root@vms200 certs]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server kubelet-csr.json | cfssl-json -bare kubelet
2020/07/18 19:14:41 [INFO] generate received request
2020/07/18 19:14:41 [INFO] received CSR
2020/07/18 19:14:41 [INFO] generating key: rsa-2048
2020/07/18 19:14:41 [INFO] encoded CSR
2020/07/18 19:14:41 [INFO] signed certificate with serial number 308150770320539429254426692017160372666438070824
2020/07/18 19:14:41 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

检查生成的证书、私钥

[root@vms200 certs]# ls kubelet* -l
-rw-r--r-- 1 root root 1098 Jul 18 19:14 kubelet.csr
-rw-r--r-- 1 root root  446 Jul 18 19:09 kubelet-csr.json
-rw------- 1 root root 1675 Jul 18 19:14 kubelet-key.pem
-rw-r--r-- 1 root root 1448 Jul 18 19:14 kubelet.pem