6、安装主控节点服务-etcd集群(12为lead，21、22为follow)

1、在10.4.7.200上创建基于证书的config配置文件
- 1、cd /opt/certs
- 2、vi ca-config.jon
2、在10.4.7.200创建生成自签证书签名请求(csr)的json配置文件
3、生成证书
4、安装配置etcd (10.4.7.12, 10.4.7.21, 10.4.7.22)

1、在10.4.7.200上创建基于证书的config配置文件

1、cd /opt/certs

2、vi ca-config.jon

输入：
{
“signing”:{
“default”:{
“expiry”:”175200h”
},
“profiles”:{
“server”:{
“expiry”:”175200h”,
“usages”:[
“signing”,
“key encipherment”,
“server auth”
]
},
“client”:{
“expiry”:”175200h”,
“usages”:[
“signing”,
“key encipherment”,
“client auth”
]
},
“peer”:{
“expiry”:”175200h”,
“usages”:[
“signing”,
“key encipherment”,
“server auth”,
“client auth”
]
}
}
}
}

client certificate：客户端使用，用于服务端认证客户端，例如etcd、etcd proxy、fleectl、docker客户端
server certificate：服务端使用，客户端以此验证服务端身份，例如docker服务端、kube-apiserver
peer certificate：双向证书，用于etcd集群成员间通行

2、在10.4.7.200创建生成自签证书签名请求(csr)的json配置文件

vi etcd-peer-csr.json
{
“CN”:”k8s-etcd”,
“hosts”:[
“10.4.7.11”,
“10.4.7.12”,
“10.4.7.21”,
“10.4.7.22”
],
“key”:{
“algo”:”rsa”,
“size”:2048
},
“names”:[
{
“C”:”CN”,
“ST”:”zhejiang”,
“L”:”hangzhou”,
“O”:”jack”,
“OU”:”ops”
}
]
}

3、生成证书

执行命令 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json

生成etcd证书和私钥并保存
执行命令 cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer etcd-peer-csr.json | cfssl-json -bare etcd-peer

4、安装配置etcd (10.4.7.12, 10.4.7.21, 10.4.7.22)

1、10.4.7.12上创建etcd用户

执行：useradd -s /sbin/nologin -M etcd
查看用户：id etcd
输出：uid=1001(etcd) gid=1001(etcd) groups=1001(etcd)

2、10.4.7.12下载软件，解压，做软连接

cd opt
mkdir src

在下载到opt/src目录：
wget https://github.com/etcd-io/etcd/releases/download/v3.2.28/etcd-v3.2.28-linux-amd64.tar.gz

解压到opt目录下：
tar -zxf etcd-v3.2.28-linux-amd64.tar.gz -C /opt

重命名，在src下：
mv etcd-v3.2.28-linux-amd64/ etcd-v3.2.28

做软链接：
ln -s /opt/etcd-v3.2.28/ /opt/etcd

3、10.4.7.12创建目录，拷贝证书、私钥

mkdir -p /opt/etcd/certs /data/etcd /data/logs/etcd-server

创建目录：
/opt/etcd/certs
/data/etcd
/data/logs/etcd-server
将运维主机上生成的ca.pem、etcd-peer-key.pem、etcd-peer.pem拷贝到/opt/etcd/certs目录中，注意私钥文件修改权限为600

scp hdss7-200:/opt/certs/ca.pem .
scp hdss7-200:/opt/certs/etcd-peer.pem .
scp hdss7-200:/opt/certs/etcd-peer-key.pem .

[root@hdss7-12 certs]# ll
total 12
-rw-r—r— 1 root root 1338 Jul 11 17:20 ca.pem
-rw———- 1 root root 1675 Jul 11 17:21 etcd-peer-key.pem
-rw-r—r— 1 root root 1428 Jul 11 17:20 etcd-peer.pem

cd ..到etcd目录

4、创建etcd启动文件

在10.4.7.12上

vi opt/etcd/etcd-server-startup.sh
输入：
#!/bin/sh
./etcd —name etcd-server-7-12 \
—data-dir /data/etcd/etcd-server \
—listen-peer-urls https://10.4.7.12:2380 \
—listen-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
—quota-backend-bytes 8000000000 \
—initial-advertise-peer-urls https://10.4.7.12:2380 \
—advertise-client-urls https://10.4.7.12:2379,http://127.0.0.1:2379 \
—initial-cluster etcd-server-7-12=https://10.4.7.12:2380,etcd-server-7-21=https://10.4.7.21:2380,etcd-server-7-22=https://10.4.7.22:2380 \
—ca-file ./certs/ca.pem \
—cert-file ./certs/etcd-peer.pem \
—key-file ./certs/etcd-peer-key.pem \
—client-cert-auth \
—trusted-ca-file ./certs/ca.pem \
—peer-ca-file ./certs/ca.pem \
—peer-cert-file ./certs/etcd-peer.pem \
—peer-key-file ./certs/etcd-peer-key.pem \
—peer-client-cert-auth \
—peer-trusted-ca-file ./certs/ca.pem \
—log-output stdout

解析：内部通信->https://10.4.7.12:2380 外部通信->https://10.4.7.12:2379

更改权限：chmod +x etcd-server-startup.sh

更改文件所属：
chown -R etcd.etcd /opt/etcd-v3.2.28/
chown -R etcd.etcd /data/etcd/
chown -R etcd.etcd /data/logs/etcd-server/

安装管理后台进程软件(当etcd挂掉后，拉起服务)：yum install supervisor

在etcd目录下
systemctl start supervisord && systemctl enable supervisord

创建supervisord启动文件：vi /etc/supervisord.d/etcd-server.ini
输入：
[program:etcd-server-7-12]
command=/opt/etcd/etcd-server-startup.sh ; the program (relative uses PATH, can take args)
numprocs=1 ; number of processes copies to start (def 1)
directory=/opt/etcd ; directory to cwd to before exec (def no cwd)
autostart=true ; start at supervisord start (default: true)
autorestart=true ; retstart at unexpected quit (default: true)
startsecs=3 ; number of secs prog must stay running (def. 1)
startretries=3 ; max # of serial start failures (default 3)
exitcodes=0,2 ; ‘expected’ exit codes for process (default 0,2)
stopsignal=QUIT ; signal used to kill process (default TERM)
stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
user=etcd ; setuid to this UNIX account to run the program
redirect_stderr=true ; redirect proc stderr to stdout (default false)
stdout_logfile=/data/logs/etcd-server/etcd.stdout.log ; stdout log path, NONE for none; default AUTO
stdout_logfile_maxbytes=64MB ; max # logfile bytes b4 rotation (default 50MB)
stdout_logfile_backups=4 ; # of stdout logfile backups (default 10)
stdout_capture_maxbytes=1MB ; number of bytes in ‘capturemode’ (default 0)
stdout_events_enabled=false ; emit events on stdout writes (default false)
killasgroup=true
stopasgroup=true

执行：supervisorctl update
etcd-server-7-12: added process group

查看状态：supervisorctl status
etcd-server-7-21 RUNNING pid 69535, uptime 0:00:16

查看日志：tail -fn 200 /data/logs/etcd-server/etcd.stdout.log

执行：netstat -luntp | grep etcd
输出：
tcp 0 0 10.4.7.21:2379 0.0.0.0: LISTEN 69536/./etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0: LISTEN 69536/./etcd
tcp 0 0 10.4.7.21:2380 0.0.0.0:* LISTEN 69536/./etcd

其他主机etcd-server-startup.sh略有不同:

三台主机创建完毕后查看健康状态(任意一个节点上，opt/etcd目录下)：./etcdctl cluster-health 或 ./etcdctl member list

5、搭建etcdkeeper查看工具(10.4.7.21上)

1、cd /opt/src

2、wget https://github.com/evildecay/etcdkeeper/releases/download/v0.7.5/etcdkeeper-v0.7.5-linux_x86_64.zip

3、解开压缩包，需安装unzip： yum install unzip -y

4、unzip etcdkeeper-*-linux_x86_64.zip

5、rm etcdkeeper-*-linux_x86_64.zip

6、mv etcdkeeper ../etcdkeeper-0.7.5

7、ln -s /opt/etcdkeeper-0.7.5/ /opt/etcdkeeper

8、cd /opt/etcdkeeper

9、chmod +x etcdkeeper

编写一个服务文件

该服务文件主要用于在后台运行etcd程序,用以提供http服务
cd /lib/systemd/system
vim etcdkeeper.service
输入：
[Unit]
Description=etcdkeeper service
After=network.target
[Service]
Type=simple
ExecStart=/opt/etcdkeeper/etcdkeeper -h 10.4.7.21 -p 8800
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
PrivateTmp=true
[Install]
WantedBy=multi-user.target

-h 指定etcdkeeper http监听的地址,这里监听的是IPV4地址10.4.7.12
-p 指定etcdkeeper http监听的端口

systemctl start etcdkeeper 启动etcdkeeper服务
systemctl stop etcdkeeper 停止etcdkeeper服务
systemctl enable etcdkeeper.service 设置开机自启动
systemctl disable etcdkeeper.service 停止开机自启动

kubernetes