二进制模式安装部署k8s - 14. CNI网络插件 - 《运维》

安装flannel

k8s设计了网络模型，但却将他的实现交给了网络插件，CNI网络插件最主要的功能就是实现POD资源能够跨主机进行通信
种类众多，以**flannel**为例
三种常用工作模式
优化SNAT规则

常见的CNI网络插件:

Flannel
Calico
Canal
Contiv
OpenContrail
NSX-T
Kube-router

安装flannel

需要在所有运算节点上安装(k8s-5-138,k8s-5-139) 关于flannel的详细说明,可参考以下文章 https://blog.csdn.net/xxb249/article/details/85642172

注意: 由于 flannel 还不支持etcd 3, 所以先要将etcd开启v2,在etcd启动脚本中添加参数 —enable-v2

[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd
Conflicts=etcd.service
Conflicts=etcd2.service
[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
ExecStart=/opt/etcd/etcd --name etcd139 \
  --data-dir /data/etcd-server \
  --listen-client-urls https://192.168.5.139:2379,http://127.0.0.1:2379 \
  --advertise-client-urls https://192.168.5.139:2379,http://127.0.0.1:2379 \
  --listen-peer-urls https://192.168.5.139:2380 \
  --initial-advertise-peer-urls https://192.168.5.139:2380 \
  --initial-cluster etcd137=https://192.168.5.137:2380,etcd138=https://192.168.5.138:2380,etcd139=https://192.168.5.139:2380 \
  --initial-cluster-token tkn \
  --initial-cluster-state new \
  --client-cert-auth \
  --trusted-ca-file /opt/etcd/certs/ca.pem \
  --cert-file /opt/etcd/certs/etcd-peer.pem \
  --key-file /opt/etcd/certs/etcd-peer-key.pem \
  --peer-client-cert-auth \
  --peer-trusted-ca-file /opt/etcd/certs/ca.pem \
  --peer-cert-file /opt/etcd/certs/etcd-peer.pem \
  --peer-key-file /opt/etcd/certs/etcd-peer-key.pem \
  --enable-v2

将flannel配置写到etcd时，需要使用ETCD_API=2 声明版本变量

[root@k8s-5-137 ~]# ETCDCTL_API=2 /opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.5.0.0/16", "Backend": {"Type": "host-gw"}}'
# 用v2命令添加的key-value,使用v3命令get不到,必须使用v2命令才能查到
[root@k8s-5-137 flannel]# /opt/etcd/etcdctl get /coreos.com/network/config
[root@k8s-5-137 flannel]# ETCDCTL_API=2 /opt/etcd/etcdctl get /coreos.com/network/config
{"Network": "172.5.0.0/16", "Backend": {"Type": "host-gw"}}

1. 下载二进制安装包

https://github.com/flannel-io/flannel/releases/tag/v0.12.0

2. 解压,配置证书

因为flannel需要使用etcd来存储一些配置,所以需要配置etcd的客户端证书

[root@k8s-5-138 ~]# cd /opt/src
[root@k8s-5-138 src]# ll
total 381636
-rw-r--r-- 1 root root  17280028 Mar 18 16:28 etcd-v3.4.3-linux-amd64.tar.gz
-rw-r--r-- 1 root root   9565406 Mar 29 10:06 flannel-v0.12.0-linux-amd64.tar.gz
-rw-r--r-- 1 root root 363943527 Mar 24 10:37 kubernetes-server-linux-amd64.tar.gz
[root@k8s-5-138 src]# mkdir /opt/flanel-v0.12.0
[root@k8s-5-138 src]# tar xf flannel-v0.12.0-linux-amd64.tar.gz -C /opt/flanel-v0.12.0/
[root@k8s-5-138 src]# ln -s /opt/flanel-v0.12.0/ /opt/flanel
[root@k8s-5-138 flannel]# mkdir cert
[root@k8s-5-138 flannel]# cd cert
[root@k8s-5-138 cert]# scp k8s-5-141.host.com:/opt/certs/ca.pem .
[root@k8s-5-138 cert]# scp k8s-5-141.host.com:/opt/certs/kube-cert/client.pem .
[root@k8s-5-138 cert]# scp k8s-5-141.host.com:/opt/certs/kube-cert/client-key.pem .
[root@k8s-5-139 cert]# ll
total 12
-rw-r--r-- 1 root root 1387 Mar 29 11:05 ca.pem
-rw------- 1 root root 1679 Mar 29 11:05 client-key.pem
-rw-r--r-- 1 root root 1407 Mar 29 11:05 client.pem

3. 创建配置文件,添加启动服务

启动参数中的—public-ip 需要修改成当前服务器的内网IP

#创建配置文件
[root@k8s-5-138 flannel]# vim subnet.env
FLANNEL_NETWORK=172.5.0.0/16
FLANNEL_SUBNET=172.5.138.1/24
FLANNEL_MTU=1500
FLANNEL_IPMASQ=false
#添加启动服务
[root@k8s-5-138 flannel]# vim /etc/systemd/system/flanneld-app.service
[Unit]
Description=flanneld service
Documentation=https://github.com/kubernetes
Conflicts=flanneld-app
[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
ExecStart=/opt/flannel/flanneld \
    --public-ip=192.168.5.138 \
      --etcd-endpoints=https://192.168.5.137:2379,https://192.168.5.139:2379,https://192.168.5.139:2379 \
      --etcd-keyfile=/opt/flannel/cert/client-key.pem \
      --etcd-certfile=/opt/flannel/cert/client.pem \
      --etcd-cafile=/opt/flannel/cert/ca.pem \
      --iface=ens33 \
      --subnet-file=/opt/flannel/subnet.env \
      --healthz-port=2401 
[Install]
WantedBy=multi-user.target

4. 添加etcd配置(在etcd主节点服务器上执行)

# 先找出etcd leader 服务器
[root@k8s-5-138 etcd]# /opt/etcd/etcdctl \
>   --endpoints https://192.168.5.137:2379,https://192.168.5.138:2379,https://192.168.5.139:2379 \
>   --cacert /opt/etcd/certs/ca.pem \
>   --cert /opt/etcd/certs/etcd-peer.pem \
>   --key /opt/etcd/certs/etcd-peer-key.pem \
>   endpoint status
https://192.168.5.137:2379, 8538572bbe136754, 3.4.3, 2.6 MB, true, false, 168690, 788172, 788172, 
https://192.168.5.138:2379, e66eb4f1843fc910, 3.4.3, 2.6 MB, false, false, 168690, 788172, 788172, 
https://192.168.5.139:2379, 2fb7bc8fa69bd29b, 3.4.3, 2.5 MB, false, false, 168690, 788172, 788172, 
# 切换到137服务器
[root@k8s-5-137 ~]# ETCDCTL_API=2 /opt/etcd/etcdctl set /coreos.com/network/config '{"Network": "172.5.0.0/16", "Backend": {"Type": "host-gw"}}'
# 用v2命令添加的key-value,使用v3命令get不到,必须使用v2命令才能查到
[root@k8s-5-137 flannel]# /opt/etcd/etcdctl get /coreos.com/network/config
[root@k8s-5-137 flannel]# ETCDCTL_API=2 /opt/etcd/etcdctl get /coreos.com/network/config
{"Network": "172.5.0.0/16", "Backend": {"Type": "host-gw"}}

5. 启动falnnel服务

[root@k8s-5-138 ~]# systemctl daemon-reload
[root@k8s-5-138 ~]# systemctl enable flannel-app
[root@k8s-5-138 ~]# systemctl start flannel-app

6. 验证容器网络是否互通

#先查看当前集群内有哪些Pod
[root@k8s-5-138 /]# kubectl get pods -o wide
NAME             READY   STATUS    RESTARTS   AGE    IP            NODE                 NOMINATED NODE   READINESS GATES
nginx-ds-2v724   1/1     Running   0          3d4h   172.5.138.2   k8s-5-138.host.com   <none>           <none>
nginx-ds-85l6c   1/1     Running   0          3d4h   172.5.139.2   k8s-5-139.host.com   <none>           <none>
#可以看到当前集群内有两个pod,分别运行在 k8s-5-138 和 k8s-5-139 节点上,如果在任何一个k8s节点服务器都可以ping通非当前节点下的pod的ip,就表示flannel服务配置成功了
[root@k8s-5-138 /]# ping 172.5.138.2
PING 172.5.138.2 (172.5.138.2) 56(84) bytes of data.
64 bytes from 172.5.138.2: icmp_seq=1 ttl=64 time=0.100 ms
64 bytes from 172.5.138.2: icmp_seq=2 ttl=64 time=0.049 ms
^C
--- 172.5.138.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.049/0.074/0.100/0.026 ms
[root@k8s-5-138 /]# ping 172.5.139.2
PING 172.5.139.2 (172.5.139.2) 56(84) bytes of data.
64 bytes from 172.5.139.2: icmp_seq=1 ttl=63 time=0.307 ms
64 bytes from 172.5.139.2: icmp_seq=2 ttl=63 time=0.333 ms
^C
--- 172.5.139.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.307/0.320/0.333/0.013 ms
[root@k8s-5-138 /]# curl 172.5.138.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[root@k8s-5-138 /]# curl 172.5.139.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>