简介

Flannel是一种基于overlay网络的跨主机容器网络解决方案，也就是将TCP数据包封装在另一种网络包里面进行路由转发和通信，

Flannel是CoreOS开发,专门用于docker多机互联的一个工具,让集群中的不同节点主机创建的容器都具有全集群唯一的虚拟ip地址

Flannel使用go语言编写

Flannel实现原理

2.1、原理说明

Flannel为每个host分配一个subnet，容器从这个subnet中分配IP，这些IP可以在host间路由，容器间无需使用nat和端口映射即可实现跨主机通信
每个subnet都是从一个更大的IP池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，其职责就是从池子中分配subnet
Flannel使用etcd存放网络配置、已分配 的subnet、host的IP等信息
Flannel数据包在主机间转发是由backend实现的，目前已经支持UDP、VxLAN、host-gw、AWS VPC和GCE路由等多种backend

2.2、数据转发流程

Docker网络（etcd-flannel) - 图1

  1. 容器直接使用目标容器的ip访问，默认通过容器内部的eth0发送出去。

  2. 报文通过veth pair被发送到vethXXX。

  3. vethXXX是直接连接到虚拟交换机docker0的，报文通过虚拟bridge docker0发送出去。

  4. 查找路由表，外部容器ip的报文都会转发到flannel0虚拟网卡，这是一个P2P的虚拟网卡，然后报文就被转发到监听在另一端的flanneld。

  5. flanneld通过etcd维护了各个节点之间的路由表，把原来的报文UDP封装一层，通过配置的iface发送出去。

  6. 报文通过主机之间的网络找到目标主机。

  7. 报文继续往上，到传输层，交给监听在8285端口的flanneld程序处理。

  8. 数据被解包，然后发送给flannel0虚拟网卡。

  9. 查找路由表，发现对应容器的报文要交给docker0。

  10. docker0找到连到自己的容器，把报文发送过去。

Flannel安装配置

3.1、环境准备

节点名称 IP地址 安装软件
c7docker1 192.168.163.130 etcd、flannel、docker
c7docker2 192.168.163.131 flannel、docker

3.2、安装etcd

etcd下载地址：https://github.com/coreos/etcd/releases

解压
[root@c7docker1 ~]# tar zxvf etcd-v3.3.10-linux-amd64.tar.gz

[root@c7docker1 ~]# cd etcd-v3.3.10-linux-amd64/

[root@c7docker1 ~]# mv etcd etcdctl /usr/local/bin/

启动
[root@c7docker1 ~]# etcd —name etcd1 —data-dir /var/lib/etcd —advertise-client-urls http://192.168.163.130:2379,http://127.0.0.1:2379 —listen-client-urls http://192.168.163.130:2379,http://127.0.0.1:2379 &
[1] 2390
[root@c7docker1 ~]# 2018-12-05 16:23:54.716751 I | etcdmain: etcd Version: 3.3.10
2018-12-05 16:23:54.716861 I | etcdmain: Git SHA: 27fc7e2
2018-12-05 16:23:54.716867 I | etcdmain: Go Version: go1.10.4
2018-12-05 16:23:54.716874 I | etcdmain: Go OS/Arch: linux/amd64
2018-12-05 16:23:54.716881 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2
2018-12-05 16:23:54.718677 I | embed: listening for peers on http://localhost:2380
2018-12-05 16:23:54.718766 I | embed: listening for client requests on 127.0.0.1:2379
2018-12-05 16:23:54.718799 I | embed: listening for client requests on 192.168.163.130:2379
2018-12-05 16:23:54.737768 I | etcdserver: name = etcd1
2018-12-05 16:23:54.738572 I | etcdserver: data dir = /var/lib/etcd
2018-12-05 16:23:54.738675 I | etcdserver: member dir = /var/lib/etcd/member
2018-12-05 16:23:54.738684 I | etcdserver: heartbeat = 100ms
2018-12-05 16:23:54.738690 I | etcdserver: election = 1000ms
2018-12-05 16:23:54.738697 I | etcdserver: snapshot count = 100000
2018-12-05 16:23:54.738712 I | etcdserver: advertise client URLs = http://127.0.0.1:2379,http://192.168.163.130:2379
2018-12-05 16:23:54.738720 I | etcdserver: initial advertise peer URLs = http://localhost:2380
2018-12-05 16:23:54.738733 I | etcdserver: initial cluster = etcd1=http://localhost:2380
2018-12-05 16:23:54.752024 I | etcdserver: starting member 8e9e05c52164694d in cluster cdf818194e3a8c32
2018-12-05 16:23:54.752081 I | raft: 8e9e05c52164694d became follower at term 0
2018-12-05 16:23:54.752101 I | raft: newRaft 8e9e05c52164694d [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2018-12-05 16:23:54.752110 I | raft: 8e9e05c52164694d became follower at term 1
2018-12-05 16:23:54.769932 W | auth: simple token is not cryptographically signed
2018-12-05 16:23:54.783666 I | etcdserver: starting server… [version: 3.3.10, cluster version: to_be_decided]
2018-12-05 16:23:54.785651 I | etcdserver: 8e9e05c52164694d as single-node; fast-forwarding 9 ticks (election ticks 10)
2018-12-05 16:23:54.786483 I | etcdserver/membership: added member 8e9e05c52164694d [http://localhost:2380] to cluster cdf818194e3a8c32
2018-12-05 16:23:55.554120 I | raft: 8e9e05c52164694d is starting a new election at term 1
2018-12-05 16:23:55.554153 I | raft: 8e9e05c52164694d became candidate at term 2
2018-12-05 16:23:55.554173 I | raft: 8e9e05c52164694d received MsgVoteResp from 8e9e05c52164694d at term 2
2018-12-05 16:23:55.554185 I | raft: 8e9e05c52164694d became leader at term 2
2018-12-05 16:23:55.554192 I | raft: raft.node: 8e9e05c52164694d elected leader 8e9e05c52164694d at term 2
2018-12-05 16:23:55.554798 I | etcdserver: published {Name:etcd1 ClientURLs:[http://127.0.0.1:2379 http://192.168.163.130:2379]} to cluster cdf818194e3a8c32
2018-12-05 16:23:55.555275 I | embed: ready to serve client requests
2018-12-05 16:23:55.555973 E | etcdmain: forgot to set Type=notify in systemd service file?
2018-12-05 16:23:55.556013 I | embed: ready to serve client requests
2018-12-05 16:23:55.556064 N | embed: serving insecure client requests on 127.0.0.1:2379, this is strongly discouraged!
2018-12-05 16:23:55.557528 I | etcdserver: setting up the initial cluster version to 3.3
2018-12-05 16:23:55.557856 N | embed: serving insecure client requests on 192.168.163.130:2379, this is strongly discouraged!
2018-12-05 16:23:55.561368 N | etcdserver/membership: set the initial cluster version to 3.3
2018-12-05 16:23:55.563071 I | etcdserver/api: enabled capabilities for version 3.3

[root@c7docker1 ~]# ps -aux |grep etc
root 2390 2.4 0.5 10515884 10944 pts/0 Sl 16:23 0:02 etcd —name etcd1 —data-dir /var/lib/etcd —advertise-client-urls http://192.168.163.130:2379,http://127.0.0.1:2379 —listen-client-urls http://192.168.163.130:2379,http://127.0.0.1:2379
root 2401 0.0 0.0 112704 972 pts/0 S+ 16:25 0:00 grep —color=auto etc

3.3、安装Flannel

flannel下载地址：https://github.com/coreos/flannel/releases

解压
[root@c7docker1 ~]# tar zxvf flannel-v0.10.0-linux-amd64.tar.gz
flanneld
mk-docker-opts.sh
README.md

[root@c7docker1 ~]# mv flanneld /usr/local/bin/

[root@c7docker1 ~]# mv mk-docker-opts.sh /usr/local/bin/

添加flannel网络配置信息到etcd：

[root@c7docker1 ~]# etcdctl —endpoints http://127.0.0.1:2379 set /coreos.com/network/config ‘{“Network”: “10.0.0.0/16”,”SubnetLen”: 24, “SubnetMin”: “10.0.1.0”, “SubnetMax”: “10.0.20.0”, “Backend”: {“Type”: “vxlan”}}’

{“Network”: “10.0.0.0/16”,”SubnetLen”: 24, “SubnetMin”: “10.0.1.0”, “SubnetMax”: “10.0.20.0”, “Backend”: {“Type”: “vxlan”}}

3.4、启动Flannel

定制flannel服务启动脚本
[root@c7docker1 ~]# cat -v /etc/systemd/system/flanneld.service
[Unit]
Description=Flanneld
Documentation=https://github.com/coreos/flannel
After=network.target
Before=docker.service

[Service]
User=root
ExecStartPost=/usr/local/bin/mk-docker-opts.sh
ExecStart=/usr/local/bin/flanneld \
—etcd-endpoints=http://192.168.163.130:2379 \
—iface=192.168.163.130 \
—ip-masq=true \
—etcd-prefix=/coreos.com/network
Restart=on-failure
Type=notify
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

启动服务
[root@c7docker1 ~]# systemctl daemon-reload

[root@c7docker1 ~]# systemctl start flanneld.service
[root@c7docker1 ~]# systemctl status flanneld.service
● flanneld.service - Flanneld
Loaded: loaded (/etc/systemd/system/flanneld.service; disabled; vendor preset: disabled)
Active: active (running) since Wed 2018-12-05 17:18:47 CST; 7s ago
Docs: https://github.com/coreos/flannel
Process: 2796 ExecStartPost=/usr/local/bin/mk-docker-opts.sh (code=exited, status=0/SUCCESS)
Main PID: 2788 (flanneld)
Tasks: 8
Memory: 6.4M
CGroup: /system.slice/flanneld.service
└─2788 /usr/local/bin/flanneld —etcd-endpoints=http://192.168.163.130:2379 —iface=192.168.163.130 —ip-masq=true —etcd-prefix=/coreos.com/network

Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.484799 2788 main.go:235] Created subnet manager: Etcd Local Manager with Previous Subnet: 10.0.3.0/24
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.484806 2788 main.go:238] Installing signal handlers
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.487986 2788 main.go:353] Found network config - Backend type: vxlan
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.488247 2788 vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.492431 2788 local_manager.go:147] Found lease (10.0.3.0/24) for current IP (192.168.163.130), reusing
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.497122 2788 main.go:300] Wrote subnet file to /run/flannel/subnet.env
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.497149 2788 main.go:304] Running backend.
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.497819 2788 vxlan_network.go:60] watching for new subnet leases
Dec 05 17:18:47 c7docker1 flanneld[2788]: I1205 17:18:47.502403 2788 main.go:396] Waiting for 22h59m59.991376578s to renew lease
Dec 05 17:18:47 c7docker1 systemd[1]: Started Flanneld.

Flannel启动过程解析：
•从etcd中获取network的配置信息
•划分subnet，并在etcd中进行注册
•将子网信息记录到/run/flannel/subnet.env中

3.5、验证Flannel网络

查看etcd中的数据
[root@c7docker1 ~]# etcdctl ls /coreos.com/network/subnets/
/coreos.com/network/subnets/10.0.3.0-24

查看flannel网卡信息
[root@c7docker1 ~]# ifconfig flannel.1
flannel.1: flags=4163 mtu 1450
inet 10.0.3.0 netmask 255.255.255.255 broadcast 0.0.0.0
inet6 fe80::88fa:c7ff:feba:5c9 prefixlen 64 scopeid 0x20
ether 8a:fa:c7:ba:05:c9 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0

3.6、配置Docker

查看flannel生成的环境变量
[root@c7docker1 ~]# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.0.0.0/16
FLANNEL_SUBNET=10.0.3.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true.

生成docker服务启动参数
[root@c7docker1 ~]# mk-docker-opts.sh -c

[root@c7docker1 ~]# cat /run/docker_opts.env
DOCKER_OPTS=” —bip=10.0.3.1/24 —ip-masq=false —mtu=1450”

修改docker服务启动脚本

[root@c7docker1 ~]# cat /lib/systemd/system/docker.service |grep -v ^#|grep -v ^$
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
EnviromentFile=/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS -H unix://
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target

重启docker服务
[root@c7docker1 ~]# systemctl daemon-reload
[root@c7docker1 ~]# systemctl restart docker.service

[root@c7docker1 ~]# ip add l
1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens32: mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:60:81:fb brd ff:ff:ff:ff:ff:ff
inet 192.168.163.130/24 brd 192.168.163.255 scope global noprefixroute ens32
valid_lft forever preferred_lft forever
inet6 fe80::cf80:92b6:1c57:71d/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: docker0: mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ab:44:38:1c brd ff:ff:ff:ff:ff:ff
inet 10.0.3.1/24 brd 10.0.3.255 scope global docker0
valid_lft forever preferred_lft forever
4: flannel.1: mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether 8a:fa:c7:ba:05:c9 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::88fa:c7ff:feba:5c9/64 scope link
valid_lft forever preferred_lft forever

c7docker2 上flannel配置和c7docker1上相同

3.7、验证容器互通

[root@c7docker1 ~]# docker run -d busybox sleep 3600
3f3122d5c2b4f38bebf361a633fdabf4030030375a50e2b94d6b515a9871cc89
[root@c7docker1 ~]# docker exec -it 3f /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused “exec: \”/bin/bash\”: stat /bin/bash: no such file or directory”: unknown
[root@c7docker1 ~]# docker exec -it 3f /bin/sh
/ # ip add l
ip: invalid argument ‘l’ to ‘ip’
/ # ip add
1: lo: mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
7: eth0@if8: mtu 1450 qdisc noqueue
link/ether 02:42:0a:00:03:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.2/24 brd 10.0.3.255 scope global eth0
valid_lft forever preferred_lft forever
/ # exit

[root@c7docker2 ~]# docker run -d busybox sleep 3600
8b1c040c2c6be47f8685c38b368a6d94c20f85561544adbb00d20200ed76d78f
[root@c7docker2 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8b1c040c2c6b busybox “sleep 3600” 4 seconds ago Up 3 seconds friendly_jang
[root@c7docker2 ~]# docker exec -it 8b /bin/bash
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused “exec: \”/bin/bash\”: stat /bin/bash: no such file or directory”: unknown
[root@c7docker2 ~]# docker exec -it 8b /bin/sh
/ # ip a
1: lo: mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
7: eth0@if8: mtu 1450 qdisc noqueue
link/ether 02:42:0a:00:0b:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.11.2/24 brd 10.0.11.255 scope global eth0
valid_lft forever preferred_lft forever
/ # exit

[root@c7docker1 ~]# docker exec -it 3f /bin/sh
/ # ping 10.0.11.2
PING 10.0.11.2 (10.0.11.2): 56 data bytes
64 bytes from 10.0.11.2: seq=0 ttl=62 time=6.843 ms
64 bytes from 10.0.11.2: seq=1 ttl=62 time=1.135 ms
^C
—- 10.0.11.2 ping statistics —-
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 1.135/3.989/6.843 ms
/ #

3.8、配置backend为host-gw

host-gw bakcend是flannel的另一个backend。与vxlan不同，host-gw不会封装数据包，而是在主机的路由表中创建到其他主机的subnet的路由条目，从而实现容器网络跨主机通信。需要说明的是，host-gw不能跨宿主机网络通信，或者说跨宿主机网络通信需要物理路由支持。

修改etcd
[root@c7docker1 ~]# etcdctl —endpoints http://127.0.0.1:2379 set /coreos.com/network/config ‘{“Network”: “10.0.0.0/16”,”SubnetLen”: 24, “SubnetMin”: “10.0.1.0”, “SubnetMax”: “10.0.20.0”, “Backend”: {“Type”: “host-gw”}}’
{“Network”: “10.0.0.0/16”,”SubnetLen”: 24, “SubnetMin”: “10.0.1.0”, “SubnetMax”: “10.0.20.0”, “Backend”: {“Type”: “host-gw”}}

重启flanneld docker服务

[root@c7docker1 ~]# systemctl restart flanneld.service docker.service

查看路由

[root@c7docker1 ~]# ip route
default via 192.168.163.2 dev ens32 proto static metric 100
10.0.3.0/24 dev docker0 proto kernel scope link src 10.0.3.1
10.0.11.0/24 via 192.168.163.131 dev ens32
192.168.163.0/24 dev ens32 proto kernel scope link src 192.168.163.130 metric 100

root@c7docker2 ~]# ip route
default via 192.168.163.2 dev ens33 proto static metric 100
10.0.3.0/24 via 192.168.163.130 dev ens33
10.0.11.0/24 dev docker0 proto kernel scope link src 10.0.11.1
192.168.163.0/24 dev ens33 proto kernel scope link src 192.168.163.131 metric 100

[root@c7docker2 ~]# docker run -d busybox sleep 3600
b07eba5ee4185b38a86e7c1624982270af82f0b81edd5f86320d4689d2d1ae4e
[root@c7docker2 ~]# docker exec -it b0 /bin/sh
/ # ip a
1: lo: mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
9: eth0@if10: mtu 1500 qdisc noqueue
link/ether 02:42:0a:00:0b:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.11.2/24 brd 10.0.11.255 scope global eth0
valid_lft forever preferred_lft forever
/ #

[root@c7docker1 ~]# docker run -d busybox sleep 3600
a6b5c5af1f9c9e5d608c63e69c77cbc51e6d4039583dd3c530cfbc970791eac6
[root@c7docker1 ~]# docker exec -it a6 /bin/sh
/ # ip ad
1: lo: mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
9: eth0@if10: mtu 1500 qdisc noqueue
link/ether 02:42:0a:00:03:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.3.2/24 brd 10.0.3.255 scope global eth0
valid_lft forever preferred_lft forever
/ # ping 10.0.11.2
PING 10.0.11.2 (10.0.11.2): 56 data bytes
64 bytes from 10.0.11.2: seq=0 ttl=62 time=4.564 ms
64 bytes from 10.0.11.2: seq=1 ttl=62 time=2.141 ms
^C
—- 10.0.11.2 ping statistics —-
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 2.141/3.352/4.564 ms
/ #