macvlan实现k8s扁平2层网络

http://www.calmkart.com/?p=376
最近有一批旧服务器待用，没有用iaas，打算把docker当vm来用，还想采用k8s的编排方案。这样用户就可以很方便的通过ip直连docker，就像虚拟机一样。
原生的docker网络方案是不行的，于是采用了macvlan网络插件实现了k8s扁平二层网络。下面记录一下实现过程。

首先肯定是装docker了
#卸载旧的docker
yum remove -y docker
#这里我是用自己搭的私人仓库和自定义的docker安装包,各位可酌情
yum install -y docker-ce.x86_64
systemctl start docker
systemctl enable docker
关闭防火墙,关闭swap
systemctl stop firewalld
systemctl disable firewalld
setenforce 0
sed -i ‘s/^SELINUX=enforcing$/SELINUX=permissive/‘ /etc/selinux/config
swapoff -a
配置k8s国内源,设置系统bridge参数
# 添加kubenetes源
cat < /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
#调系统参数
cat < /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl —system
用国内源下载k8s相关镜像，重新打tag成k8s.gcr.io(就算可以翻墙，这一步也可以加速很多)
#docker镜像换国内
cat < ~/get_image.sh
#!/bin/bash
images=(
kube-apiserver:v1.13.1
kube-controller-manager:v1.13.1
kube-scheduler:v1.13.1
kube-proxy:v1.13.1
pause:3.1
etcd:3.2.24
coredns:1.2.6
)

for imageName in ${images[@]} ; do
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName k8s.gcr.io/$imageName
done
EOF

下载镜像
bash ~/get_image.sh
下载cni网络插件包
https://github.com/containernetworking/plugins/releases
当前0.7.4版下载地址是
https://github.com/containernetworking/plugins/releases/download/v0.7.4/cni-plugins-amd64-v0.7.4.tgz
mkdir -p /opt/cni/bin/
cd /opt/cni/bin
wget https://github.com/containernetworking/plugins/releases/download/v0.7.4/cni-plugins-amd64-v0.7.4.tgz
tar xzvf cni-plugins-amd64-v0.7.4.tgz
安装k8s全家桶
yum install -y kubelet kubeadm kubectl —disableexcludes=kubernetes
systemctl enable kubelet && systemctl start kubelet
kubeadm做k8s初始化,记录node用来join的token连接(当然用crt也行)
kubeadm init
最后配置网络插件macvlan
mkdir -p /etc/cni/net.d
vim /etc/cni/net.d/10-macvlan.conf
#用host-local的配法,各配置项很简单不详述
{
“name”: “macvlannet”,
“type”: “macvlan”,
“master”: “br0”,
“ipam”: {
“type”: “host-local”,
“subnet”: “10.52.101.0/24”,
“rangeStart”: “10.52.101.40”,
“rangeEnd”: “10.52.101.250”,
“gateway”: “10.52.101.254”,
“routes”: [
{ “dst”: “0.0.0.0/0” }
]
}
}
#用dhcp的配法
{
“name”: “macvlannet”,
“type”: “macvlan”,
“master”: “br0”,
“ipam”: {
“type”: “dhcp”,
}
}

特别贴士——>在使用DHCP插件之前，需要先启动dhcp daemon
/opt/cni/bin/dhcp daemon &
看下主节点是否ok
#先参考官方文档配下环境变量啥的
#To make kubectl work for your non-root user, run these commands, which are also part of the kubeadm init output:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

export KUBECONFIG=/etc/kubernetes/admin.conf

查看主节点是否ready
kubectl get nodes
#查看各个pods情况
kubectl get pods —all-namespaces
#如果有问题,就用describe自己查查吧,一般都不难解决
kubectl describe pod
让master也可以运行pod
kubectl taint nodes —all node-role.kubernetes.io/master-
然后随便弄个pod试试管用了没
#编辑yaml
vim pod-macvlan.yaml

apiVersion: v1
kind: Pod
metadata:
name: nginx-macvlan
labels:
app: web
annotations:
cni: “macvlan”
spec:
containers:
- name: key-value-store
image: nginx:latest
imagePullPolicy: IfNotPresent
ports:
- containerPort: 6379
#创建pod
kubectl create -f pod-macvlan.yaml
#查看下pod情况
kubectl get pods -o wide
一般情况下是没问题，pod已经和宿主机在同一个2层网络了，同2层的其他网络也可以到该pod直通。
这时候就可以直接把pod当vm用了,和桥接的vm几乎没啥区别。
最后把其他机器作为node加入k8s集群中
重复上面步骤
除了kubeadm init 换成
#举例
kubeadm join 10.52.101.28:6443 —token b7eber.9e26oenyos7j17v8 —discovery-token-ca-cert-hash sha256:9faeb9f6ce6281d2ae149c9424ec34297e6809ce1f88c920281f1e5da8b1b1f8
之外，其他不用变化啥。(确实多pull了几个kubeapi啥的镜像，不过无所谓了，不pull也行)
最终结果
nodes

pods

测试某一个pod

本机直接ip访问pod

ok,搞定,打完收工