生产环境部署集群参考
https://github.com/kubernetes-sigs/kubespray
https://github.com/easzlab/kubeasz
https://github.com/kubernetes/kubernetes/releases

集群各节点

环境准备

OS IP hostname 配置
centos7 172.16.10.144 master 2C4G
centos7 172.16.10.196 work1 2C4G
centos7 172.16.10.198 work2 2C4G

系统设置
关闭各服务器iptables，selinux，swap，设置时钟同步(最好配置crontab任务)

  1. # sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config 
  2. # reboot
  3. # systemctl stop firewalld.service
  4. # systemctl disable firewalld.service
  5. # swapoff -a
  6. # yum -y install ntpdate 
  7. # /usr/sbin/ntpdate ntp1.aliyun.com 
  8. # timedatectl set-timezone Asia/Shanghai

设置系统参数 - 允许路由转发，不对bridge的数据进行处理

  1. # cat >/etc/sysctl.d/k8s.conf <<EOF
  2. net.ipv4.ip_forward = 1
  3. net.bridge.bridge-nf-call-ip6tables = 1
  4. net.bridge.bridge-nf-call-iptables = 1
  5. EOF
  6. # sysctl -p /etc/sysctl.d/k8s.conf

若报错

sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory 解决方法 modprobe br_netfilter

配置hosts

# cat /etc/hosts
172.16.10.144  master
172.16.10.196  work1
172.16.10.198  work2

安装修改docker服务文件并启动docker

# yum install -y yum-utils device-mapper-persistent-data lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# yum install -y docker-ce docker-ce-cli containerd.io
# systemctl daemon-reload
# service docker restart

添加yum源

# cat >/etc/yum.repos.d/k8s.repo <<EOF
[k8s]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
EOF

安装apiserver的客户端命令

# yum install -y kubelet kubeadm kubectl
# systemctl enable docker kubelet

master节点

修改kubelet 的配置文件
由于k8s默认要求关闭系统的swap，可以在/etc/sysconfig/kubelet中添加参数”—fail-swap-on=false”以及在后面初始化命令中添加参数—ignore-preflight-errors=’Swap’解决

[root@master ~]# cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS="--fail-swap-on=false"    #取消关闭Swap的限制

查看依赖镜像

[root@master ~]# kubeadm config images list
I0605 11:21:24.077235   11461 version.go:96] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I0605 11:21:24.077480   11461 version.go:97] falling back to the local client version: v1.14.2
k8s.gcr.io/kube-apiserver:v1.14.2
k8s.gcr.io/kube-controller-manager:v1.14.2
k8s.gcr.io/kube-scheduler:v1.14.2
k8s.gcr.io/kube-proxy:v1.14.2
k8s.gcr.io/pause:3.1
k8s.gcr.io/etcd:3.3.10
k8s.gcr.io/coredns:1.3.1

同单机部署版一样，由于无法跨越阻碍pull k8s.gcr.io下的镜像，所以配置阿里云加速器

mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
    "registry-mirrors": ["https://xkl7mn17.mirror.aliyuncs.com"]
}
EOF

pull镜像到本地后重新tag

[root@master ~]# cat imagelist
kube-apiserver:v1.14.2
kube-controller-manager:v1.14.2
kube-scheduler:v1.14.2
kube-proxy:v1.14.2
pause:3.1
etcd:3.3.10
coredns:1.3.1
[root@master ~]# cat PImage.sh
for i in `cat imagelist`; do
  docker pull registry.aliyuncs.com/google_containers/$i
  docker tag registry.aliyuncs.com/google_containers/$i k8s.gcr.io/$i
  docker rmi registry.aliyuncs.com/google_containers/$i
done;

初始化kubeadmin生成node token
PodNetwork采用 Flannel，默认设置的网段为 10.244.0.0./16，在 init 命令中使用 —pod-network-cidr 指定
init的配置参数可参考https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file

[root@master ~]# kubeadm init --kubernetes-version v1.14.2 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
[init] Using Kubernetes version: v1.14.2
[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.10.144]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [172.16.10.144 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [172.16.10.144 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 17.503113 seconds
[upload-config] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.14" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --experimental-upload-certs
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: areu2c.yhax00c02pfjuxpr
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.10.144:6443 --token areu2c.yhax00c02pfjuxpr \
    --discovery-token-ca-cert-hash sha256:4d4fdc401de2283767e767f2bd8c3c97562aebc835d570b5c05931ccbdfe1e49

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

网络配置
https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/
https://github.com/coreos/flannel

配置flannel
[root@master ~]# kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml
podsecuritypolicy.extensions/psp.flannel.unprivileged created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created
查看node状态
[root@master ~]#  kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   21m     v1.14.2
查看组件状态
[root@master ~]#  kubectl get componentstatus
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}

完成后查看pod运行状态

[root@master opt]# kubectl get pods -n kube-system
NAME                             READY   STATUS    RESTARTS   AGE
coredns-fb8b8dccf-fg4z6          1/1     Running   0          19m
coredns-fb8b8dccf-zr5xx          1/1     Running   0          19m
etcd-master                      1/1     Running   0          18m
kube-apiserver-master            1/1     Running   0          18m
kube-controller-manager-master   1/1     Running   0          18m
kube-flannel-ds-amd64-gwrqt      1/1     Running   0          87s
kube-flannel-ds-amd64-q5qwj      1/1     Running   0          92s
kube-flannel-ds-amd64-x5q67      1/1     Running   0          17m
kube-proxy-4xv8d                 1/1     Running   0          19m
kube-proxy-br8vr                 1/1     Running   0          87s
kube-proxy-dq4nj                 1/1     Running   0          92s
kube-scheduler-master            1/1     Running   0          18m

work节点

禁用swap

# cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

pull镜像重新tag

# docker pull mirrorgooglecontainers/pause:3.1
# docker pull mirrorgooglecontainers/kube-proxy:v1.14.2
# docker tag mirrorgooglecontainers/pause:3.1 k8s.gcr.io/pause:3.1
# docker tag mirrorgooglecontainers/kube-proxy:v1.14.2 k8s.gcr.io/kube-proxy:v1.14.2

加入到集群

# kubeadm join 172.16.10.144:6443 --token areu2c.yhax00c02pfjuxpr \
>     --discovery-token-ca-cert-hash sha256:4d4fdc401de2283767e767f2bd8c3c97562aebc835d570b5c05931ccbdfe1e49

成功后可查看到各节点状态为ready


[root@master ~]#  kubectl get nodes
NAME     STATUS   ROLES    AGE     VERSION
master   Ready    master   21m     v1.14.2
work1    Ready    <none>   2m35s   v1.14.2
work2    Ready    <none>   2m39s   v1.14.2

TIPS kubectl reset iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X ipvsadm -C https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-reset/