系统配置

  1. 机器数量
  2. 3台
  4. 操作系统
  5. Centos8.1---7.3
  7. 设置主机名称
  8. 分别设置主机名称为：
  9. master node1 node2
  11. 每台机器必须设置域名解析
  12. 192.168.81.30 master
  13. 192.168.81.31 node1
  14. 192.168.81.32 node2
  16. 禁用开机启动防火墙
  17. # systemctl disable firewalld
  19. 永久禁用SELinux
  20. # sed -i 's/SELINUX=permissive/SELINUX=disabled/' /etc/sysconfig/selinux
  21. SELINUX=disabled
  23. 关闭系统Swap
  24. 1.8版本之后的新规定
  25. Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。
  26. 修改/etc/fstab文件，注释掉SWAP的自动挂载，使用free -m确认swap已经关闭。
  27. [root@master /]# sed -i 's/.*swap.*/#&/' /etc/fstab

基础软件安装

  1. docker安装
  2. yum install -y yum-utils device-mapper-persistent-data lvm2 
  3. && yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo 
  4. && yum makecache && 
  5. yum -y install docker-ce -y && 
  6. systemctl enable docker.service && systemctl start docker
  8. 所有机器安装kubeadm和kubelet
  9. 配置aliyun的yum源
  10. # cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  11. [kubernetes]
  12. name=Kubernetes
  13. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
  14. enabled=1
  15. gpgcheck=1
  16. repo_gpgcheck=1
  17. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  18. EOF
  20. 安装最新版kubeadm
  21. # yum makecache
  22. # yum install -y kubelet kubeadm kubectl ipvsadm
  24. 由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用
  25. 【yum install -y --nogpgcheck kubelet kubeadm kubectl】
  28. 说明：如果想安装指定版本的kubeadmin【v17.2的无法使用yum默认版本的】
  29. #yum install kubelet-1.16.0-0.x86_64 kubeadm-1.16.0-0.x86_64 kubectl-1.16.0-0.x86_64
  31. 配置内核参数
  32. # cat <<EOF >  /etc/sysctl.d/k8s.conf
  33. net.bridge.bridge-nf-call-ip6tables = 1
  34. net.bridge.bridge-nf-call-iptables = 1
  35. vm.swappiness=0
  36. EOF
  38. # sysctl --system
  39. # modprobe br_netfilter
  40. # sysctl -p /etc/sysctl.d/k8s.conf
  42. 加载ipvs相关内核模块
  43. 如果重新开机，需要重新加载（可以写在 /etc/rc.local 中开机自动加载）
  44. # modprobe ip_vs
  45. # modprobe ip_vs_rr
  46. # modprobe ip_vs_wrr
  47. # modprobe ip_vs_sh
  48. # modprobe nf_conntrack_ipv4
  50. 查看是否加载成功
  51. # lsmod | grep ip_vs

image.png

获取镜像

  1. 三个节点都要下载
  2. - 注意下载时把版本号修改到官方最新版，即使下载了最新版也可能版本不对应，需要按报错提示下载
  3. -     每次部署都会有版本更新，具体版本要求，运行初始化过程失败会有版本提示
  4. -     kubeadm的版本和镜像的版本必须是对应的
  5.   用命令查看版本当前kubeadm对应的k8s镜像版本
  6. 【最新的版本 [root@master ~]#  kubeadm config images list】
  7. [root@master ~]# kubeadm config images list
  8. k8s.gcr.io/kube-apiserver:v1.17.2
  9. k8s.gcr.io/kube-controller-manager:v1.17.2
  10. k8s.gcr.io/kube-scheduler:v1.17.2
  11. k8s.gcr.io/kube-proxy:v1.17.2
  12. k8s.gcr.io/pause:3.1
  13. k8s.gcr.io/etcd:3.4.3-0
  14. k8s.gcr.io/coredns:1.6.5
  16. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.2
  17. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.17.2 k8s.gcr.io/kube-apiserver:v1.17.2
  19. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.2
  20. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.17.2 k8s.gcr.io/kube-controller-manager:v1.17.2
  22. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.2
  23. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.17.2 k8s.gcr.io/kube-scheduler:v1.17.2
  25. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.2
  26. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.17.2 k8s.gcr.io/kube-proxy:v1.17.2
  28. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1
  29. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
  31. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0
  32. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0
  34. docker pull coredns/coredns:1.6.5
  35. docker tag coredns/coredns:1.6.5 k8s.gcr.io/coredns:1.6.5

image.png

所有节点配置启动kubelet

  1. 配置kubelet使用国内pause镜像
  2. 获取docker的cgroups
  3. DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f4)
  4. # echo $DOCKER_CGROUPS
  5. cgroupfs 1 则
  6. 【使用如下命令】
  7. DOCKER_CGROUPS=$(docker info |grep cgro|awk -F':| ' '{print $NF}')
  8. # echo $DOCKER_CGROUPS
  9. cgroupfs
  10. 配置kubelet的cgroups
  11. [root@master ~]# cat >/etc/sysconfig/kubelet<<EOF
  12. KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=k8s.gcr.io/pause:3.1"
  13. EOF
  15. 启动
  16. # systemctl daemon-reload
  17. # systemctl enable kubelet && systemctl start kubelet

说明

  1. 做完如上操作后kubelet并没启动  是正常的
  2. 简单地说就是在kubeadm init 之前kubelet会不断重启。

初始化集群

  1. 在master节点进行初始化操作【默认yum安装会报这个】
  2. this version of kubeadm only supports deploying 
  3. clusters with the control plane version >= 1.24.0. Current version: v1.17.2
  4. 初始化的时候 根据要求下载对应版本
  5. 补安装
  6. docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.15-0
  7. docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.15-0 k8s.gcr.io/etcd:3.3.15-0
  10. docker pull coredns/coredns:1.6.2
  11. docker tag coredns/coredns:1.6.2 k8s.gcr.io/coredns:1.6.2
  12. 特别说明：
  13. 初始化完成必须要记录下初始化过程最后的命令，如下所示
  14. To start using your cluster, you need to run the following as a regular user:
  16.   mkdir -p $HOME/.kube
  17.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  18.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  21. Then you can join any number of worker nodes by running the following on each as root:
  23. kubeadm join 192.168.81.30:6443 --token dcg8kk.t42rp6f32p2szdda \
  24.     --discovery-token-ca-cert-hash sha256:7b21de9e4be342bd5d971fbc9859169c7606c29d735f057a767ffcdd439bce10 
  26. 其中有以下关键内容：
  27. [kubelet] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”
  28. [certificates]生成相关的各种证书
  29. [kubeconfig]生成相关的kubeconfig文件
  30. [bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到
  32. 在master节点配置使用kubectl
  33. [root@master ~]#  mkdir -p $HOME/.kube
  34. [root@master ~]# cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  35. [root@master ~]#  chown $(id -u):$(id -g) $HOME/.kube/config
  37. 查看node节点
  38. [root@master ~]# kubectl get nodes
  39. NAME     STATUS     ROLES    AGE     VERSION
  40. master   NotReady   master   6m31s   v1.16.0

image.png

初始化过程

  1. [root@master ~]# kubeadm init --kubernetes-version=v1.17.2 --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.81.30 --ignore-preflight-errors=Swap
  2. [init] Using Kubernetes version: v1.17.2
  3. [preflight] Running pre-flight checks
  4.     [WARNING KubernetesVersion]: Kubernetes version is greater than kubeadm version. Please consider to upgrade kubeadm. Kubernetes version: 1.17.2. Kubeadm version: 1.16.x
  5.     [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
  6.     [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.17. Latest validated version: 18.09
  7. [preflight] Pulling images required for setting up a Kubernetes cluster
  8. [preflight] This might take a minute or two, depending on the speed of your internet connection
  9. [preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
  10. [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
  11. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
  12. [kubelet-start] Activating the kubelet service
  13. [certs] Using certificateDir folder "/etc/kubernetes/pki"
  14. [certs] Generating "ca" certificate and key
  15. [certs] Generating "apiserver" certificate and key
  16. [certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.81.30]
  17. [certs] Generating "apiserver-kubelet-client" certificate and key
  18. [certs] Generating "front-proxy-ca" certificate and key
  19. [certs] Generating "front-proxy-client" certificate and key
  20. [certs] Generating "etcd/ca" certificate and key
  21. [certs] Generating "etcd/server" certificate and key
  22. [certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.81.30 127.0.0.1 ::1]
  23. [certs] Generating "etcd/peer" certificate and key
  24. [certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.81.30 127.0.0.1 ::1]
  25. [certs] Generating "etcd/healthcheck-client" certificate and key
  26. [certs] Generating "apiserver-etcd-client" certificate and key
  27. [certs] Generating "sa" key and public key
  28. [kubeconfig] Using kubeconfig folder "/etc/kubernetes"
  29. [kubeconfig] Writing "admin.conf" kubeconfig file
  30. [kubeconfig] Writing "kubelet.conf" kubeconfig file
  31. [kubeconfig] Writing "controller-manager.conf" kubeconfig file
  32. [kubeconfig] Writing "scheduler.conf" kubeconfig file
  33. [control-plane] Using manifest folder "/etc/kubernetes/manifests"
  34. [control-plane] Creating static Pod manifest for "kube-apiserver"
  35. [control-plane] Creating static Pod manifest for "kube-controller-manager"
  36. [control-plane] Creating static Pod manifest for "kube-scheduler"
  37. [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
  38. [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
  39. [apiclient] All control plane components are healthy after 16.502507 seconds
  40. [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
  41. [kubelet] Creating a ConfigMap "kubelet-config-1.17" in namespace kube-system with the configuration for the kubelets in the cluster
  42. [upload-certs] Skipping phase. Please see --upload-certs
  43. [mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
  44. [mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
  45. [bootstrap-token] Using token: dcg8kk.t42rp6f32p2szdda
  46. [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
  47. [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
  48. [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
  49. [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
  50. [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
  51. [addons] Applied essential addon: CoreDNS
  52. [addons] Applied essential addon: kube-proxy
  54. Your Kubernetes control-plane has initialized successfully!
  56. To start using your cluster, you need to run the following as a regular user:
  58.   mkdir -p $HOME/.kube
  59.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  60.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  62. You should now deploy a pod network to the cluster.
  63. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  64.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  66. Then you can join any number of worker nodes by running the following on each as root:
  68. kubeadm join 192.168.81.30:6443 --token dcg8kk.t42rp6f32p2szdda \
  69.     --discovery-token-ca-cert-hash sha256:7b21de9e4be342bd5d971fbc9859169c7606c29d735f057a767ffcdd439bce10

image.png

配置网络插件

  1. master节点下载yaml配置文件
  2. 特别说明：版本会经常更新，如果配置成功，就手动去
  3. https://raw.githubusercontent.com/coreos/flannel/master/Documentation/ 
  4. 下载最新版yaml文件
  5. [root@master ~]# cd ~ && mkdir flannel && cd flannel
  6. [root@master flannel]# curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
  8. 修改配置文件kube-flannel.yml
  9. [root@master flannel]# grep install-cni -A 2 kube-flannel.yml 
  10. [root@master flannel]# docker pull  docker.io/rancher/mirrored-flannelcni-flannel:v0.19.2
  12. 【确保后续没问题下载所有kube-flannel.yml文件的image】
  13. [root@master flannel]# docker pull docker.io/rancher/mirrored-flannelcni-flannel-cni-plugin:v1.1.0
  14. 说明：默认的镜像是quay.io/coreos/flannel:v0.10.0-amd64，
  15. 如果你能pull下来就不用修改镜像地址，否则，修改yml中镜像地址为阿里镜像源,
  16. 要修改所有的镜像版本，里面有好几条flannel镜像地址
  17. image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
  19. 指定启动网卡
  20. [root@master flannel]# cp kube-flannel.yml kube-flannel.yml_init
  22. flanneld启动参数加上--iface=<iface-name>
  24.       containers:
  25.       - name: kube-flannel
  26.         image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
  27.         command:
  28.         - /opt/bin/flanneld
  30.         args:
  31.         - --ip-masq
  32.         - --kube-subnet-mgr
  33.         - --iface=ens33 【增加这一行---网卡名称】
  34.         - --iface=eth0   
  36. ⚠️⚠️⚠️--iface=ens33 的值，是你当前的网卡,或者可以指定多网卡
  38. 启动
  39. [root@master flannel]# kubectl apply -f ~/flannel/kube-flannel.yml
  40. namespace/kube-flannel created
  41. clusterrole.rbac.authorization.k8s.io/flannel created
  42. clusterrolebinding.rbac.authorization.k8s.io/flannel created
  43. serviceaccount/flannel created
  44. configmap/kube-flannel-cfg created
  45. daemonset.apps/kube-flannel-ds created
  48.  查看
  49. [root@master flannel]# kubectl get pods --namespace kube-system
  50. NAME                             READY   STATUS    RESTARTS   AGE
  51. coredns-5644d7b6d9-bs4gj         1/1     Running   0          31m
  52. coredns-5644d7b6d9-xf25l         1/1     Running   0          31m
  53. etcd-master                      1/1     Running   0          30m
  54. kube-apiserver-master            1/1     Running   0          30m
  55. kube-controller-manager-master   1/1     Running   0          30m
  56. kube-proxy-mp8tf                 1/1     Running   0          31m
  57. kube-scheduler-master            1/1     Running   0          30m
  60. [root@master flannel]#  kubectl get service
  61. NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
  62. kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   32m
  63. [root@master flannel]# kubectl get svc --namespace kube-system
  64. NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
  65. kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   32m

image.png
image.png
image.png

配置所有node节点加入集群

  1. 在所有node节点操作，此命令为初始化master成功后返回的结果
  2. kubeadm join 192.168.81.30:6443 --token dcg8kk.t42rp6f32p2szdda \
  3.     --discovery-token-ca-cert-hash sha256:7b21de9e4be342bd5d971fbc9859169c7606c29d735f057a767ffcdd439bce10 
  6. 然后master查看
  7. [root@master flannel]#  kubectl get nodes
  8. NAME     STATUS   ROLES    AGE     VERSION
  9. master   Ready    master   42m     v1.16.0
  10. node1    Ready    <none>   5m34s   v1.16.0
  11. node2    Ready    <none>   5m30s   v1.16.0

image.png