1. 安装环境

  • Kubernetes：v1.16.2
  • Docker CE：19.03.4-3.el7.x86_64
  • 创建三台CentOS虚拟主机，一个Master节点，2个Node节点：
    • 系统版本：CentOS Linux release 8.0.1905 (Core)
    • 内核版本：4.18.0-80.el8.x86_64
    • Master角色主机配置为内存8GB，CPU总核心数为2，系统盘空间为50GB
    • Node角色主机配置为内存2GB，CPU总核心数为2，系统盘空间为50GB
    • 主机之间网络互通，并且主机能够访问外网（需要拉取镜像）

总结一下安装条件：

  1. 64位Linux操作系统，3.10及以上的内核版本
  2. x86或ARM架构
  3. 主机之间网络互通，主机能够访问外网
  4. Master角色机器至少是2核CPU和2GB内存

    2. 部署前准备工作

    每台机器都要做如下初始化环境的操作。

配置hosts

  1. hostnamectl --static set-hostname {{hostname}} # {{hostname}}为当前机器的主机名称
  3. # 为每台服务器添加host解析记录
  4. cat >>/etc/hosts<<EOF
  5. 192.168.0.15 kubernetes-0
  6. 192.168.0.16 kubernetes-1
  7. 192.168.0.17 kubernetes-2
  8. EOF

关闭selinux和防火墙

  1. sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
  2. setenforce 0
  3. systemctl disable firewalld
  4. systemctl stop firewalld

关闭虚拟内存

为什么关闭虚拟内存？

  • 现有的workloads QoS不支持swap
  • 性能预测需要内存延迟可重复验证，开启swap对预测有影响
  • 会减慢任务执行速度，并且加大磁盘带宽，引入隔离性的问题

参考：

  1. swapoff -a
  2. sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

配置内核参数

  1. cat <<EOF > /etc/sysctl.d/k8s.conf
  2. net.bridge.bridge-nf-call-ip6tables = 1
  3. net.bridge.bridge-nf-call-iptables = 1
  4. net.ipv4.ip_forward = 1
  5. vm.swappiness=0
  6. EOF
  8. sysctl --system

加载IPVS模块

  1. # 该文件保证节点重启后能自己加载所需模块
  2. cat > /etc/sysconfig/modules/ipvs.modules <<EOF
  3. #!/bin/bash
  4. modprobe -- ip_vs
  5. modprobe -- ip_vs_rr
  6. modprobe -- ip_vs_wrr
  7. modprobe -- ip_vs_sh
  8. modprobe -- nf_conntrack_ipv4
  9. EOF
  11. chmod 755 /etc/sysconfig/modules/ipvs.modules
  12. # 加载模块
  13. bash /etc/sysconfig/modules/ipvs.modules
  14. # 查看加载情况
  15. lsmod | grep -e ip_vs -e nf_conntrack_ipv4

加载完模块要保证每个节点都安装了ipset软件包，为了方便查看ipvs的代理规则，再安装管理工具ipvsadm：

  1. yum install -y ipset ipvsadm

配置yum源

使用阿里源

  1. cat << EOF > /etc/yum.repos.d/kubernetes.repo
  2. [kubernetes]
  3. [kubernetes]
  4. name=Kubernetes
  5. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
  6. enabled=1
  7. gpgcheck=1
  8. repo_gpgcheck=1
  9. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  10. EOF
  12. yum makecache

安装Docker

  1. sudo yum install -y yum-utils \
  2. device-mapper-persistent-data \
  3. lvm2
  5. sudo yum-config-manager \
  6.   --add-repo \
  7.   https://download.docker.com/linux/centos/docker-ce.repo
  9. sudo dnf install https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
  11. sudo yum install -y docker-ce
  12. systemctl enable docker && systemctl start docker

安装kubernetes

  1. sudo yum install -y kubelet kubeadm kubectl
  2. systemctl enable kubelet

配置Cgroup驱动

官方文档推荐Cgroup驱动使用systemd，理由如下：

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager. Systemd has a tight integration with cgroups and will allocate cgroups per process. It’s possible to configure your container runtime and the kubelet to use cgroupfs. Using cgroupfs alongside systemd means that there will then be two different cgroup managers. Control groups are used to constrain resources that are allocated to processes. A single cgroup manager will simplify the view of what resources are being allocated and will by default have a more consistent view of the available and in-use resources. When we have two managers we end up with two views of those resources. We have seen cases in the field where nodes that are configured to use cgroupfs for the kubelet and Docker, and systemd for the rest of the processes running on the node becomes unstable under resource pressure. Changing the settings such that your container runtime and kubelet use systemd as the cgroup driver stabilized the system. Please note the native.cgroupdriver=systemd option in the Docker setup below.

  1. cat <<EOF > /etc/docker/daemon.json
  2. {
  3.   "exec-opts": ["native.cgroupdriver=systemd"],
  4.   "log-driver": "json-file",
  5.   "log-opts": {
  6.     "max-size": "100m"
  7.   },
  8.   "storage-driver": "overlay2"
  9. }
  10. EOF
  12. systemctl restart docker
  13. # 检查配置是否成功
  14. docker info | grep Cgroup

3. 部署kubernetes集群

3.1. 部署Master节点

修改初始化配置文件

  1. # 打印出默认配置
  2. kubeadm config print init-defaults > kubeadm-init.yaml
  4. # 根据自身环境重新修改配置文件
  5. cat kubeadm-init.yaml
  1. apiVersion: kubeadm.k8s.io/v1beta2
  2. kind: InitConfiguration
  3. ---
  4. apiVersion: kubeadm.k8s.io/v1beta2
  5. certificatesDir: /etc/kubernetes/pki
  6. clusterName: kubernetes
  7. controllerManager: {
  8.   extraArgs: {
  9.     horizontal-pod-autoscaler-use-rest-clients: "true",
  10.     horizontal-pod-autoscaler-sync-period: "10s",
  11.     node-monitor-grace-period: "120s"
  12.   }
  13. }
  14. dns:
  15.   type: CoreDNS
  16. etcd:
  17.   local:
  18.     dataDir: /var/lib/etcd
  19. imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
  20. kind: ClusterConfiguration
  21. kubernetesVersion: v1.16.2
  22. scheduler: {}

初始化集群

  1. kubeadm init --config kubeadm-init.yaml
  3. ...
  4. Your Kubernetes control-plane has initialized successfully!
  6. To start using your cluster, you need to run the following as a regular user:
  8.   mkdir -p $HOME/.kube
  9.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  10.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  12. You should now deploy a pod network to the cluster.
  13. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  14.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  16. Then you can join any number of worker nodes by running the following on each as root:
  18. kubeadm join 192.168.0.15:6443 --token nbbusq.p92jg36mvvatzuhv \
  19.     --discovery-token-ca-cert-hash sha256:fac472c7dce2fd3ce33327c2e0e925a252f43033540ea3096337f9e9ea781e23

为kubectl命令准备Kubeconfig文件

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

部署网络插件

  1. kubectl get nodes
  3. NAME           STATUS      ROLES    AGE     VERSION
  4. kubernetes-0   NotReady    master   2d16h   v1.16.2

Master节点NotReady的原因是没有部署网络插件，导致master节点网络没有就绪，网络插件有很多，比如Flannel、Calico等等，这里我们选择Weave：

  1. kubectl apply -f "https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d '\n')"

再次查看节点状态，master节点已变为Ready状态。

3.2. 部署Node节点

在其余节点分别执行以下命令：

  1. kubeadm join 192.168.0.15:6443 --token nbbusq.p92jg36mvvatzuhv \
  2.     --discovery-token-ca-cert-hash sha256:fac472c7dce2fd3ce33327c2e0e925a252f43033540ea3096337f9e9ea781e23

部署Node节点不需要--experimental-control-plane参数，部署Master集群时，其他Master加入集群需要加上该参数。

3.3. 部署容器存储插件

这里选用基于Ceph的存储插件Rook，需要三个配置文件

  1. kubectl create -f common.yaml
  2. kubectl create -f operator.yaml
  4. # 确认rook-ceph-operator是Running状态再进行下一步操作
  5. kubectl -n rook-ceph get pod
  7. kubectl create -f cluster-test.yaml