节点规划信息
| 名称 | IP | 功能 |
|---|---|---|
| htst-7-11 | 10.4.7.11 | master |
| htst-7-12 | 10.4.7.12 | master |
| htst-7-21 | 10.4.7.21 | node |
| htst-7-22 | 10.4.7.22 | node |
| htst-7-200 | 10.4.7.200 |
基础环境配置
环境信息
| 系统 | CentOS7.6.1810 |
|---|---|
| 内核版本 | 4.9.220 |
| 软件 | 版本 |
|---|---|
| kubernetes | 1.18.2 |
| docker | 19.0.3 |
环境初始化
（1）、配置主机名，以k8s-master01为例
~]# hostnamectl set-hostname htst-7-11
（1）、配置主机hosts映射
~]# vim /etc/hosts
10.4.7.11 htst-7-11
10.4.7.12 htst-7-12
10.4.7.21 htst-7-21
10.4.7.22 htst-7-22
10.4.7.200 htst-7-200
（3）、配置网络
~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
PEERDNS=yes
NAME=ens33
DEVICE=ens33
ONBOOT=yes
IPADDR=10.4.7.51
GATEWAY=10.4.7.254
NETMASK=255.255.255.0
DNS1=10.4.7.0
注意，这个GATEWAY和DNS1要根据实际情况来，虚拟机的话就去看看虚拟机网络配置。
~]# systemctl restart network
（4）、禁用防火墙
~]# systemctl stop firewalld
~]# systemctl disable firewalld
（3）、关闭selinux
~]# setenforce 0
~]# sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/sysconfig/selinux
~]# sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config
（4）、关闭swap分区
~]# swapoff -a && sysctl -w vm.swappiness=0
~]# sed -i.bak '/swap/s/^/#/' /etc/fstab
（5）、时间同步
~]# yum install chrony -y
~]# systemctl enable chronyd
~]# systemctl start chronyd
~]# chronyc sources
（6）、配置ulimt
~]# ulimit -SHn 65535
（7）、配置内核参数
~]# cat >> /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF
使之生效
~]# sysctl -p
（8）、master之间添加互信（按需）
~]# ssh-keygen
~]# ssh-copy-id 10.4.7.11
~]# ssh-copy-id 10.4.7.12
内核升级
由于centos7.6的系统默认内核版本是3.10，3.10的内核有很多BUG，最常见的一个就是group memory leak。
（1）、下载所需要的内核版本，我这里采用rpm安装，所以直接下载的rpm包
~]# wget https://cbs.centos.org/kojifiles/packages/kernel/4.9.220/37.el7/x86_64/kernel-4.9.220-37.el7.x86_64.rpm
（2）、执行rpm升级即可
~]# rpm -ivh kernel-4.9.220-37.el7.x86_64.rpm
（3）、升级完reboot，然后查看内核是否成功升级
~]# reboot
~]# uname -r
组件安装
安装ipvs
（1）、安装ipvs需要的软件
由于我准备使用ipvs作为kube-proxy的代理模式，所以需要安装相应的软件包。
yum install ipvsadm ipset sysstat conntrack libseccomp -y
（2）、加载模块
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
modprobe -- ip_tables
modprobe -- ip_set
modprobe -- xt_set
modprobe -- ipt_set
modprobe -- ipt_rpfilter
modprobe -- ipt_REJECT
modprobe -- ipip
EOF
注意：在内核4.19版本nf_conntrack_ipv4已经改为nf_conntrack
配置重启自动加载
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack
安装docker-ce
# 安装需要的软件
yum install -y yum-utils device-mapper-persistent-data lvm2
# 添加yum源
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
查看是否有docker-ce包
# yum list | grep docker-ce
containerd.io.x86_64 1.2.13-3.1.el7 docker-ce-stable
docker-ce.x86_64 3:19.03.8-3.el7 docker-ce-stable
docker-ce-cli.x86_64 1:19.03.8-3.el7 docker-ce-stable
docker-ce-selinux.noarch 17.03.3.ce-1.el7 docker-ce-stable
安装docker-ce
~]# wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
~]# yum install -y docker-ce
~]# systemctl start docker
~]# systemctl enable docker
配置镜像加速
~]# curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io
~]# systemctl restart docker
修改Cgroup Driver
~]# vim /etc/docker/daemon.json
{
"registry-mirrors": ["https://v16stybc.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
# 重新加载docker
~]# systemctl daemon-reload
~]# systemctl restart docker
安装kubernetes组件
添加yum源
~]# cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
安装软件
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
将kubelet设置为开机自启动
systemctl enable kubelet.service
以上操作在所有节点执行
集群初始化
配置VIP
高可用采用的是HAProxy+Keepalived，HAProxy和KeepAlived以守护进程的方式在所有Master节点部署。
安装软件
yum install keepalived haproxy -y
配置haproxy
所有master节点的配置相同，如下：
~]# vim /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
#---------------------------------------------------------------------
# kubernetes apiserver frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kubernetes
mode tcp
bind *:16443
option tcplog
default_backend kubernetes-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kubernetes-apiserver
mode tcp
balance roundrobin
server htst-7-11 10.4.7.11:6443 check
server htst-7-12 10.4.7.12:6443 check
#---------------------------------------------------------------------
# collection haproxy statistics message
#---------------------------------------------------------------------
listen stats
bind *:9999
stats auth admin:P@ssW0rd
stats refresh 5s
stats realm HAProxy\ Statistics
stats uri /admin?stats
配置keepalived
htst-7-11
~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
# 定义脚本
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 2
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth33
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.4.7.200
}
# 调用脚本
track_script {
check_apiserver
}
}
htst-7-12
! Configuration File for keepalived
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 192.168.200.1
smtp_connect_timeout 30
router_id LVS_DEVEL
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
# 定义脚本
vrrp_script check_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 2
weight -5
fall 3
rise 2
}
vrrp_instance VI_1 {
state MASTER
interface eth33
virtual_router_id 51
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
10.4.7.200
}
# 调用脚本
track_script {
check_apiserver
}
}
先把健康检查关闭，等部署好了过后再打开
编写健康检测脚本check-apiserver.sh
~]# vim /etc/keepalived/check_apiserver.sh
#!/bin/bash
function check_apiserver(){
for ((i=0;i<5;i++))
do
apiserver_job_id=${pgrep kube-apiserver}
if [[ ! -z ${apiserver_job_id} ]];then
return
else
sleep 2
fi
done
apiserver_job_id=0
}
# 1->running 0->stopped
check_apiserver
if [[ $apiserver_job_id -eq 0 ]];then
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
启动haproxy和keepalived
systemctl enable --now keepalived
systemctl enable --now haproxy
部署master
（1）、在htst-7-11上，编写kubeadm.yaml配置文件(这里的k8s版本要和安装的docker版本，kubectl，kubelet和kubeadm等软件相匹配)，如下：
cat >> kubeadm.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.22.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
controlPlaneEndpoint: "k8s-lb:16443"
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16
serviceSubnet: 10.96.0.0/12
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
SupportIPVSProxyMode: true
mode: ipvs
EOF
提前下载镜像
kubeadm config images pull --config kubeadm.yaml
进行初始化
kubeadm init --config kubeadm.yaml --upload-certs
W0509 22:37:40.702752 65728 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-lb] and IPs [10.96.0.1 10.1.10.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [10.1.10.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [10.1.10.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0509 22:37:47.750722 65728 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0509 22:37:47.764989 65728 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 20.024575 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
f25e738324e4f027703f24b55d47d28f692b4edc21c2876171ff87877dc8f2ef
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 3k4vr0.x3y2nc3ksfnei4y1
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \
--discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791 \
--control-plane --certificate-key f25e738324e4f027703f24b55d47d28f692b4edc21c2876171ff87877dc8f2ef
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \
--discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791
配置环境变量
cat >> /root/.bashrc <<EOF
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc
查看节点状态
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady master 3m1s v1.18.2
安装网络插件
wget https://docs.projectcalico.org/v3.8/manifests/calico.yaml
如果有节点是多网卡，所以需要在资源清单文件中指定内网网卡
vi calico.yaml
......
spec:
containers:
- env:
- name: DATASTORE_TYPE
value: kubernetes
- name: IP_AUTODETECTION_METHOD # DaemonSet中添加该环境变量
value: interface=ens33 # 指定内网网卡
- name: WAIT_FOR_DATASTORE
value: "true"
......
kubectl apply -f calico.yaml # 安装calico网络插件
当网络插件安装完成后，查看node节点信息如下：
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready master 10m v1.18.2
可以看到状态已经从NotReady变为ready了。
（2）、将master02加入集群
提前下载镜像
kubeadm config images pull --config kubeadm.yaml
加入集群
kubeadm join k8s-lb:16443 --token 3k4vr0.x3y2nc3ksfnei4y1 \
--discovery-token-ca-cert-hash sha256:a5f761f332bd45a199d0676875e7f58c323226df6fb9b4f0b977b6f63b252791 \
--control-plane --certificate-key f25e738324e4f027703f24b55d47d28f692b4edc21c2876171ff87877dc8f2ef
输出如下：
...
This node has joined the cluster and a new control plane instance was created:
* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Control plane (master) label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.
* A new etcd member was added to the local/stacked etcd cluster.
To start administering your cluster from this node, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Run 'kubectl get nodes' to see this node join the cluster.
...
配置环境变量
cat >> /root/.bashrc <<EOF
export KUBECONFIG=/etc/kubernetes/admin.conf
EOF
source /root/.bashrc
另一台的操作一样。
查看集群状态
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready master 41m v1.18.2
k8s-master02 Ready master 29m v1.18.2
k8s-master03 Ready master 27m v1.18.2
查看集群组件状态
# kubectl get pod -n kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-77c5fc8d7f-stl57 1/1 Running 0 26m 192.168.32.130 k8s-master01 <none> <none>
calico-node-ppsph 1/1 Running 0 26m 10.1.10.100 k8s-master01 <none> <none>
calico-node-tl6sq 0/1 Init:2/3 0 26m 10.1.10.101 k8s-master02 <none> <none>
calico-node-w92qh 1/1 Running 0 26m 10.1.10.102 k8s-master03 <none> <none>
coredns-546565776c-vtlhr 1/1 Running 0 42m 192.168.32.129 k8s-master01 <none> <none>
coredns-546565776c-wz9bk 1/1 Running 0 42m 192.168.32.131 k8s-master01 <none> <none>
etcd-k8s-master01 1/1 Running 0 42m 10.1.10.100 k8s-master01 <none> <none>
etcd-k8s-master02 1/1 Running 0 30m 10.1.10.101 k8s-master02 <none> <none>
etcd-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>
kube-apiserver-k8s-master01 1/1 Running 0 42m 10.1.10.100 k8s-master01 <none> <none>
kube-apiserver-k8s-master02 1/1 Running 0 30m 10.1.10.101 k8s-master02 <none> <none>
kube-apiserver-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>
kube-controller-manager-k8s-master01 1/1 Running 1 42m 10.1.10.100 k8s-master01 <none> <none>
kube-controller-manager-k8s-master02 1/1 Running 1 30m 10.1.10.101 k8s-master02 <none> <none>
kube-controller-manager-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>
kube-proxy-6sbpp 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>
kube-proxy-dpppr 1/1 Running 0 42m 10.1.10.100 k8s-master01 <none> <none>
kube-proxy-ln7l7 1/1 Running 0 30m 10.1.10.101 k8s-master02 <none> <none>
kube-scheduler-k8s-master01 1/1 Running 1 42m 10.1.10.100 k8s-master01 <none> <none>
kube-scheduler-k8s-master02 1/1 Running 1 30m 10.1.10.101 k8s-master02 <none> <none>
kube-scheduler-k8s-master03 1/1 Running 0 28m 10.1.10.102 k8s-master03 <none> <none>
查看CSR
kubectl get csr
NAME AGE SIGNERNAME REQUESTOR CONDITION
csr-cfl2w 42m kubernetes.io/kube-apiserver-client-kubelet system:node:k8s-master01 Approved,Issued
csr-mm7g7 28m kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:3k4vr0 Approved,Issued
csr-qzn6r 30m kubernetes.io/kube-apiserver-client-kubelet system:bootstrap:3k4vr0 Approved,Issued
部署node
若有收获，就点个赞吧
0 人点赞
