本部分将会在三台控制节点上部署 Kubernetes 控制服务，并配置高可用的集群架构。并且还会创建一个用于外部访问的负载均衡器。每个控制节点上需要部署的服务包括：Kubernetes API Server、Scheduler 以及 Controller Manager 等。

事前准备

以下命令需要在每台控制节点上面都运行一遍，包括 controller-0、controller-1 和 controller-2。可以使用 gcloud 命令登录每个控制节点。例如:

  1. gcloud compute ssh controller-0

可以使用 tmux 同时登录到三点控制节点上，加快部署步骤。

部署 Kubernetes 控制平面

创建 Kubernetes 配置目录

  1. sudo mkdir -p /etc/kubernetes/config

下载并安装 Kubernetes Controller 二进制文件

  1. wget -q --show-progress --https-only --timestamping \
  2.   "https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kube-apiserver" \
  3.   "https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kube-controller-manager" \
  4.   "https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kube-scheduler" \
  5.   "https://storage.googleapis.com/kubernetes-release/release/v1.12.0/bin/linux/amd64/kubectl"
  7. chmod +x kube-apiserver kube-controller-manager kube-scheduler kubectl
  8. sudo mv kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

配置 Kubernetes API Server

  1. sudo mkdir -p /var/lib/kubernetes/
  3. sudo mv ca.pem ca-key.pem kubernetes-key.pem kubernetes.pem \
  4.   service-account-key.pem service-account.pem \
  5.   encryption-config.yaml /var/lib/kubernetes/

使用节点的内网 IP 地址作为 API server 与集群内部成员的广播地址。首先查询当前节点的内网 IP 地址：

  1. INTERNAL_IP=$(curl -s -H "Metadata-Flavor: Google" \
  2.   http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip)

生成 kube-apiserver.service systemd 配置文件：

  1. cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
  2. [Unit]
  3. Description=Kubernetes API Server
  4. Documentation=https://github.com/kubernetes/kubernetes
  6. [Service]
  7. ExecStart=/usr/local/bin/kube-apiserver \\
  8.   --advertise-address=${INTERNAL_IP} \\
  9.   --allow-privileged=true \\
  10.   --apiserver-count=3 \\
  11.   --audit-log-maxage=30 \\
  12.   --audit-log-maxbackup=3 \\
  13.   --audit-log-maxsize=100 \\
  14.   --audit-log-path=/var/log/audit.log \\
  15.   --authorization-mode=Node,RBAC \\
  16.   --bind-address=0.0.0.0 \\
  17.   --client-ca-file=/var/lib/kubernetes/ca.pem \\
  18.   --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  19.   --enable-swagger-ui=true \\
  20.   --etcd-cafile=/var/lib/kubernetes/ca.pem \\
  21.   --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\
  22.   --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\
  23.   --etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \\
  24.   --event-ttl=1h \\
  25.   --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
  26.   --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
  27.   --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
  28.   --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
  29.   --kubelet-https=true \\
  30.   --runtime-config=api/all \\
  31.   --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
  32.   --service-cluster-ip-range=10.32.0.0/24 \\
  33.   --service-node-port-range=30000-32767 \\
  34.   --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
  35.   --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\
  36.   --v=2
  37. Restart=on-failure
  38. RestartSec=5
  40. [Install]
  41. WantedBy=multi-user.target
  42. EOF

配置 Kubernetes Controller Manager

生成 kube-controller-manager.service systemd 配置文件：

  1. sudo mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
  3. cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
  4. [Unit]
  5. Description=Kubernetes Controller Manager
  6. Documentation=https://github.com/kubernetes/kubernetes
  8. [Service]
  9. ExecStart=/usr/local/bin/kube-controller-manager \\
  10.   --address=0.0.0.0 \\
  11.   --cluster-cidr=10.200.0.0/16 \\
  12.   --cluster-name=kubernetes \\
  13.   --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  14.   --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  15.   --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  16.   --leader-elect=true \\
  17.   --root-ca-file=/var/lib/kubernetes/ca.pem \\
  18.   --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  19.   --service-cluster-ip-range=10.32.0.0/24 \\
  20.   --use-service-account-credentials=true \\
  21.   --v=2
  22. Restart=on-failure
  23. RestartSec=5
  25. [Install]
  26. WantedBy=multi-user.target
  27. EOF

配置 Kubernetes Scheduler

生成 kube-scheduler.service systemd 配置文件：

  1. sudo mv kube-scheduler.kubeconfig /var/lib/kubernetes/
  3. cat <<EOF | sudo tee /etc/kubernetes/config/kube-scheduler.yaml
  4. apiVersion: componentconfig/v1alpha1
  5. kind: KubeSchedulerConfiguration
  6. clientConnection:
  7.   kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
  8. leaderElection:
  9.   leaderElect: true
  10. EOF
  12. cat <<EOF | sudo tee /etc/systemd/system/kube-scheduler.service
  13. [Unit]
  14. Description=Kubernetes Scheduler
  15. Documentation=https://github.com/kubernetes/kubernetes
  17. [Service]
  18. ExecStart=/usr/local/bin/kube-scheduler \\
  19.   --config=/etc/kubernetes/config/kube-scheduler.yaml \\
  20.   --v=2
  21. Restart=on-failure
  22. RestartSec=5
  24. [Install]
  25. WantedBy=multi-user.target
  26. EOF

启动控制器服务

  1. sudo systemctl daemon-reload
  2. sudo systemctl enable kube-apiserver kube-controller-manager kube-scheduler
  3. sudo systemctl start kube-apiserver kube-controller-manager kube-scheduler

请等待 10 秒以便 Kubernetes API Server 初始化。

开启 HTTP 健康检查

Google Network Load Balancer 将用在在三个 API Server 之前作负载均衡，并可以终止 TLS 并验证客户端证书。但是该负载均衡仅支持 HTTP 健康检查，因而这里部署 nginx 来代理 API Server 的 /healthz 连接。

/healthz API 默认不需要认证。

  1. sudo apt-get update
  2. sudo apt-get install -y nginx
  4. cat > kubernetes.default.svc.cluster.local <<EOF
  5. server {
  6.   listen      80;
  7.   server_name kubernetes.default.svc.cluster.local;
  9.   location /healthz {
  10.      proxy_pass                    https://127.0.0.1:6443/healthz;
  11.      proxy_ssl_trusted_certificate /var/lib/kubernetes/ca.pem;
  12.   }
  13. }
  14. EOF
  16. sudo mv kubernetes.default.svc.cluster.local \
  17.     /etc/nginx/sites-available/kubernetes.default.svc.cluster.local
  18. sudo ln -s /etc/nginx/sites-available/kubernetes.default.svc.cluster.local /etc/nginx/sites-enabled/
  20. sudo systemctl restart nginx
  21. sudo systemctl enable nginx

验证

  1. kubectl get componentstatuses --kubeconfig admin.kubeconfig

将输出结果

  1. NAME                 STATUS    MESSAGE              ERROR
  2. controller-manager   Healthy   ok
  3. scheduler            Healthy   ok
  4. etcd-2               Healthy   {"health": "true"}
  5. etcd-0               Healthy   {"health": "true"}
  6. etcd-1               Healthy   {"health": "true"}

验证 Nginx HTTP 健康检查

  1. curl -H "Host: kubernetes.default.svc.cluster.local" -i http://127.0.0.1/healthz

将输出

  1. HTTP/1.1 200 OK
  2. Server: nginx/1.14.0 (Ubuntu)
  3. Date: Mon, 14 May 2018 13:45:39 GMT
  4. Content-Type: text/plain; charset=utf-8
  5. Content-Length: 2
  6. Connection: keep-alive
  8. ok

记得在每台控制节点上面都运行一遍，包括 controller-0、controller-1 和 controller-2。

Kubelet RBAC 授权

本节将会配置 API Server 访问 Kubelet API 的 RBAC 授权。访问 Kubelet API 是获取 metrics、日志以及执行容器命令所必需的。

这里设置 Kubeket --authorization-mode 为 Webhook 模式。Webhook 模式使用 SubjectAccessReview API 来决定授权。

  1. gcloud compute ssh controller-0

创建 system:kube-apiserver-to-kubelet ClusterRole 以允许请求 Kubelet API 和执行许用来管理 Pods 的任务:

  1. cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
  2. apiVersion: rbac.authorization.k8s.io/v1beta1
  3. kind: ClusterRole
  4. metadata:
  5.   annotations:
  6.     rbac.authorization.kubernetes.io/autoupdate: "true"
  7.   labels:
  8.     kubernetes.io/bootstrapping: rbac-defaults
  9.   name: system:kube-apiserver-to-kubelet
  10. rules:
  11.   - apiGroups:
  12.       - ""
  13.     resources:
  14.       - nodes/proxy
  15.       - nodes/stats
  16.       - nodes/log
  17.       - nodes/spec
  18.       - nodes/metrics
  19.     verbs:
  20.       - "*"
  21. EOF

Kubernetes API Server 使用客户端凭证授权 Kubelet 为 kubernetes 用户，此凭证用 --kubelet-client-certificate flag 来定义。

绑定 system:kube-apiserver-to-kubelet ClusterRole 到 kubernetes 用户:

  1. cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
  2. apiVersion: rbac.authorization.k8s.io/v1beta1
  3. kind: ClusterRoleBinding
  4. metadata:
  5.   name: system:kube-apiserver
  6.   namespace: ""
  7. roleRef:
  8.   apiGroup: rbac.authorization.k8s.io
  9.   kind: ClusterRole
  10.   name: system:kube-apiserver-to-kubelet
  11. subjects:
  12.   - apiGroup: rbac.authorization.k8s.io
  13.     kind: User
  14.     name: kubernetes
  15. EOF

Kubernetes 前端负载均衡器

本节将会建立一个位于 Kubernetes API Servers 前端的外部负载均衡器。 kubernetes-the-hard-way 静态 IP 地址将会配置在这个负载均衡器上。

本指南创建的虚拟机内部并没有操作负载均衡器的权限，需要到创建这些虚拟机的那台机器上去做下面的操作。

创建外部负载均衡器网络资源：

  1. KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  2.   --region $(gcloud config get-value compute/region) \
  3.   --format 'value(address)')
  5. gcloud compute http-health-checks create kubernetes \
  6.   --description "Kubernetes Health Check" \
  7.   --host "kubernetes.default.svc.cluster.local" \
  8.   --request-path "/healthz"
  10. gcloud compute firewall-rules create kubernetes-the-hard-way-allow-health-check \
  11.   --network kubernetes-the-hard-way \
  12.   --source-ranges 209.85.152.0/22,209.85.204.0/22,35.191.0.0/16 \
  13.   --allow tcp
  15. gcloud compute target-pools create kubernetes-target-pool \
  16.   --http-health-check kubernetes
  18. gcloud compute target-pools add-instances kubernetes-target-pool \
  19.   --instances controller-0,controller-1,controller-2
  21. gcloud compute forwarding-rules create kubernetes-forwarding-rule \
  22.   --address ${KUBERNETES_PUBLIC_ADDRESS} \
  23.   --ports 6443 \
  24.   --region $(gcloud config get-value compute/region) \
  25.   --target-pool kubernetes-target-pool

验证

查询 kubernetes-the-hard-way 静态 IP 地址:

  1. KUBERNETES_PUBLIC_ADDRESS=$(gcloud compute addresses describe kubernetes-the-hard-way \
  2.   --region $(gcloud config get-value compute/region) \
  3.   --format 'value(address)')

发送一个查询 Kubernetes 版本信息的 HTTP 请求

  1. curl --cacert ca.pem https://${KUBERNETES_PUBLIC_ADDRESS}:6443/version

结果为

  1. {
  2.   "major": "1",
  3.   "minor": "12",
  4.   "gitVersion": "v1.12.0",
  5.   "gitCommit": "0ed33881dc4355495f623c6f22e7dd0b7632b7c0",
  6.   "gitTreeState": "clean",
  7.   "buildDate": "2018-09-27T16:55:41Z",
  8.   "goVersion": "go1.10.4",
  9.   "compiler": "gc",
  10.   "platform": "linux/amd64"
  11. }

下一步：部署 Kubernetes Worker 节点。