2.4 在 kubernetes 上安装 - 《持续集成部署GitLab CICD》

1. kubernetes 安装
2. 配置 gitlab
3. gitlab 添加 k8s 集群

1. kubernetes 安装
2. 配置 gitlab
3. gitlab 添加 k8s 集群

1. kubernetes 安装

（1）更新内核

$ curl -fsSL https://gitee.com/jack_zang/kubernetes/raw/master/install/kubeadm_install_kernel.sh | bash
$ reboot

(2) 开启 ipvs

$ curl -fsSL https://gitee.com/jack_zang/kubernetes/raw/master/install/kubeadm_enable_ipvs.sh | bash

(3) 安装 kubernetes

#$ curl -fsSL https://gitee.com/jack_zang/kubernetes/raw/master/install/kubeadm_install_kubernetes.sh | bash
$ curl -fsSL https://gitee.com/jack_zang/kubernetes/raw/master/install/kubeadm_install_kubernetes.sh|sed  s/1.16.6/1.14.8/g | bash

(4) 创建初始化配置文件

$ cat > /etc/kubernetes/kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: v1.14.8
controlPlaneEndpoint: "192.168.20.61:6443"
apiServer:
  certSANs:
  - 192.168.20.61
networking:
  podSubnet: 10.244.0.0/16
imageRepository: "registry.aliyuncs.com/google_containers"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
EOF

(5) 初始化集群

$ kubeadm init --config /etc/kubernetes/kubeadm-config.yaml
$ mkdir -p $HOME/.kube
$ cp -f /etc/kubernetes/admin.conf ${HOME}/.kube/config
$ curl -fsSL https://docs.projectcalico.org/v3.9/manifests/calico.yaml| sed "s@192.168.0.0/16@10.244.0.0/16@g" | kubectl apply -f -

(6) 其它 node 节点加入集群

$ kubeadm token create --print-join-command        # master 节点执行，然后拷贝到 Node 节点执行

(7) 创建持久存储

//安装nfs

$ curl -fsSL https://gitee.com/jack_zang/kubernetes/raw/master/nfs/nfs_install.sh | bash

//创建用户

$ kubectl apply -f https://gitee.com/jack_zang/kubernetes/raw/master/nfs/serviceaccount.yaml
$ kubectl apply -f https://gitee.com/jack_zang/kubernetes/raw/master/nfs/nfs_deployment.yaml
$ kubectl apply -f https://gitee.com/jack_zang/kubernetes/raw/master/nfs/class.yaml

(8) 安装 ingress-control

参考：https://kubernetes.github.io/ingress-nginx/deploy/#bare-metal

//nodeport方式

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml

//负载均衡器方式

$ curl -fsSL https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/deploy.yaml | sed "s@NodePort@LoadBalancer@g" | kubectl apply -f -

(9) 安装 metallb

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml
$ kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
$ cat > metalLB-config.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.20.20-192.168.20.25
EOF
$ kubectl apply -f metalLB-config.yaml

2. 配置 gitlab

由于国内的一些原因，有些镜像无法下载，所以需要对其做一些配置。

//修改 helm 工具，修改 image 镜像下载地址，镜像名称保持不变

$ vim /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/pod.rb
...
       def container_specification
          {
            name: 'helm',
            image: "harbor.xiodi.cn/tools/#{Gitlab::Kubernetes::Helm::HELM_VERSION}-kube-#{Gitlab::Kubernetes::Helm::KUBECTL_VERSION}",
            env: generate_pod_env(command),
            command: %w(/bin/sh),
            args: %w(-c $(COMMAND_SCRIPT))
          }
        end
...

//修改 helm init 命令

$ vim /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/init_command.rb
...
       def init_helm_command
          command = %w[helm init --stable-repo-url http://mirror.azure.cn/kubernetes/charts/ --tiller-image harbor.xiodi.cn/tools/tiller:v2.16.3 ] + init_command_flags

          command.shelljoin
        end

...

原始镜像名称：gcr.io/kubernetes-helm/tiller:v2.16.3

//修改 client

$ vim /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/client_command.rb
...
       def init_command
          if local_tiller_enabled?
            <<~HEREDOC.chomp
            export HELM_HOST="localhost:44134"
            tiller -listen ${HELM_HOST} -alsologtostderr &
            helm init --client-only
            HEREDOC
          else
            # Here we are always upgrading to the latest version of Tiller when
            # installing an app. We ensure the helm version stored in the
            # database is correct by also updating this after transition to
            # :installed,:updated in Clusters::Concerns::ApplicationStatus
            'helm init --upgrade --stable-repo-url http://mirror.azure.cn/kubernetes/charts/ --tiller-image harbor.xiodi.cn/tools/tiller:v2.16.3'
          end
        end
...

3. gitlab 添加 k8s 集群

//api地址

$ kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'
https://192.168.20.61:6443

// ca 证书

$ kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-8kxc9   kubernetes.io/service-account-token   3      23m

$ kubectl get secret default-token-8kxc9 -o jsonpath="{['data']['ca\.crt']}" | base64 --decode
-----BEGIN CERTIFICATE-----
MIICyDCCAbCgAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
cm5ldGVzMB4XDTIwMDQwMTEzNDkzMloXDTMwMDMzMDEzNDkzMlowFTETMBEGA1UE
AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAK84
O0dwlYZYquCwDOf1mK7aHtxb5cjkb+pGEwPXvqy4n7ynVa7lvzaP/woGNg/5xIwg
GjDHTQoaFD/KJxp2AZBg4XV6etlxsqfR6NASPlaz3L0fbxbYI4NEDzCHiZrpiB4w
TK9jlh20xcC1jgWKmWgVYfJTF7QoT2pNTPGhXYvgaXJ0fCdcdUVvjvisAS2yvXBq
rKi74f2BWUQSdsoB+UQtyZNcZUP789SXqBACI1Qdz/Zcd+Oxt+cgtheRpDdokyNO
8/6ToGxy9JDvUkesoG+UKrL6Y/VUW8OvAws7iu/Ja3uQKIVFdWdt3oqbJW8VHaLW
qDd4XYB8tKJvH1M2xT0CAwEAAaMjMCEwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAETqbi7w0fs0QELYADCDSMgSiG+8
Q3N84QCtEW3QSpfXFP5tRcagKOPn0Mt87Qpu2klKQ5XErwU8AE5eS4NGM3mykzl8
oHjR1ej1OkWKMtobai/qbPfIg3EkPnC/0PjSrhM43B/MSIWj09YnnlD0WbGZTdBt
WS1aOoo4ntw20l2WKcpDsg5K/x5k6HWNnx8gthYpwajY854XA2CbPJSapeY6KSmE
to7no/BJTEK0xclhlJ2amxpXZsGYIUyFvVjbsbWmyuPKfO8kDEfnYYdh8UxR/RLb
1H7LPB6bvyJsWESVyUrTUZFQN6fDUUcqODGtVEDD30odVHDbfS2U8gsRChE=
-----END CERTIFICATE-----

//创建服务令牌

$ cat > gitlab-admin-service-account.yaml <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: gitlab-admin
  namespace: kube-system
EOF

$ kubectl apply -f gitlab-admin-service-account.yaml
$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}')

// 安装 helm
// 安装 Runner