k8s的DNS实现了服务在集群内被自动发现，那如何使得服务在K8S集群’外’被使用和访问呢?
可以使用NodePort型的Service或者Ingress资源

一、NodePort 型的Service

这种方式是通过nginx将服务转发给没一个k8s节点, 有多少个节点,就要在nginx里配置多少个节点
这种方式很不灵活,现在已经不会用这种方式来暴露服务了

二、Ingress

Ingress是K8S API的标准资源类型之一,也是一种核心资源，它本质上是一组基于域名和URL路径，把用户的请求转发至指定Service资源的规则
可以将集群外部的请求流量，转发至集群内部，从而实现’服务暴露’
Ingress控制器是能够为Ingress资源监听某套接字，然后根据Ingress规则匹配机制路由调度流量的一个组件

常用的 Ingress控制器的实现软件

  • Ingress-nginx
  • HAProxy
  • Traefik
  • …

本次实验使用最常用的Traefik

2.1 创建pod定义文件(把定义文件放到 k8s-yaml.od.com对应的的目录下,在k8s-5-141服务器上操作)

  1. [root@k8s-5-141 /]# cd /data/k8s-yaml
  2. [root@k8s-5-141 k8s-yaml]# mkdir traefik
  3. [root@k8s-5-141 k8s-yaml]# ll
  4. total 0
  5. drwxr-xr-x 2 root root 69 Apr  1 10:51 coredns
  6. drwxr-xr-x 2 root root  6 Apr  2 16:34 traefik
  8. [root@k8s-5-141 k8s-yaml]# cd traefik
  10. #创建定义文件,因为我没有做vip,所以代理地址设置的是 - --kubernetes.endpoint=https://192.168.5.137:7443 (也可以设置到另外一台代理服务器)
  11. [root@k8s-5-141 k8s-yaml]# vim ds.yaml
  13. apiVersion: apps/v1
  14. kind: DaemonSet
  15. metadata:
  16.   name: traefik-ingress
  17.   namespace: kube-system
  18.   labels:
  19.     k8s-app: traefik-ingress
  20. spec:
  21.   selector: 
  22.     matchLabels:
  23.       name: traefik-ingress
  24.       k8s-app: traefik-ingress
  25.   template:
  26.     metadata:
  27.       labels:
  28.         k8s-app: traefik-ingress
  29.         name: traefik-ingress
  30. #--------prometheus自动发现增加内容--------
  31.       annotations:
  32.         prometheus_io_scheme: "traefik"
  33.         prometheus_io_path: "/metrics"
  34.         prometheus_io_port: "8080"
  35. #--------增加结束--------------------------
  36.     spec:
  37.       serviceAccountName: traefik-ingress-controller
  38.       terminationGracePeriodSeconds: 60
  39.       containers:
  40.       - image: harbor.od.com/public/traefik:v1.7.2
  41.         name: traefik-ingress
  42.         ports:
  43.         - name: controller
  44.           containerPort: 80
  45.           hostPort: 81
  46.         - name: admin-web
  47.           containerPort: 8080
  48.         securityContext:
  49.           capabilities:
  50.             drop:
  51.             - ALL
  52.             add:
  53.             - NET_BIND_SERVICE
  54.         args:
  55.         - --api
  56.         - --kubernetes
  57.         - --logLevel=INFO
  58.         - --insecureskipverify=true
  59.         - --kubernetes.endpoint=https://192.168.5.137:7443
  60.         - --accesslog
  61.         - --accesslog.filepath=/var/log/traefik_access.log
  62.         - --traefiklog
  63.         - --traefiklog.filepath=/var/log/traefik.log
  64.         - --metrics.prometheus
  66. [root@k8s-5-141 k8s-yaml]# vim ingress.yaml
  68. apiVersion: extensions/v1beta1
  69. kind: Ingress
  70. metadata:
  71.   name: traefik-web-ui
  72.   namespace: kube-system
  73.   annotations:
  74.     kubernetes.io/ingress.class: traefik
  75. spec:
  76.   rules:
  77.   - host: traefik.od.com 
  78.     http:
  79.       paths:
  80.       - path: /
  81.         backend:
  82.           serviceName: traefik-ingress-service
  83.           servicePort: 8080
  86. [root@k8s-5-141 k8s-yaml]# vim rbac.yaml
  89. apiVersion: v1
  90. kind: ServiceAccount
  91. metadata:
  92.   name: traefik-ingress-controller
  93.   namespace: kube-system
  94. ---
  95. apiVersion: rbac.authorization.k8s.io/v1beta1
  96. kind: ClusterRole
  97. metadata:
  98.   name: traefik-ingress-controller
  99. rules:
  100.   - apiGroups:
  101.       - ""
  102.     resources:
  103.       - services
  104.       - endpoints
  105.       - secrets
  106.     verbs:
  107.       - get
  108.       - list
  109.       - watch
  110.   - apiGroups:
  111.       - extensions
  112.     resources:
  113.       - ingresses
  114.     verbs:
  115.       - get
  116.       - list
  117.       - watch
  118. ---
  119. kind: ClusterRoleBinding
  120. apiVersion: rbac.authorization.k8s.io/v1beta1
  121. metadata:
  122.   name: traefik-ingress-controller
  123. roleRef:
  124.   apiGroup: rbac.authorization.k8s.io
  125.   kind: ClusterRole
  126.   name: traefik-ingress-controller
  127. subjects:
  128. - kind: ServiceAccount
  129.   name: traefik-ingress-controller
  130.   namespace: kube-system
  134. [root@k8s-5-141 k8s-yaml]# vim svc.yaml
  136. kind: Service
  137. apiVersion: v1
  138. metadata:
  139.   name: traefik-ingress-service
  140.   namespace: kube-system
  141. spec:
  142.   selector:
  143.     k8s-app: traefik-ingress
  144.   ports:
  145.     - protocol: TCP
  146.       port: 80
  147.       name: controller
  148.     - protocol: TCP
  149.       port: 8080
  150.       name: admin-web

2.2 拉取 traefik 镜像 (在任何一台可以连接到 私有镜像仓库,并且安装了docker的服务器上操作都行)

  1. [root@k8s-5-138 redis]# docker pull traefik:v1.7.2-alpine
  2. v1.7.2-alpine: Pulling from library/traefik
  3. 4fe2ade4980c: Pull complete 
  4. 8d9593d002f4: Pull complete 
  5. 5d09ab10efbd: Pull complete 
  6. 37b796c58adc: Pull complete 
  7. Digest: sha256:cf30141936f73599e1a46355592d08c88d74bd291f05104fe11a8bcce447c044
  8. Status: Downloaded newer image for traefik:v1.7.2-alpine
  9. docker.io/library/traefik:v1.7.2-alpine
  11. [root@k8s-5-138 redis]# docker tag traefik:v1.7.2-alpine harbor.od.com/public/traefik:v1.7.2
  12. [root@k8s-5-138 redis]# docker push !$
  13. docker push harbor.od.com/public/traefik:v1.7.2
  14. The push refers to repository [harbor.od.com/public/traefik]
  15. a02beb48577f: Pushed 
  16. ca22117205f4: Pushed 
  17. 3563c211d861: Pushed 
  18. df64d3292fd6: Pushed 
  19. v1.7.2: digest: sha256:6115155b261707b642341b065cd3fac2b546559ba035d0262650b3b3bbdd10ea size: 1157

2.3 配置nginx (在k8s-5-141服务器上操作)

  1. [root@k8s-5-141 conf.d]# vim od.com.conf 
  3. upstream default_backend_traefik {
  5.     server 192.168.5.138:81    max_fails=3 fail_timeout=10s;
  7.     server 192.168.5.139:81    max_fails=3 fail_timeout=10s;
  9. }
  11. server {
  13.     server_name *.od.com;
  15.     location / {
  17.         proxy_pass http://default_backend_traefik;
  19.         proxy_set_header Host       $http_host;
  21.         proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
  23.     }
  25. }

2.4 配置DNS(在k8s-5-140服务器上操作)

  1. [root@k8s-5-140 ~]# cd /var/named
  3. # 在 od.com.zone 中添加 traefik           A    192.168.5.141
  4. [root@k8s-5-140 named]# vim od.com.zone 
  6. [root@k8s-5-140 named]# cat od.com.zone 
  7. $ORIGIN od.com.
  9. $TTL 600  ; 10 minutes
  11. @       IN SOA  dns.od.com. dnsadmin.od.com. (
  13.         2021031605 ; serial
  15.         10800      ; refresh (3 hours)
  17.         900        ; retry (15 minutes)
  19.         604800     ; expire (1 week)
  21.         86400      ; minimum (1 day)
  23.         )
  25.         NS   dns.od.com.
  27. $TTL 60 ; 1 minute
  29. dns                A    192.168.5.140
  30. harbor           A    192.168.5.141
  31. k8s-yaml       A    192.168.5.141
  32. traefik           A    192.168.5.141
  34. #重启dns服务
  35. [root@k8s-5-140 named]# systemctl restart named
  38. #验证
  39. [root@k8s-5-140 named]# ping traefik.od.com
  40. PING traefik.od.com (192.168.5.141) 56(84) bytes of data.
  41. 64 bytes from 192.168.5.141 (192.168.5.141): icmp_seq=1 ttl=64 time=0.323 ms
  42. 64 bytes from 192.168.5.141 (192.168.5.141): icmp_seq=2 ttl=64 time=0.209 ms
  44. [root@k8s-5-140 named]# dig -t A traefik.od.com @192.168.5.140 +short
  45. 192.168.5.141

2.5 部署 traefik 服务 (在任意一台k8s节点服务器上)

  1. [root@alice002 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/rbac.yaml
  3. serviceaccount/traefik-ingress-controller created
  5. clusterrole.rbac.authorization.k8s.io/traefik-ingress-controller created
  7. clusterrolebinding.rbac.authorization.k8s.io/traefik-ingress-controller created
  9. [root@alice002 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/ds.yaml
  11. daemonset.extensions/traefik-ingress created
  13. [root@alice002 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/svc.yaml
  15. service/traefik-ingress-service created
  17. [root@alice002 ~]# kubectl apply -f http://k8s-yaml.od.com/traefik/ingress.yaml
  19. ingress.extensions/traefik-web-ui created

2.6 部署dashboard

  1. #下载镜像
  2. [root@k8s-5-138 pod_template]# docker pull k8scn/kubernetes-dashboard-amd64:v1.8.3
  3. v1.8.3: Pulling from k8scn/kubernetes-dashboard-amd64
  4. a4026007c47e: Pull complete 
  5. Digest: sha256:ebc993303f8a42c301592639770bd1944d80c88be8036e2d4d0aa116148264ff
  6. Status: Downloaded newer image for k8scn/kubernetes-dashboard-amd64:v1.8.3
  7. docker.io/k8scn/kubernetes-dashboard-amd64:v1.8.3
  9. [root@k8s-5-138 pod_template]# docker tag k8scn/kubernetes-dashboard-amd64:v1.8.3 harbor.od.com/public/dashboard:v1.8.3
  10. [root@k8s-5-138 pod_template]# docker push !$ 
  11. docker push harbor.od.com/public/dashboard:v1.8.3 
  12. The push refers to repository [harbor.od.com/public/dashboard]
  13. 23ddb8cbb75a: Pushed 
  14. v1.8.3: digest: sha256:ebc993303f8a42c301592639770bd1944d80c88be8036e2d4d0aa116148264ff size: 529
  16. #创建dashboard定义文件(在k8s-5-141服务器上操作)
  17. [root@k8s-5-141 k8s-yaml]# mkdir dashboard
  18. [root@k8s-5-141 k8s-yaml]# cd dashboard
  20. # 1. deployment.yaml
  21. [root@k8s-5-141 dashboard]# vim deployment.yaml
  23. apiVersion: apps/v1
  24. kind: Deployment
  25. metadata:
  26.   name: kubernetes-dashboard
  27.   namespace: kube-system
  28.   labels:
  29.     k8s-app: kubernetes-dashboard
  30.     kubernetes.io/cluster-service: "true"
  31.     addonmanager.kubernetes.io/mode: Reconcile
  32. spec:
  33.   selector:
  34.     matchLabels:
  35.       k8s-app: kubernetes-dashboard
  36.   template:
  37.     metadata:
  38.       labels:
  39.         k8s-app: kubernetes-dashboard
  40.       annotations:
  41.         scheduler.alpha.kubernetes.io/critical-pod: ''
  42.     spec:
  43.       priorityClassName: system-cluster-critical
  44.       containers:
  45.       - name: kubernetes-dashboard
  46.         image: harbor.od.com/public/dashboard:v1.18.3
  47.         resources:
  48.           limits:
  49.             cpu: 100m
  50.             memory: 300Mi
  51.           requests:
  52.             cpu: 50m
  53.             memory: 100Mi
  54.         ports:
  55.         - containerPort: 8443
  56.           protocol: TCP
  57.         args:
  58.           # PLATFORM-SPECIFIC ARGS HERE
  59.           - --auto-generate-certificates
  60.         volumeMounts:
  61.         - name: tmp-volume
  62.           mountPath: /tmp
  63.         livenessProbe:
  64.           httpGet:
  65.             scheme: HTTPS
  66.             path: /
  67.             port: 8443
  68.           initialDelaySeconds: 30
  69.           timeoutSeconds: 30
  70.       volumes:
  71.       - name: tmp-volume
  72.         emptyDir: {}
  73.       serviceAccountName: kubernetes-dashboard-admin
  74.       tolerations:
  75.       - key: "CriticalAddonsOnly"
  76.         operator: "Exists"
  77.       imagePullSecrets:
  78.       - name: harbor
  80. # ingress.yaml
  81. [root@k8s-5-141 dashboard]# vim ingress.yaml
  83. apiVersion: extensions/v1beta1
  84. kind: Ingress
  85. metadata:
  86.   name: kubernetes-dashboard
  87.   namespace: kube-system
  88.   annotations:
  89.     kubernetes.io/ingress.class: traefik
  90. spec:
  91.   rules:
  92.   - host: dashboard.od.com
  93.   - host: dashboard.grep.pro
  94.     http:
  95.       paths:
  96.       - backend:
  97.           serviceName: kubernetes-dashboard
  98.           servicePort: 443
  101. # 3. rbac.yaml
  102. [root@k8s-5-141 dashboard]# vim rbac.yaml
  103. apiVersion: v1
  104. kind: ServiceAccount
  105. metadata:
  106.   labels:
  107.     k8s-app: kubernetes-dashboard
  108.     addonmanager.kubernetes.io/mode: Reconcile
  109.   name: kubernetes-dashboard-admin
  110.   namespace: kube-system
  111. ---
  112. apiVersion: rbac.authorization.k8s.io/v1
  113. kind: ClusterRoleBinding
  114. metadata:
  115.   name: kubernetes-dashboard-admin
  116.   namespace: kube-system
  117.   labels:
  118.     k8s-app: kubernetes-dashboard
  119.     addonmanager.kubernetes.io/mode: Reconcile
  120. roleRef:
  121.   apiGroup: rbac.authorization.k8s.io
  122.   kind: ClusterRole
  123.   name: cluster-admin
  124. subjects:
  125. - kind: ServiceAccount
  126.   name: kubernetes-dashboard-admin
  127.   namespace: kube-system
  130. # 4. svc.yaml
  131. [root@k8s-5-141 dashboard]# vim svc.yaml
  133. apiVersion: v1
  134. kind: Service
  135. metadata:
  136.   name: kubernetes-dashboard
  137.   namespace: kube-system
  138.   labels:
  139.     k8s-app: kubernetes-dashboard
  140.     kubernetes.io/cluster-service: "true"
  141.     addonmanager.kubernetes.io/mode: Reconcile
  142. spec:
  143.   selector:
  144.     k8s-app: kubernetes-dashboard
  145.   ports:
  146.   - port: 443
  147.     targetPort: 8443
  150. #部署dashboard到k8s (切换到任意一个K8s运算节点上操作)
  152. [root@k8s-5-138 dashboard]# kubectl apply -f http://k8s-yaml.od.com/dashboard/rbac.yaml
  153. serviceaccount/kubernetes-dashboard-admin created
  154. clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-admin created
  156. [root@k8s-5-138 dashboard]# kubectl apply -f http://k8s-yaml.od.com/dashboard/deployment.yaml
  157. deployment.apps/kubernetes-dashboard created
  159. [root@k8s-5-138 dashboard]# kubectl apply -f http://k8s-yaml.od.com/dashboard/svc.yaml
  160. service/kubernetes-dashboard created
  162. [root@k8s-5-138 dashboard]# kubectl apply -f http://k8s-yaml.od.com/dashboard/ingress.yaml
  163. ingress.extensions/kubernetes-dashboard created
  165. [root@k8s-5-138 dashboard]# kubectl get all -n kube-system |grep dashboard
  166. pod/kubernetes-dashboard-7c55767659-mpjdw   1/1     Running   0          2m50s
  167. service/kubernetes-dashboard      ClusterIP   192.168.209.29   <none>        443/TCP                  2m19s
  168. deployment.apps/kubernetes-dashboard   1/1     1            1           2m50s
  169. replicaset.apps/kubernetes-dashboard-7c55767659   1         1         1       2m50s

2.7 设置DNS

  2. # 增加  dashboard          A    192.168.5.141
  3. [root@k8s-5-140 named]# vim /var/named/od.com.zone
  5. $ORIGIN od.com.
  7. $TTL 600  ; 10 minutes
  9. @       IN SOA  dns.od.com. dnsadmin.od.com. (
  11.         2021031605 ; serial
  13.         10800      ; refresh (3 hours)
  15.         900        ; retry (15 minutes)
  17.         604800     ; expire (1 week)
  19.         86400      ; minimum (1 day)
  21.         )
  23.         NS   dns.od.com.
  25. $TTL 60 ; 1 minute
  27. dns                A    192.168.5.140
  28. harbor             A    192.168.5.141
  29. k8s-yaml           A    192.168.5.141
  30. traefik            A    192.168.5.141
  31. dashboard          A    192.168.5.141
  33. [root@k8s-5-140 named]# systemctl restart named
  35. [root@k8s-5-140 named]# dig  -t A dashboard.od.com @192.168.5.140 +short
  36. 192.168.5.141
  38. [root@k8s-5-138 dashboard]# dig -t A dashboard.od.com @192.168.0.2 +short
  39. 192.168.5.141

2.8 签发证书
在k8s-5-141服务器上操作

  1. [root@k8s-5-141 dashboard]# cd /opt/certs/
  2. [root@k8s-5-141 certs]# mkdir dashboard-cert
  3. [root@k8s-5-141 /]# cd /opt/certs/dashboard-cert/
  5. [root@k8s-5-141 /]# cat > dashboard-csr.json <<EOF
  6. {
  7.    "CN": "Dashboard",
  8.    "hosts": [],
  9.    "key": {
  10.        "algo": "rsa",
  11.        "size": 2048
  12.    },
  13.    "names": [
  14.        {
  15.            "C": "CN",
  16.            "L": "ShenZhen",
  17.            "ST": "GuangDong",
  18.            "O": "batar",
  19.            "OU": "batar-zhonggu"
  20.        }
  21.    ]
  22. }
  23. EOF
  24. [root@alice001 certs]# openssl req -new -key dashboard.od.com.key -out dashboard.od.com.csr -subj "/CN=dashboard.od.com/C=CN/ST=GuangDong/L=ShenZhen/O=batar/OU=batar-zhonggu"
  26. [root@alice001 certs]# openssl x509 -req -in dashboard.od.com.csr -CA ../ca.pem -CAkey ../ca-key.pem -CAcreateserial -out dashboard.od.com.crt -days 3650
  28. Signature ok