8 配置管理

8.1 secret

Secret 解决了密码、token、密钥等敏感数据的配置问题，而不需要把这些敏感数据暴露到镜像或者Pod Spec 中。Secret 可以以Volume 或者环境变量的方式使用
Secret有三种类型

  • Opaque：base64 编码格式的 secret，用来存储密码、密钥等；但数据也可以通过base64 –decode解码得到原始数据，所有加密性很弱。
  • Service Account：用来访问Kubernetes API，由Kubernetes自动创建，并且会自动挂载到Pod的 /run/secrets/kubernetes.io/serviceaccount 目录中。
  • kubernetes.io/dockerconfigjson ： 用来存储私有docker registry的认证信息。
    1. apiVersion: v1
    2. kind: Secret
    3. metadata:
    4. name: mysecret
    5. type: Opaque
    6. data:
    7. username: YWRtaW4=
    8. password: MWYyZDFlMmU2N2Rm
    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4. name: mypod
    5. spec:
    6. containers:
    7. - name: nginx
    8.   image: nginx
    9.   env:
    10.     - name: SECRET_USERNAME
    11.       valueFrom:
    12.         secretKeyRef:
    13.           name: mysecret
    14.           key: username
    15.     - name: SECRET_PASSWORD
    16.       valueFrom:
    17.         secretKeyRef:
    18.           name: mysecret
    19.           key: password
    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4. name: mypod
    5. spec:
    6. containers:
    7. - name: nginx
    8.   image: nginx
    9.   volumeMounts:
    10.   - name: foo
    11.     mountPath: "/etc/foo"
    12.     readOnly: true
    13. volumes:
    14. - name: foo
    15.   secret:
    16.     secretName: mysecret
    image.png
    image.png
    image.png

    8.2 configMap

    存储不加密数据到etcd,让pod以变量或者Volume挂载到容器中，一般的配置文件都可以用这种方式

    8.2.1 以Volume挂载到pod容器中

    1. redis.host=127.0.0.1
    2. redis.port=6379
    3. redis.password=123456
    1. apiVersion: v1
    2. kind: Pod
    3. metadata:
    4. name: mypod
    5. spec:
    6. containers:
    7.   - name: busybox
    8.     image: busybox
    9.     command: [ "/bin/sh","-c","cat /etc/config/redis.properties" ]
    10.     volumeMounts:
    11.     - name: config-volume
    12.       mountPath: /etc/config
    13. volumes:
    14.   - name: config-volume
    15.     configMap:
    16.       name: redis-config
    17. restartPolicy: Never
    image.png

    8.2.2 以变量形式挂载到pod容器中

    ```yaml apiVersion: v1 kind: ConfigMap metadata: name: myconfig namespace: default data: special.level: info special.type: hello
  1. ```yaml
  2. apiVersion: v1
  3. kind: Pod
  4. metadata:
  5.   name: mypod
  6. spec:
  7.   containers:
  8.     - name: busybox
  9.       image: busybox
  10.       command: [ "/bin/sh", "-c", "echo $(LEVEL) $(TYPE)" ]
  11.       env:
  12.         - name: LEVEL
  13.           valueFrom:
  14.             configMapKeyRef:
  15.               name: myconfig
  16.               key: special.level
  17.         - name: TYPE
  18.           valueFrom:
  19.             configMapKeyRef:
  20.               name: myconfig
  21.               key: special.type
  22.   restartPolicy: Never

image.png

9 集群安全机制RBAC

可参考：https://www.cnblogs.com/jhno1/p/15607638.html

10 Ingress

k8s对外通过NodePort,LoadBalance暴露服务，但是有很大缺点。NodePort会每个应用都会占用一个端口；LoadBalance方式最大的缺点则是每个service一个LB又有点浪费和麻烦，并且需要k8s之外的支持
而用ingress 只需要一个NodePort或者一个LoadBalance就可以满足所有service对外服务
image.png

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: ingress-nginx
  5.   labels:
  6.     app.kubernetes.io/name: ingress-nginx
  7.     app.kubernetes.io/part-of: ingress-nginx
  9. ---
  11. kind: ConfigMap
  12. apiVersion: v1
  13. metadata:
  14.   name: nginx-configuration
  15.   namespace: ingress-nginx
  16.   labels:
  17.     app.kubernetes.io/name: ingress-nginx
  18.     app.kubernetes.io/part-of: ingress-nginx
  20. ---
  21. kind: ConfigMap
  22. apiVersion: v1
  23. metadata:
  24.   name: tcp-services
  25.   namespace: ingress-nginx
  26.   labels:
  27.     app.kubernetes.io/name: ingress-nginx
  28.     app.kubernetes.io/part-of: ingress-nginx
  30. ---
  31. kind: ConfigMap
  32. apiVersion: v1
  33. metadata:
  34.   name: udp-services
  35.   namespace: ingress-nginx
  36.   labels:
  37.     app.kubernetes.io/name: ingress-nginx
  38.     app.kubernetes.io/part-of: ingress-nginx
  40. ---
  41. apiVersion: v1
  42. kind: ServiceAccount
  43. metadata:
  44.   name: nginx-ingress-serviceaccount
  45.   namespace: ingress-nginx
  46.   labels:
  47.     app.kubernetes.io/name: ingress-nginx
  48.     app.kubernetes.io/part-of: ingress-nginx
  50. ---
  51. apiVersion: rbac.authorization.k8s.io/v1beta1
  52. kind: ClusterRole
  53. metadata:
  54.   name: nginx-ingress-clusterrole
  55.   labels:
  56.     app.kubernetes.io/name: ingress-nginx
  57.     app.kubernetes.io/part-of: ingress-nginx
  58. rules:
  59.   - apiGroups:
  60.       - ""
  61.     resources:
  62.       - configmaps
  63.       - endpoints
  64.       - nodes
  65.       - pods
  66.       - secrets
  67.     verbs:
  68.       - list
  69.       - watch
  70.   - apiGroups:
  71.       - ""
  72.     resources:
  73.       - nodes
  74.     verbs:
  75.       - get
  76.   - apiGroups:
  77.       - ""
  78.     resources:
  79.       - services
  80.     verbs:
  81.       - get
  82.       - list
  83.       - watch
  84.   - apiGroups:
  85.       - ""
  86.     resources:
  87.       - events
  88.     verbs:
  89.       - create
  90.       - patch
  91.   - apiGroups:
  92.       - "extensions"
  93.       - "networking.k8s.io"
  94.     resources:
  95.       - ingresses
  96.     verbs:
  97.       - get
  98.       - list
  99.       - watch
  100.   - apiGroups:
  101.       - "extensions"
  102.       - "networking.k8s.io"
  103.     resources:
  104.       - ingresses/status
  105.     verbs:
  106.       - update
  108. ---
  109. apiVersion: rbac.authorization.k8s.io/v1beta1
  110. kind: Role
  111. metadata:
  112.   name: nginx-ingress-role
  113.   namespace: ingress-nginx
  114.   labels:
  115.     app.kubernetes.io/name: ingress-nginx
  116.     app.kubernetes.io/part-of: ingress-nginx
  117. rules:
  118.   - apiGroups:
  119.       - ""
  120.     resources:
  121.       - configmaps
  122.       - pods
  123.       - secrets
  124.       - namespaces
  125.     verbs:
  126.       - get
  127.   - apiGroups:
  128.       - ""
  129.     resources:
  130.       - configmaps
  131.     resourceNames:
  132.       # Defaults to "<election-id>-<ingress-class>"
  133.       # Here: "<ingress-controller-leader>-<nginx>"
  134.       # This has to be adapted if you change either parameter
  135.       # when launching the nginx-ingress-controller.
  136.       - "ingress-controller-leader-nginx"
  137.     verbs:
  138.       - get
  139.       - update
  140.   - apiGroups:
  141.       - ""
  142.     resources:
  143.       - configmaps
  144.     verbs:
  145.       - create
  146.   - apiGroups:
  147.       - ""
  148.     resources:
  149.       - endpoints
  150.     verbs:
  151.       - get
  153. ---
  154. apiVersion: rbac.authorization.k8s.io/v1beta1
  155. kind: RoleBinding
  156. metadata:
  157.   name: nginx-ingress-role-nisa-binding
  158.   namespace: ingress-nginx
  159.   labels:
  160.     app.kubernetes.io/name: ingress-nginx
  161.     app.kubernetes.io/part-of: ingress-nginx
  162. roleRef:
  163.   apiGroup: rbac.authorization.k8s.io
  164.   kind: Role
  165.   name: nginx-ingress-role
  166. subjects:
  167.   - kind: ServiceAccount
  168.     name: nginx-ingress-serviceaccount
  169.     namespace: ingress-nginx
  171. ---
  172. apiVersion: rbac.authorization.k8s.io/v1beta1
  173. kind: ClusterRoleBinding
  174. metadata:
  175.   name: nginx-ingress-clusterrole-nisa-binding
  176.   labels:
  177.     app.kubernetes.io/name: ingress-nginx
  178.     app.kubernetes.io/part-of: ingress-nginx
  179. roleRef:
  180.   apiGroup: rbac.authorization.k8s.io
  181.   kind: ClusterRole
  182.   name: nginx-ingress-clusterrole
  183. subjects:
  184.   - kind: ServiceAccount
  185.     name: nginx-ingress-serviceaccount
  186.     namespace: ingress-nginx
  188. ---
  190. apiVersion: apps/v1
  191. kind: Deployment
  192. metadata:
  193.   name: nginx-ingress-controller
  194.   namespace: ingress-nginx
  195.   labels:
  196.     app.kubernetes.io/name: ingress-nginx
  197.     app.kubernetes.io/part-of: ingress-nginx
  198. spec:
  199.   replicas: 1
  200.   selector:
  201.     matchLabels:
  202.       app.kubernetes.io/name: ingress-nginx
  203.       app.kubernetes.io/part-of: ingress-nginx
  204.   template:
  205.     metadata:
  206.       labels:
  207.         app.kubernetes.io/name: ingress-nginx
  208.         app.kubernetes.io/part-of: ingress-nginx
  209.       annotations:
  210.         prometheus.io/port: "10254"
  211.         prometheus.io/scrape: "true"
  212.     spec:
  213.       hostNetwork: true
  214.       # wait up to five minutes for the drain of connections
  215.       terminationGracePeriodSeconds: 300
  216.       serviceAccountName: nginx-ingress-serviceaccount
  217.       nodeSelector:
  218.         kubernetes.io/os: linux
  219.       containers:
  220.         - name: nginx-ingress-controller
  221.           image: lizhenliang/nginx-ingress-controller:0.30.0
  222.           args:
  223.             - /nginx-ingress-controller
  224.             - --configmap=$(POD_NAMESPACE)/nginx-configuration
  225.             - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
  226.             - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
  227.             - --publish-service=$(POD_NAMESPACE)/ingress-nginx
  228.             - --annotations-prefix=nginx.ingress.kubernetes.io
  229.           securityContext:
  230.             allowPrivilegeEscalation: true
  231.             capabilities:
  232.               drop:
  233.                 - ALL
  234.               add:
  235.                 - NET_BIND_SERVICE
  236.             # www-data -> 101
  237.             runAsUser: 101
  238.           env:
  239.             - name: POD_NAME
  240.               valueFrom:
  241.                 fieldRef:
  242.                   fieldPath: metadata.name
  243.             - name: POD_NAMESPACE
  244.               valueFrom:
  245.                 fieldRef:
  246.                   fieldPath: metadata.namespace
  247.           ports:
  248.             - name: http
  249.               containerPort: 80
  250.               protocol: TCP
  251.             - name: https
  252.               containerPort: 443
  253.               protocol: TCP
  254.           livenessProbe:
  255.             failureThreshold: 3
  256.             httpGet:
  257.               path: /healthz
  258.               port: 10254
  259.               scheme: HTTP
  260.             initialDelaySeconds: 10
  261.             periodSeconds: 10
  262.             successThreshold: 1
  263.             timeoutSeconds: 10
  264.           readinessProbe:
  265.             failureThreshold: 3
  266.             httpGet:
  267.               path: /healthz
  268.               port: 10254
  269.               scheme: HTTP
  270.             periodSeconds: 10
  271.             successThreshold: 1
  272.             timeoutSeconds: 10
  273.           lifecycle:
  274.             preStop:
  275.               exec:
  276.                 command:
  277.                   - /wait-shutdown
  279. ---
  281. apiVersion: v1
  282. kind: LimitRange
  283. metadata:
  284.   name: ingress-nginx
  285.   namespace: ingress-nginx
  286.   labels:
  287.     app.kubernetes.io/name: ingress-nginx
  288.     app.kubernetes.io/part-of: ingress-nginx
  289. spec:
  290.   limits:
  291.   - min:
  292.       memory: 90Mi
  293.       cpu: 100m
  294.     type: Container
  1. ---
  2. # http
  3. apiVersion: networking.k8s.io/v1beta1
  4. kind: Ingress
  5. metadata:
  6.   name: example-ingress
  7. spec:
  8.   rules:
  9.   - host: example.ctnrs.com
  10.     http:
  11.       paths:
  12.       - path: /
  13.         backend:
  14.           serviceName: web
  15.           servicePort: 80
  17. ---
  18. # https
  19. apiVersion: networking.k8s.io/v1beta1
  20. kind: Ingress
  21. metadata:
  22.   name: tls-example-ingress
  23. spec:
  24.   tls:
  25.   - hosts:
  26.     - sslexample.ctnrs.com
  27.     secretName: secret-tls
  28.   rules:
  29.     - host: sslexample.ctnrs.com
  30.       http:
  31.         paths:
  32.         - path: /
  33.           backend:
  34.             serviceName: web
  35.             servicePort: 80

image.png

11 Helm

11.1 什么是helm

K8S 上的应用对象，都是由特定的资源描述组成，包括deployment、service 等。都保存各自文件中或者集中写到一个配置文件。然后kubectl apply –f 部署。但是在微服务架构中，服务多达几十个，不管是版本控制还是资源管理都有很大问题。
Helm 是一个Kubernetes 的包管理工具，就像Linux 下的包管理器，如yum/apt 等，可以很方便的将之前打包好的yaml 文件部署到kubernetes 上。

11.2 helm的作用

  1. 使用helm可以把这些yaml作为一个整体管理
  2. 实现yaml高效复用

  3. 使用heml应用级别的版本管理

    11.3 helm几个重要概念

  4. Helm：一个命令行客户端工具，主要用于Kubernetes 应用chart 的创建、打包、发布和管理。

  5. Chart：应用描述，一系列用于描述k8s 资源相关文件的集合。
  6. Release：基于Chart 的部署实体，一个chart 被Helm 运行后将会生成对应的一个release；将在k8s 中创建出真实运行的资源对象。
  7. Repository：用于发布和存储Chart的仓库

    11.4 helm安装(v3版本)

    11.4.1 安装并解压helm

    1. # https://github.com/helm/helm/releases
    2. wget https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz
    3. tar -zxvf helm-v3.0.0-linux-amd64.tar.gz

    image.png

    11.4.2 配置helm镜像仓库

    1. #配置镜像仓库 阿里或者微软
    2. helm repo add stable https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
    3. helm repo add stable http://mirror.azure.cn/kubernetes/charts/
    4. #更新仓库地址
    5. helm repo update
    6. #查看仓库源
    7. helm repo list
    8. #删除仓库源
    9. helm repo remove <name>

    11.5 使用helm快速部署应用

    1. #快速部署一个weave
    2. #查找仓库
    3. helm search repo weave
    4. #安装weave
    5. helm install ui stable/weave-scope
    6. #查看已安装列表
    7. helm list
    8. #修改service type为NodePort对外暴露端口
    9. kubectl edit svc ui-weave-scope
    image.png

image.png
image.png

11.6 自定义Chart

11.6.1 自定义chart demo运行

image.png

  • Chart.yaml：当前chart属性配置信息
  • templates：编写yaml文件放在这个目录中
  • values.yaml：yaml文件可以使用全局变量
  1. #创建自定义chart
  2. helm create mychart
  3. #templates 中创建一个yaml
  4. kubectl create deployment web --image=nginx -o yaml --dry-run > m1.yaml
  5. #执行yaml为了创建service.yaml
  6. kubectl apply -f m1.yaml
  7. #templates中创建service.yaml
  8. kubectl expose deployment web --port=80 --target-port=80 --type=NodePort --dry-run -o yaml >service.yaml
  9. #删除创建的pod
  10. kubectl delete deployment web
  11. #使用helm安装mychat
  12. helm install web mychart/
  13. #更新mychat
  14. helm upgrade web mychart/

11.6.2 chart 模板使用

修改values.yaml
image.png
在templates的yaml文件中使用values.yaml自定义变量
使用通用表达式方式 {{ .Values.变量名称}}
{{ .Release.Name}} ：表示当前版本名称

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   creationTimestamp: null
  5.   labels:
  6.     app: {{ .Values.label}}
  7.   name: {{ .Release.Name}}.deploy
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: {{ .Values.label}}
  13.   strategy: {}
  14.   template:
  15.     metadata:
  16.       creationTimestamp: null
  17.       labels:
  18.         app: {{ .Values.label}}
  19.     spec:
  20.       containers:
  21.       - image: {{ .Values.image}}
  22.         name: nginx
  23.         resources: {}
  24. status: {}
  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   creationTimestamp: null
  5.   labels:
  6.     app: {{ .Values.label}}
  7.   name: {{ .Release.Name}}.svc
  8. spec:
  9.   ports:
  10.   - port: {{ .Values.port}}
  11.     protocol: TCP
  12.     targetPort: 80
  13.   selector:
  14.     app: {{ .Values.label}}
  15.   type: NodePort
  16. status:
  17.   loadBalancer: {}

image.png
image.png

12 持久化存储

pod中存储文件在数据卷，emoptydiy中，pod重启数据就丢失了，所有需要对数据持久化存储

12.1 nfs 网络存储

部署nfs网络步骤

  1. 在一台新的服务器上,node节点上 安装nfs服务端
  2. 设置挂载目录 挂载目录需要在服务器上存在
  3. 在nfs服务器上启动nfs服务
  4. 在k8s集群部署应用使用nfs持久网络存储
    1. #在一台新的服务器上,node节点上 安装nfs服务端
    2. yum install -y nfs-utils
    3. #启动nfs
    4. systemctl start nfs
    5. # 挂载目录
    6. /data/nfs *(rw,sync,no_root_squash,no_subtree_check)
    7. #暴露端口
    8. kubectl expose deployment nginx-dep1 --port=80 --target-port=80 --type=NodePort
    服务器设置挂载目录 挂载目录需要在服务器上存在
    image.png
    1. apiVersion: apps/v1
    2. kind: Deployment
    3. metadata:
    4. name: nginx-dep1
    5. spec:
    6. replicas: 1
    7. selector:
    8.  matchLabels:
    9.    app: nginx
    10. template:
    11.  metadata:
    12.    labels:
    13.      app: nginx
    14.  spec:
    15.    containers:
    16.    - name: nginx
    17.      image: nginx
    18.      volumeMounts:
    19.      - name: wwwroot
    20.        mountPath: /usr/share/nginx/html
    21.      ports:
    22.      - containerPort: 80
    23.    volumes:
    24.      - name: wwwroot
    25.        nfs:
    26.          server: 192.168.19.134
    27.          path: /data/nfs

在服务器端创建一个文件
image.png
进入pod里面查看 会同步进去
image.png

image.png

12.2 pc和pvc

PersistentVolume（PV）：持久化存储，对存储资源进行抽象，对外提供可以调用的地方（生产者）
PersistentVolumeClaim（PVC）：用于调用，不需要关系内部实现细节（消费者）

  1. apiVersion: v1
  2. kind: PersistentVolume
  3. metadata:
  4.   name: my-pv
  5. spec:
  6.   capacity:
  7.     storage: 5Gi
  8.   accessModes:
  9.     - ReadWriteMany
  10.   nfs:
  11.     path: /data/nfs
  12.     server: 192.168.19.134
  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: nginx-dep1
  5. spec:
  6.   replicas: 2
  7.   selector:
  8.     matchLabels:
  9.       app: nginx
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: nginx
  14.     spec:
  15.       containers:
  16.       - name: nginx
  17.         image: nginx
  18.         volumeMounts:
  19.         - name: wwwroot
  20.           mountPath: /usr/share/nginx/html
  21.         ports:
  22.         - containerPort: 80
  23.       volumes:
  24.       - name: wwwroot
  25.         persistentVolumeClaim:
  26.           claimName: my-pvc
  28. ---
  29. apiVersion: v1
  30. kind: PersistentVolumeClaim
  31. metadata:
  32.   name: my-pvc
  33. spec:
  34.   accessModes:
  35.     - ReadWriteMany
  36.   resources:
  37.     requests:
  38.       storage: 5Gi

image.png