1. 编写的 yml 文件部署，报错，识别不了容器参数

1.1 错误信息：

  1. Error from server (BadRequest): error when creating "apollo.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 8, error found in #10 byte of ...|,"value":8070},{"nam|..., bigger context ...|ntainers":[{"env":[{"name":"SERVER_PORT","value":8070},{"name":"SPRING_DATASOURCE_URL","value":"jdbc|...
  2. Error from server (BadRequest): error when creating "apollo.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 8, error found in #10 byte of ...|,"value":8080},{"nam|..., bigger context ...|ntainers":[{"env":[{"name":"SERVER_PORT","value":8080},{"name":"SPRING_DATASOURCE_URL","value":"jdbc|...
  3. Error from server (BadRequest): error when creating "apollo.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 8, error found in #10 byte of ...|,"value":8090},{"nam|..., bigger context ...|ntainers":[{"env":[{"name":"SERVER_PORT","value":8090},{"name":"SPRING_DATASOURCE_URL","value":"jdbc|...
  4. [root@saaspy ~]#

1.2 yml 配置

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: apollo
  5. ---
  6. apiVersion: v1
  7. data:
  8.   .dockerconfigjson: eyJhdXRocyI6eyJodHRwOi8vc2NkaGIuZjMzMjIubmV0OjMyMDAwIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6ImhlYmVuIzEyMzQ1IiwiZW1haWwiOiIzNTgxMDk0NjZAcXEuY29tIiwiYXV0aCI6IllXUnRhVzQ2YUdWaVpXNGpNVEl6TkRVPSJ9fX0=
  9. kind: Secret
  10. metadata:
  11.   name: secret-name
  12.   namespace: apollo
  13. type: kubernetes.io/dockerconfigjson
  14. ---
  15. apiVersion: apps/v1
  16. kind: Deployment
  17. metadata:
  18.   name: apollo-portal
  19.   namespace: apollo
  20. spec:
  21.   selector:
  22.     matchLabels:
  23.       app: apollo-portal
  24.   template:
  25.     metadata:
  26.       labels:
  27.         app: apollo-portal
  28.     spec:
  29.       imagePullSecrets:
  30.         - name: secret-name
  31.       containers:
  32.         - name: apollo-portal
  33.           image: scdhb.f3322.net:32000/apollo/apollo-portal:v1.8.2
  34.           imagePullPolicy: Always
  35.           ports:
  36.             - containerPort: 8070
  37.           env:
  38.             - name: SERVER_PORT
  39.               value: 8070
  40.             - name: SPRING_DATASOURCE_URL
  41.               value: jdbc:mysql://apollo-mysql:3306/ApolloPortalDB?sslMode=DISABLED&serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false
  42.             - name: SPRING_DATASOURCE_USERNAME
  43.               value: root
  44.             - name: SPRING_DATASOURCE_PASSWORD
  45.               value: root
  46.             - name: DEV_META
  47.               value: http://apollo-config:8080/
  48. ---
  49. apiVersion: v1
  50. kind: Service
  51. metadata:
  52.   name: apollo-portal
  53.   namespace: apollo
  54. spec:
  55.   type: NodePort
  56.   selector:
  57.     app: apollo-portal
  58.   ports:
  59.     - name: http
  60.       port: 8070
  61. ---
  62. apiVersion: apps/v1
  63. kind: Deployment
  64. metadata:
  65.   name: apollo-config
  66.   namespace: apollo
  67. spec:
  68.   selector:
  69.     matchLabels:
  70.       app: apollo-config
  71.   template:
  72.     metadata:
  73.       labels:
  74.         app: apollo-config
  75.     spec:
  76.       imagePullSecrets:
  77.         - name: secret-name
  78.       containers:
  79.         - name: apollo-config
  80.           image: scdhb.f3322.net:32000/apollo/apollo-config:v1.8.1
  81.           imagePullPolicy: Always
  82.           ports:
  83.             - containerPort: 8080
  84.           env:
  85.             - name: SERVER_PORT
  86.               value: 8080
  87.             - name: SPRING_DATASOURCE_URL
  88.               value: jdbc:mysql://apollo-mysql:3306/ApolloConfigDB?sslMode=DISABLED&serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false
  89.             - name: SPRING_DATASOURCE_USERNAME
  90.               value: root
  91.             - name: SPRING_DATASOURCE_PASSWORD
  92.               value: root
  93. ---
  94. apiVersion: v1
  95. kind: Service
  96. metadata:
  97.   name: apollo-config
  98.   namespace: apollo
  99. spec:
  100.   type: NodePort
  101.   selector:
  102.     app: apollo-config
  103.   ports:
  104.     - name: http
  105.       port: 8080
  106. ---
  107. apiVersion: apps/v1
  108. kind: Deployment
  109. metadata:
  110.   name: apollo-admin
  111.   namespace: apollo
  112. spec:
  113.   selector:
  114.     matchLabels:
  115.       app: apollo-admin
  116.   template:
  117.     metadata:
  118.       labels:
  119.         app: apollo-admin
  120.     spec:
  121.       imagePullSecrets:
  122.         - name: secret-name
  123.       containers:
  124.         - name: apollo-admin
  125.           image: scdhb.f3322.net:32000/apollo/apollo-admin:v1.8.1
  126.           imagePullPolicy: Always
  127.           ports:
  128.             - containerPort: 8090
  129.           env:
  130.             - name: SERVER_PORT
  131.               value: 8090
  132.             - name: SPRING_DATASOURCE_URL
  133.               value: jdbc:mysql://apollo-mysql:3306/ApolloConfigDB?sslMode=DISABLED&serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false
  134.             - name: SPRING_DATASOURCE_USERNAME
  135.               value: root
  136.             - name: SPRING_DATASOURCE_PASSWORD
  137.               value: root
  138. ---
  139. apiVersion: v1
  140. kind: Service
  141. metadata:
  142.   name: apollo-admin
  143.   namespace: apollo
  144. spec:
  145.   type: NodePort
  146.   selector:
  147.     app: apollo-admin
  148.   ports:
  149.     - name: http
  150.       port: 8090
  151. ---
  152. apiVersion: apps/v1
  153. kind: Deployment
  154. metadata:
  155.   name: apollo-mysql
  156.   namespace: apollo
  157. spec:
  158.   selector:
  159.     matchLabels:
  160.       app: apollo-mysql
  161.   template:
  162.     metadata:
  163.       labels:
  164.         app: apollo-mysql
  165.     spec:
  166.       imagePullSecrets:
  167.         - name: secret-name
  168.       containers:
  169.         - name: apollo-mysql
  170.           image: scdhb.f3322.net:32000/base-soft/mysql:release
  171.           imagePullPolicy: Always
  172.           ports:
  173.             - containerPort: 3306
  174.           env:
  175.             - name: MYSQL_ROOT_PASSWORD
  176.               value: root
  178. ---
  179. apiVersion: v1
  180. kind: Service
  181. metadata:
  182.   name: apollo-mysql
  183.   namespace: apollo
  184. spec:
  185.   type: NodePort
  186.   selector:
  187.     app: apollo-mysql
  188.   ports:
  189.     - name: http
  190.       port: 3306

1.3 部署结果

  1. [root@saaspy ~]# kubectl delete ns apollo
  2. namespace "apollo" deleted
  3. [root@saaspy ~]#
  4. [root@saaspy ~]#
  5. [root@saaspy ~]# kubectl apply -f apollo.yaml
  6. namespace/apollo created
  7. secret/secret-name created
  8. service/apollo-portal created
  9. service/apollo-config created
  10. service/apollo-admin created
  11. deployment.apps/apollo-mysql created
  12. service/apollo-mysql created
  13. Error from server (BadRequest): error when creating "apollo.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 8, error found in #10 byte of ...|,"value":8070},{"nam|..., bigger context ...|ntainers":[{"env":[{"name":"SERVER_PORT","value":8070},{"name":"SPRING_DATASOURCE_URL","value":"jdbc|...
  14. Error from server (BadRequest): error when creating "apollo.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 8, error found in #10 byte of ...|,"value":8080},{"nam|..., bigger context ...|ntainers":[{"env":[{"name":"SERVER_PORT","value":8080},{"name":"SPRING_DATASOURCE_URL","value":"jdbc|...
  15. Error from server (BadRequest): error when creating "apollo.yaml": Deployment in version "v1" cannot be handled as a Deployment: v1.Deployment.Spec: v1.DeploymentSpec.Template: v1.PodTemplateSpec.Spec: v1.PodSpec.Containers: []v1.Container: v1.Container.Env: []v1.EnvVar: v1.EnvVar.Value: ReadString: expects " or n, but found 8, error found in #10 byte of ...|,"value":8090},{"nam|..., bigger context ...|ntainers":[{"env":[{"name":"SERVER_PORT","value":8090},{"name":"SPRING_DATASOURCE_URL","value":"jdbc|...
  16. [root@saaspy ~]#

1.4 原因，及解决办法

1.4.1 原因

如果 Deployment 中使用了 env，并且 env 中有一些值是数字，就会抛出这个错误，解决办法就是将数字添加引号，类似下面截图中内容。
所以，在 deployment 中将数字类型的参数使用 “” 英文引号包裹起来。

1.4.2 解决

修改 yaml 文件

  1. apiVersion: v1
  2. kind: Namespace
  3. metadata:
  4.   name: apollo
  5. ---
  6. apiVersion: v1
  7. data:
  8.   .dockerconfigjson: eyJhdXRocyI6eyJodHRwOi8vc2NkaGIuZjMzMjIubmV0OjMyMDAwIjp7InVzZXJuYW1lIjoiYWRtaW4iLCJwYXNzd29yZCI6ImhlYmVuIzEyMzQ1IiwiZW1haWwiOiIzNTgxMDk0NjZAcXEuY29tIiwiYXV0aCI6IllXUnRhVzQ2YUdWaVpXNGpNVEl6TkRVPSJ9fX0=
  9. kind: Secret
  10. metadata:
  11.   name: secret-name
  12.   namespace: apollo
  13. type: kubernetes.io/dockerconfigjson
  14. ---
  15. apiVersion: apps/v1
  16. kind: Deployment
  17. metadata:
  18.   name: apollo-portal
  19.   namespace: apollo
  20. spec:
  21.   selector:
  22.     matchLabels:
  23.       app: apollo-portal
  24.   template:
  25.     metadata:
  26.       labels:
  27.         app: apollo-portal
  28.     spec:
  29.       imagePullSecrets:
  30.         - name: secret-name
  31.       containers:
  32.         - name: apollo-portal
  33.           image: scdhb.f3322.net:32000/apollo/apollo-portal:v1.8.2
  34.           imagePullPolicy: Always
  35.           ports:
  36.             - containerPort: 8070
  37.           env:
  38.             - name: SERVER_PORT
  39.               value: "8070"
  40.             - name: SPRING_DATASOURCE_URL
  41.               value: "jdbc:mysql://apollo-mysql:3306/ApolloPortalDB?sslMode=DISABLED&serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false"
  42.             - name: SPRING_DATASOURCE_USERNAME
  43.               value: root
  44.             - name: SPRING_DATASOURCE_PASSWORD
  45.               value: root
  46.             - name: DEV_META
  47.               value: http://apollo-config:8080/
  48. ---
  49. apiVersion: v1
  50. kind: Service
  51. metadata:
  52.   name: apollo-portal
  53.   namespace: apollo
  54. spec:
  55.   type: NodePort
  56.   selector:
  57.     app: apollo-portal
  58.   ports:
  59.     - name: http
  60.       port: 8070
  61. ---
  62. apiVersion: apps/v1
  63. kind: Deployment
  64. metadata:
  65.   name: apollo-config
  66.   namespace: apollo
  67. spec:
  68.   selector:
  69.     matchLabels:
  70.       app: apollo-config
  71.   template:
  72.     metadata:
  73.       labels:
  74.         app: apollo-config
  75.     spec:
  76.       imagePullSecrets:
  77.         - name: secret-name
  78.       containers:
  79.         - name: apollo-config
  80.           image: scdhb.f3322.net:32000/apollo/apollo-config:v1.8.1
  81.           imagePullPolicy: Always
  82.           ports:
  83.             - containerPort: 8080
  84.           env:
  85.             - name: SERVER_PORT
  86.               value: "8080"
  87.             - name: SPRING_DATASOURCE_URL
  88.               value: "jdbc:mysql://apollo-mysql:3306/ApolloConfigDB?sslMode=DISABLED&serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false"
  89.             - name: SPRING_DATASOURCE_USERNAME
  90.               value: root
  91.             - name: SPRING_DATASOURCE_PASSWORD
  92.               value: root
  93. ---
  94. apiVersion: v1
  95. kind: Service
  96. metadata:
  97.   name: apollo-config
  98.   namespace: apollo
  99. spec:
  100.   type: NodePort
  101.   selector:
  102.     app: apollo-config
  103.   ports:
  104.     - name: http
  105.       port: 8080
  106. ---
  107. apiVersion: apps/v1
  108. kind: Deployment
  109. metadata:
  110.   name: apollo-admin
  111.   namespace: apollo
  112. spec:
  113.   selector:
  114.     matchLabels:
  115.       app: apollo-admin
  116.   template:
  117.     metadata:
  118.       labels:
  119.         app: apollo-admin
  120.     spec:
  121.       imagePullSecrets:
  122.         - name: secret-name
  123.       containers:
  124.         - name: apollo-admin
  125.           image: scdhb.f3322.net:32000/apollo/apollo-admin:v1.8.1
  126.           imagePullPolicy: Always
  127.           ports:
  128.             - containerPort: 8090
  129.           env:
  130.             - name: SERVER_PORT
  131.               value: "8090"
  132.             - name: SPRING_DATASOURCE_URL
  133.               value: "jdbc:mysql://apollo-mysql:3306/ApolloConfigDB?sslMode=DISABLED&serverTimezone=Asia/Shanghai&characterEncoding=utf8&useSSL=false"
  134.             - name: SPRING_DATASOURCE_USERNAME
  135.               value: root
  136.             - name: SPRING_DATASOURCE_PASSWORD
  137.               value: root
  138. ---
  139. apiVersion: v1
  140. kind: Service
  141. metadata:
  142.   name: apollo-admin
  143.   namespace: apollo
  144. spec:
  145.   type: NodePort
  146.   selector:
  147.     app: apollo-admin
  148.   ports:
  149.     - name: http
  150.       port: 8090
  151. ---
  152. apiVersion: apps/v1
  153. kind: Deployment
  154. metadata:
  155.   name: apollo-mysql
  156.   namespace: apollo
  157. spec:
  158.   selector:
  159.     matchLabels:
  160.       app: apollo-mysql
  161.   template:
  162.     metadata:
  163.       labels:
  164.         app: apollo-mysql
  165.     spec:
  166.       imagePullSecrets:
  167.         - name: secret-name
  168.       containers:
  169.         - name: apollo-mysql
  170.           image: scdhb.f3322.net:32000/base-soft/mysql:release
  171.           imagePullPolicy: Always
  172.           ports:
  173.             - containerPort: 3306
  174.           env:
  175.             - name: MYSQL_ROOT_PASSWORD
  176.               value: root
  178. ---
  179. apiVersion: v1
  180. kind: Service
  181. metadata:
  182.   name: apollo-mysql
  183.   namespace: apollo
  184. spec:
  185.   type: NodePort
  186.   selector:
  187.     app: apollo-mysql
  188.   ports:
  189.     - name: http
  190.       port: 3306

运行结果：

  1. [root@saaspy ~]# kubectl apply -f apollo.yml
  2. namespace/apollo created
  3. secret/secret-name created
  4. deployment.apps/apollo-portal created
  5. service/apollo-portal created
  6. deployment.apps/apollo-config created
  7. service/apollo-config created
  8. deployment.apps/apollo-admin created
  9. service/apollo-admin created
  10. deployment.apps/apollo-mysql created
  11. service/apollo-mysql created
  12. [root@saaspy ~]#
  13. [root@saaspy ~]#
  14. [root@saaspy ~]# kubectl get pod,svc -n apollo
  15. NAME                                 READY   STATUS             RESTARTS   AGE
  16. pod/apollo-admin-67b5487986-4n66r    0/1     CrashLoopBackOff   4          4m6s
  17. pod/apollo-config-585c468c59-s6ls8   0/1     CrashLoopBackOff   4          4m7s
  18. pod/apollo-mysql-d579db4f8-mbzgs     1/1     Running            0          4m6s
  19. pod/apollo-portal-b8697d789-x8tth    1/1     Running            5          4m7s
  21. NAME                    TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
  22. service/apollo-admin    NodePort   172.31.229.214   <none>        8090:31442/TCP   4m6s
  23. service/apollo-config   NodePort   172.31.207.54    <none>        8080:30478/TCP   4m6s
  24. service/apollo-mysql    NodePort   172.31.248.214   <none>        3306:31093/TCP   4m6s
  25. service/apollo-portal   NodePort   172.31.61.50     <none>        8070:31194/TCP   4m7s
  26. [root@saaspy ~]#

说明，不要关注 pod 的状态失败，因为还未建立数据库。

2. k8s 建立的 mysql 远程访问问题

我的k8s是个远程环境，不知道同事怎么设置的，只可以运行部分 kubectl 的命令，有的直接报错 timeout，尤其是在 kubectl exec 命令时（后面会提到）。

2.1 mysql yaml 配置，不可远程访问数据库

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: apollo-mysql
  5.   namespace: apollo
  6. spec:
  7.   selector:
  8.     matchLabels:
  9.       app: apollo-mysql
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: apollo-mysql
  14.     spec:
  15.       imagePullSecrets:
  16.         - name: secret-name
  17.       containers:
  18.         - name: apollo-mysql
  19.           image: scdhb.f3322.net:32000/base-soft/mysql:release
  20.           imagePullPolicy: Always
  21.           ports:
  22.             - containerPort: 3306
  23.           env:
  24.             - name: MYSQL_ROOT_PASSWORD
  25.               value: root
  27. ---
  28. apiVersion: v1
  29. kind: Service
  30. metadata:
  31.   name: apollo-mysql
  32.   namespace: apollo
  33. spec:
  34.   type: NodePort
  35.   selector:
  36.     app: apollo-mysql
  37.   ports:
  38.     - name: http
  39.       port: 3306

mysql 能够正常运行，可是远程连接时候报错 mysql 10060 错误，知道没有权限访问，毕竟 root 是默认在 localhost 主机访问的。那么就需要修改 root@localhost（或者新建用户），然而本次不能够通过 kubectl exec 进入容器内部，于是乎想到 kubectl exec 远程执行命令（或者容器启动部署时，执行命令）。

2.2 mysql 部署时指定 command，修改 root@localhost（不行）

命令，command: [“mysql -uroot -p$(MYSQL_ROOT_PASSWORD)”, “RENAME USER root@localhost TO root@%“]
mysql yaml 配置

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: apollo-mysql
  5.   namespace: apollo
  6. spec:
  7.   selector:
  8.     matchLabels:
  9.       app: apollo-mysql
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: apollo-mysql
  14.     spec:
  15.       imagePullSecrets:
  16.         - name: secret-name
  17.       containers:
  18.         - name: apollo-mysql
  19.           image: scdhb.f3322.net:32000/base-soft/mysql:release
  20.           imagePullPolicy: Always
  21.           ports:
  22.             - containerPort: 3306
  23.           env:
  24.             - name: MYSQL_ROOT_PASSWORD
  25.               value: root
  26.           command: ["mysql -uroot -p$(MYSQL_ROOT_PASSWORD)", "RENAME USER `root`@`localhost` TO `root`@`%`"]
  28. ---
  29. apiVersion: v1
  30. kind: Service
  31. metadata:
  32.   name: apollo-mysql
  33.   namespace: apollo
  34. spec:
  35.   type: NodePort
  36.   selector:
  37.     app: apollo-mysql
  38.   ports:
  39.     - name: http
  40.       port: 3306

结果让人很失望，没有找到 mysql 命令（尚未分析是不是自己的 command 没有写对）
结果：

  1. [root@saaspy ~]# kubectl describe pod/apollo-mysql-547d7cdf56-qszn5 -n apollo
  2. Name:           apollo-mysql-547d7cdf56-qszn5
  3. Namespace:      apollo
  4. Priority:       0
  5. Node:           cn-shanghai.192.168.0.91/192.168.0.91
  6. Start Time:     Sat, 07 Aug 2021 10:47:52 +0800
  7. Labels:         app=apollo-mysql
  8.                 pod-template-hash=547d7cdf56
  9. Annotations:    kubernetes.io/psp: ack.privileged
  10. Status:         Running
  11. IP:             192.168.0.228
  12. Controlled By:  ReplicaSet/apollo-mysql-547d7cdf56
  13. Containers:
  14.   apollo-mysql:
  15.     Container ID:  docker://98cb6f79000a7d0c39a7b6aafbcd0e7ad1dffa9c4fd20b7dbfd7a23bd3f1a048
  16.     Image:         scdhb.f3322.net:32000/base-soft/mysql:release
  17.     Image ID:      docker-pullable://scdhb.f3322.net:32000/base-soft/mysql@sha256:4476fa792a8cd534386c23ebe70b4d931f72ad7bb60db888d3a188a6287202f2
  18.     Port:          3306/TCP
  19.     Host Port:     0/TCP
  20.     Command:
  21.       mysql -uroot -p$(MYSQL_ROOT_PASSWORD)
  22.       RENAME USER `root`@`localhost` TO `root`@`%`
  23.     State:          Waiting
  24.       Reason:       CrashLoopBackOff
  25.     Last State:     Terminated
  26.       Reason:       ContainerCannotRun
  27.       Message:      OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"mysql -uroot -proot\": executable file not found in $PATH": unknown
  28.       Exit Code:    127
  29.       Started:      Sat, 07 Aug 2021 10:48:10 +0800
  30.       Finished:     Sat, 07 Aug 2021 10:48:10 +0800
  31.     Ready:          False
  32.     Restart Count:  2
  33.     Environment:
  34.       MYSQL_ROOT_PASSWORD:  root
  35.     Mounts:
  36.       /var/run/secrets/kubernetes.io/serviceaccount from default-token-tvpvz (ro)
  37. Conditions:
  38.   Type              Status
  39.   Initialized       True
  40.   Ready             False
  41.   ContainersReady   False
  42.   PodScheduled      True
  43. Volumes:
  44.   default-token-tvpvz:
  45.     Type:        Secret (a volume populated by a Secret)
  46.     SecretName:  default-token-tvpvz
  47.     Optional:    false
  48. QoS Class:       BestEffort
  49. Node-Selectors:  <none>
  50. Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
  51.                  node.kubernetes.io/unreachable:NoExecute for 300s
  52. Events:
  53.   Type     Reason          Age               From                               Message
  54.   ----     ------          ----              ----                               -------
  55.   Normal   Scheduled       45s               default-scheduler                  Successfully assigned apollo/apollo-mysql-547d7cdf56-qszn5 to cn-shanghai.192.168.0.91
  56.   Normal   AllocIPSucceed  43s               terway-daemon                      Alloc IP 192.168.0.228/24 for Pod
  57.   Warning  BackOff         16s               kubelet, cn-shanghai.192.168.0.91  Back-off restarting failed container
  58.   Normal   Pulling         3s (x4 over 43s)  kubelet, cn-shanghai.192.168.0.91  Pulling image "scdhb.f3322.net:32000/base-soft/mysql:release"
  59.   Normal   Pulled          2s (x4 over 42s)  kubelet, cn-shanghai.192.168.0.91  Successfully pulled image "scdhb.f3322.net:32000/base-soft/mysql:release"
  60.   Normal   Created         2s (x4 over 42s)  kubelet, cn-shanghai.192.168.0.91  Created container apollo-mysql
  61.   Warning  Failed          2s (x4 over 42s)  kubelet, cn-shanghai.192.168.0.91  Error: failed to start container "apollo-mysql": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"mysql -uroot -proot\": executable file not found in $PATH": unknown
  62. [root@saaspy ~]#

Warning Failed 2s (x4 over 42s) kubelet, cn-shanghai.192.168.0.91 Error: failed to start container “apollo-mysql”: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused “exec: \”mysql -uroot -proot\”: executable file not found in $PATH”: unknown

这个修改 root 的命令需要在容器部署启动后（运行后）才能执行，初步怀疑是命令先于容器执行。

那就试试另一种方式： kubectl exec 执行命令。

2.3 mysql 部署完成后，kubectl exec 直接执行命令（不进入容器）

2.3.1 kubectl exec 命令小例

  1. [root@k3s-m1 ~]# kubectl exec -h
  2. Execute a command in a container.
  4. Examples:
  5.   # Get output from running 'date' command from pod mypod, using the first container by default
  6.   kubectl exec mypod -- date
  8.   # Get output from running 'date' command in ruby-container from pod mypod
  9.   kubectl exec mypod -c ruby-container -- date
  11.   # Switch to raw terminal mode, sends stdin to 'bash' in ruby-container from pod mypod
  12.   # and sends stdout/stderr from 'bash' back to the client
  13.   kubectl exec mypod -c ruby-container -i -t -- bash -il
  15.   # List contents of /usr from the first container of pod mypod and sort by modification time.
  16.   # If the command you want to execute in the pod has any flags in common (e.g. -i),
  17.   # you must use two dashes (--) to separate your command's flags/arguments.
  18.   # Also note, do not surround your command and its flags/arguments with quotes
  19.   # unless that is how you would execute it normally (i.e., do ls -t /usr, not "ls -t /usr").
  20.   kubectl exec mypod -i -t -- ls -t /usr
  22.   # Get output from running 'date' command from the first pod of the deployment mydeployment, using the first container
  23. by default
  24.   kubectl exec deploy/mydeployment -- date
  26.   # Get output from running 'date' command from the first pod of the service myservice, using the first container by
  27. default
  28.   kubectl exec svc/myservice -- date
  30. Options:
  31.   -c, --container='': Container name. If omitted, the first container in the pod will be chosen
  32.   -f, --filename=[]: to use to exec into the resource
  33.       --pod-running-timeout=1m0s: The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one
  34. pod is running
  35.   -i, --stdin=false: Pass stdin to the container
  36.   -t, --tty=false: Stdin is a TTY
  38. Usage:
  39.   kubectl exec (POD | TYPE/NAME) [-c CONTAINER] [flags] -- COMMAND [args...] [options]
  41. Use "kubectl options" for a list of global command-line options (applies to all commands).
  42. [root@k3s-m1 ~]#
  43. [root@k3s-m1 ~]# kubectl exec huawei004-mysql-base-77f6856d66-l5jw4 -n huawei004 date
  44. kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
  45. Sat Aug  7 03:12:51 UTC 2021
  46. [root@k3s-m1 ~]#
  47. [root@k3s-m1 ~]# kubectl exec huawei004-mysql-base-77f6856d66-l5jw4 -n huawei004 -- date
  48. Sat Aug  7 03:13:00 UTC 2021
  49. [root@k3s-m1 ~]#

在自己的上执行，报错 timeout

  1. [root@saaspy ~]# kubectl exec apollo-mysql-d579db4f8-vhpwv -n apollo date
  2. Error from server: error dialing backend: dial tcp 192.168.0.91:10250: i/o timeout
  3. [root@saaspy ~]#

唉，还是只有采取部署时的 command，返回去分析错误原因。

2.4 解决

2.4.1 首先说明无法访问，不是 root 账号权限问题

kubectl 部署运行时的 command 属于启动前运行，故而 command 方式是不行的。
后来改变策略，重新部署成功，可以远程访问了，查看 mysql 的 user 表，远程访问是有权限的：

  1. mysql> select Host,User from user;
  2. +-----------+---------------+
  3. | Host      | User          |
  4. +-----------+---------------+
  5. | %         | root          |
  6. | localhost | mysql.session |
  7. | localhost | mysql.sys     |
  8. | localhost | root          |
  9. +-----------+---------------+
  10. 4 rows in set (0.10 sec)
  12. mysql>

2.4.2 改变 service type 为 LoadBalancer，使用 EXTERNAL-IP 访问

主要是改变了 service 的网络type类型 LoadBalancer
mysql yaml 配置（不带 cm（configMap））

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: xh-apollo-mysql2
  5.   namespace: xh-apollo
  6. spec:
  7.   selector:
  8.     matchLabels:
  9.       app: xh-apollo-mysql2
  10.   template:
  11.     metadata:
  12.       labels:
  13.         app: xh-apollo-mysql2
  14.     spec:
  15.       imagePullSecrets:
  16.         - name: secret-name
  17.       containers:
  18.         - name: xh-apollo-mysql2
  19.           image: scdhb.f3322.net:32000/base-soft/mysql:release
  20.           imagePullPolicy: Always
  21.           ports:
  22.             - containerPort: 3306
  23.           env:
  24.             - name: MYSQL_ROOT_PASSWORD
  25.               value: root
  26. #          command: ["RENAME USER `root`@`localhost` TO `root`@`%`"]
  28. ---
  29. apiVersion: v1
  30. kind: Service
  31. metadata:
  32.   name: xh-apollo-mysql2
  33.   namespace: xh-apollo
  34. spec:
  35.   type: LoadBalancer
  36.   selector:
  37.     app: xh-apollo-mysql2
  38.   ports:
  39.     - name: mysql-svc-port
  40.       port: 30084
  41.       targetPort: 3306

mysql yaml 配置（带 cm（configMap））

  1. ---
  2. apiVersion: v1
  3. kind: Service
  4. metadata:
  5.   name: xh-apollo-admin
  6.   namespace: xh-apollo
  7. spec:
  8.   type: NodePort
  9.   selector:
  10.     app: xh-apollo-admin
  11.   ports:
  12.     - name: http
  13.       port: 8090
  14. ---
  15. apiVersion: v1
  16. kind: ConfigMap
  17. metadata:
  18.   name: xh-apollo-mysql-cm
  19.   namespace: xh-apollo
  20.   labels:
  21.     app: xh-apollo-mysql-cm
  22. data:
  23.   my.cnf: |-
  24.     [client]
  25.     default-character-set=utf8mb4
  26.     [mysql]
  27.     default-character-set=utf8mb4
  28.     [mysqld]
  29.     character-set-server = utf8mb4
  30.     collation-server = utf8mb4_unicode_ci
  31.     init_connect='SET NAMES utf8mb4'
  32.     skip-character-set-client-handshake = true
  33.     max_connections=2000
  34.     secure_file_priv=/var/lib/mysql
  35.     bind-address=0.0.0.0
  36.     symbolic-links=0
  37.     sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'
  39. ---
  40. apiVersion: apps/v1
  41. kind: Deployment
  42. metadata:
  43.   name: xh-apollo-mysql
  44.   namespace: xh-apollo
  45. spec:
  46.   selector:
  47.     matchLabels:
  48.       app: xh-apollo-mysql
  49.   template:
  50.     metadata:
  51.       labels:
  52.         app: xh-apollo-mysql
  53.     spec:
  54.       imagePullSecrets:
  55.         - name: secret-name
  56.       containers:
  57.         - name: xh-apollo-mysql
  58.           image: scdhb.f3322.net:32000/base-soft/mysql:release
  59.           imagePullPolicy: Always
  60.           ports:
  61.             - containerPort: 3306
  62.           env:
  63.             - name: MYSQL_ROOT_PASSWORD
  64.               value: root
  65.           args:
  66.             - --datadir
  67.             - /var/lib/mysql
  68.           volumeMounts:
  69.             - name: config
  70.               mountPath: /etc/mysql/conf.d/my.cnf
  71.               subPath: my.cnf
  72.       volumes:
  73.         - name: config
  74.           configMap:
  75.             name: xh-apollo-mysql-cm
  77. #          command: ["RENAME USER `root`@`localhost` TO `root`@`%`"]
  79. ---
  80. apiVersion: v1
  81. kind: Service
  82. metadata:
  83.   name: xh-apollo-mysql
  84.   namespace: xh-apollo
  85. spec:
  86.   type: LoadBalancer
  87.   selector:
  88.     app: xh-apollo-mysql
  89.   ports:
  90.     - name: mysql-svc-port
  91.       port: 30083
  92.       targetPort: 3306
  93. #      nodePort: 30083
  1. 查看部署结果：
  1. [root@saaspy ~]# kubectl get svc,pod,cm -n xh-apollo
  2. NAME                       TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)           AGE
  3. service/xh-apollo-admin    NodePort       172.31.106.174   <none>           8090:30757/TCP    4m4s
  4. service/xh-apollo-config   NodePort       172.31.36.142    <none>           8080:31979/TCP    4m4s
  5. service/xh-apollo-mysql    LoadBalancer   172.31.52.61     xx.xx.xx.xx     30083:30281/TCP   4m3s
  6. service/xh-apollo-mysql2   LoadBalancer   172.31.135.127   xx.xx.xx.xx   30084:30247/TCP   9s
  7. service/xh-apollo-portal   NodePort       172.31.0.146     <none>           8070:30439/TCP    4m4s
  9. NAME                                    READY   STATUS             RESTARTS   AGE
  10. pod/xh-apollo-admin-b6d9d48cf-snzwz     0/1     CrashLoopBackOff   4          4m4s
  11. pod/xh-apollo-config-599bdb7b9b-cz656   0/1     Error              5          4m4s
  12. pod/xh-apollo-mysql-7c6856978f-2rp6n    1/1     Running            0          4m3s
  13. pod/xh-apollo-mysql2-65f7f7956b-6kg6n   1/1     Running            0          9s
  14. pod/xh-apollo-portal-d85bffdc-qbq4p     0/1     Error              5          4m5s
  16. NAME                           DATA   AGE
  17. configmap/xh-apollo-mysql-cm   1      4m4s
  18. [root@saaspy ~]#
  1. 访问的时候就可以通过 EXTERNAL-IP 加 port 访问啦。

3 容器内部使用 curl、ping 命令

3.1 先要查看系统 cat release 或者 cat /etc/release

  1. bash-4.4# pwd
  2. /opt
  3. bash-4.4# cat /etc/*release
  4. 3.8.2
  5. NAME="Alpine Linux"
  6. ID=alpine
  7. VERSION_ID=3.8.2
  8. PRETTY_NAME="Alpine Linux v3.8"
  9. HOME_URL="http://alpinelinux.org"
  10. BUG_REPORT_URL="http://bugs.alpinelinux.org"
  11. bash-4.4#
  12. bash-4.4# cd /etc
  13. bash-4.4# pwd
  14. /etc
  15. bash-4.4# cat *release
  16. 3.8.2
  17. NAME="Alpine Linux"
  18. ID=alpine
  19. VERSION_ID=3.8.2
  20. PRETTY_NAME="Alpine Linux v3.8"
  21. HOME_URL="http://alpinelinux.org"
  22. BUG_REPORT_URL="http://bugs.alpinelinux.org"
  23. bash-4.4#

3.2 是否存在命令 curl 、ping

  1. bash-4.4# curl loclahost:8080
  2. bash: curl: command not found
  3. bash-4.4#
  4. bash-4.4# ping localhost
  5. PING localhost (127.0.0.1): 56 data bytes
  6. 64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.037 ms
  7. 64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.094 ms
  8. 64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.040 ms
  9. ^C
  10. --- localhost ping statistics ---
  11. 3 packets transmitted, 3 packets received, 0% packet loss
  12. round-trip min/avg/max = 0.037/0.057/0.094 ms
  13. bash-4.4#

3.3 Alpine Linux v3.8 安装 curl

  1. 1.设置国内镜像源
  3. sed -i 's/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g' /etc/apk/repositories
  5. 2.安装curl
  6. apk add curl

3.4 其它系统

  1. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt# ping 47.101.139.203
  2. bash: ping: command not found
  3. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt# cat /etc/*release
  4. PRETTY_NAME="Debian GNU/Linux 10 (buster)"
  5. NAME="Debian GNU/Linux"
  6. VERSION_ID="10"
  7. VERSION="10 (buster)"
  8. VERSION_CODENAME=buster
  9. ID=debian
  10. HOME_URL="https://www.debian.org/"
  11. SUPPORT_URL="https://www.debian.org/support"
  12. BUG_REPORT_URL="https://bugs.debian.org/"
  13. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt#
  14. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt# apt update
  15. Get:1 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
  16. Get:2 http://security.debian.org/debian-security buster/updates/main amd64 Packages [299 kB]
  17. Get:3 http://deb.debian.org/debian buster InRelease [122 kB]
  18. Get:4 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
  19. Get:5 http://deb.debian.org/debian buster/main amd64 Packages [7907 kB]
  20. Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages [15.2 kB]
  21. Fetched 8460 kB in 6s (1321 kB/s)
  22. Reading package lists... Done
  23. Building dependency tree
  24. Reading state information... Done
  25. 8 packages can be upgraded. Run 'apt list --upgradable' to see them.
  26. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt#
  27. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt# apt-get curl
  28. E: Invalid operation curl
  29. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt# apt-get curl -y
  30. E: Invalid operation curl
  31. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt# apt-get install curl -y
  32. Reading package lists... Done
  33. Building dependency tree
  34. Reading state information... Done
  35. The following package was automatically installed and is no longer required:
  36.   lsb-base
  37. ...
  38. ...
  39. Setting up curl (7.64.0-4+deb10u2) ...
  40. Processing triggers for libc-bin (2.28-10) ...
  41. root@xh-pandora-register01-678cf8bb8d-c8bb7:/opt#
  43. # ping 命令
  44. root@xh-pandora-register01-8477d49589-p4qxl:/opt#
  45. root@xh-pandora-register01-8477d49589-p4qxl:/opt# apt-get install inetutils-ping -y
  46. Reading package lists... Done
  47. Building dependency tree
  48. Reading state information... Done
  49. The following package was automatically installed and is no longer required:
  50.   lsb-base
  51. Use 'apt autoremove' to remove it.
  52. The following additional packages will be installed:
  53.   libidn11 netbase
  54. The following NEW packages will be installed:
  55.   inetutils-ping libidn11 netbase
  56. 0 upgraded, 3 newly installed, 0 to remove and 8 not upgraded.
  57. Need to get 362 kB of archives.
  58. After this operation, 718 kB of additional disk space will be used.
  59. 0% [Connecting to deb.debian.org]
  60. Get:1 http://deb.debian.org/debian buster/main amd64 netbase all 5.6 [19.4 kB]
  61. Get:2 http://deb.debian.org/debian buster/main amd64 libidn11 amd64 1.33-2.2 [116 kB]
  62. Get:3 http://deb.debian.org/debian buster/main amd64 inetutils-ping amd64 2:1.9.4-7+deb10u1 [226 kB]
  63. Fetched 362 kB in 42s (8578 B/s)
  64. debconf: delaying package configuration, since apt-utils is not installed
  65. Selecting previously unselected package netbase.
  66. (Reading database ... 7067 files and directories currently installed.)
  67. Preparing to unpack .../archives/netbase_5.6_all.deb ...
  68. Unpacking netbase (5.6) ...
  69. Selecting previously unselected package libidn11:amd64.
  70. Preparing to unpack .../libidn11_1.33-2.2_amd64.deb ...
  71. Unpacking libidn11:amd64 (1.33-2.2) ...
  72. Selecting previously unselected package inetutils-ping.
  73. Preparing to unpack .../inetutils-ping_2%3a1.9.4-7+deb10u1_amd64.deb ...
  74. Unpacking inetutils-ping (2:1.9.4-7+deb10u1) ...
  75. Setting up libidn11:amd64 (1.33-2.2) ...
  76. Setting up netbase (5.6) ...
  77. Setting up inetutils-ping (2:1.9.4-7+deb10u1) ...
  78. Processing triggers for libc-bin (2.28-10) ...
  79. root@xh-pandora-register01-8477d49589-p4qxl:/opt#

4 ack 的 pod 数量限制

4.1 选择了 terway 网络，每个节点 pod 数量有限制

k8s 在建立 pod ，需要规划分配网络，有了限制，导致不能创建 pod，大量 pod 状态为 pending。
部署服务，不能按照pod 限制的最大数量来卡死 pod 实际部署数量，要预留数量，便于容器的滚动升级（ack 在计算 pod 个数时，Terminating 也算一个，在 k8s 中，针对单一个 pod 来升级，会被计算为两个，一个创建中 containerCreating，一个终止中 terminating）。至于保留多少pod，应该根据最大的并行升级数量来算，比如，需要升级3个pod，那么就要预留 3 个pod。
若是不规划好pod预留量，有可能会导致 k8s 崩溃（删除不掉，新建不了，二者相互等待）