本文记录将极客时间，杨波，Spring Boot 与 Kubernetes 云原生微服务实践 专栏的示例项目 Staffjoy， 部署到本地 Kubernetes 环境的过程。

环境：
CPU: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz 2.90 GHz
内存： 16.0 GB
操作系统： Windows 10 专业版，21H2，19044.1645
Docker Desktop For Windows: 4.8.1 (78998)
Kubernetes: v1.24.0

1 配置 k8s

本节配置 k8s/test 目录下的文件，本文中的 k8s 配置文件路径默认为相对 k8s/test 目录的路径
1）config/config.yaml
复制 config/config.yaml.example 文件为 config/config.yaml 文件，然后修改其内容，如下所示：

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: common-config
  5. data:
  6.   SPRING_PROFILES_ACTIVE: test
  7.   SERVER_PORT: "80"
  8.   EMAIL_SERVICE_ENDPOINT: http://email-service
  9.   COMPANY_SERVICE_ENDPOINT: http://company-service
  10.   ACCOUNT_SERVICE_ENDPOINT: http://account-service
  11.   BOT_SERVICE_ENDPOINT: http://bot-service
  12.   SMS_SERVICE_ENDPOINT: http://sms-service
  13.   SENTRY_DSN: https://80aaf4ae889b414f9fe72e3904cd5246@sentry.io/1380198
  14.   SIGNING_SECRET: your_signing_secret
  15.   INTERCOM_ACCESS_TOKEN: YOUR_INTERCOM_ACCESS_TOKEN
  16.   INTERCOM_APP_ID: TBD
  17.   INTERCOM_SIGNING_SECRET: TBD
  18.   ALIYUN_ACCESS_KEY: YOUR_ALIYUN_ACCESS_KEY
  19.   ALIYUN_ACCESS_SECRET: YOUR_ALIYUN_ACCESS_SECRET
  20.   RECAPTCHA_PUBLIC: test-recaptcha-public
  21.   RECAPTCHA_PRIVATE: test-recaptcha-private
  22.   ACCOUNT_DATASOURCE_URL: jdbc:mysql://mysql-svc/staffjoy_account?useUnicode=true&characterEncoding=utf-8&serverTimezone=GMT%2B8
  23.   ACCOUNT_DATASOURCE_USERNAME: root
  24.   ACCOUNT_DATASOURCE_PASSWORD: "12345678"
  25.   COMPANY_DATASOURCE_URL: jdbc:mysql://mysql-svc/staffjoy_company?useUnicode=true&characterEncoding=utf-8&serverTimezone=GMT%2B8
  26.   COMPANY_DATASOURCE_USERNAME: root
  27.   COMPANY_DATASOURCE_PASSWORD: "12345678"

2）mysql-svc.yaml

  1. # https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-mapping-external-services
  2. kind: Service
  3. apiVersion: v1
  4. metadata:
  5.   name: mysql-svc
  6. spec:
  7.   type: ClusterIP
  8.   ports:
  9.     - port: 3306
  10.       targetPort: 3306
  11. ---
  12. # https://confluence.atlassian.com/jirakb/configuring-database-connection-results-in-error-host-xxxxxxx-is-not-allowed-to-connect-to-this-mysql-server-358908249.html
  13. kind: Endpoints
  14. apiVersion: v1
  15. metadata:
  16.   name: mysql-svc
  17. subsets:
  18.   - addresses:
  19.       - ip: 127.0.0.1 # UPDATE TO YOUR MYSQL IP ADDRESS
  20.     ports:
  21.       - port: 3306

2 本地 Kubernetes 环境搭建

本文使用 Docker Desktop for Windows 的 Kubernetes 搭建本地 kubernetes 环境。

2.1 安装 Docker Desktop for Windows 及 Kubernetes

Docker Desktop for Windows 下载地址： https://docs.docker.com/desktop/windows/install/

image.png
Docker Desktop 及 Kubernetes 版本

安装并启动 Docker Desktop 后，点击【设置>Kubernetes>Enable Kubernetes】，启用 Kubernetes。
image.png
Docker Desktop 启用 Kubernetes

2.2 校验 Kubernetes 安装

1) Docker Desktop 启用 Kubernetes Context
Doccker Desktop 可能安装多个 Kubernetes Context，在使用 Kubernetes 前，确保 Docker Desktop 当前运行的 Kubernetes Context 是我们想要的，本文使用的 Kubernetes Context 为 docker-desktop。
image.png
Docker Desktop 设置 Kubernetes Context 为 docker-desktop
2) 查看 Kubernetes 版本
命令： kubectl version —output=yaml
执行结果：

  1. clientVersion:
  2.   buildDate: "2022-05-03T13:46:05Z"
  3.   compiler: gc
  4.   gitCommit: 4ce5a8954017644c5420bae81d72b09b735c21f0
  5.   gitTreeState: clean
  6.   gitVersion: v1.24.0
  7.   goVersion: go1.18.1
  8.   major: "1"
  9.   minor: "24"
  10.   platform: windows/amd64
  11. kustomizeVersion: v4.5.4
  12. serverVersion:
  13.   buildDate: "2022-05-03T13:38:19Z"
  14.   compiler: gc
  15.   gitCommit: 4ce5a8954017644c5420bae81d72b09b735c21f0
  16.   gitTreeState: clean
  17.   gitVersion: v1.24.0
  18.   goVersion: go1.18.1
  19.   major: "1"
  20.   minor: "24"
  21.   platform: linux/amd64

3) 查看当前 Kubernetes Context
命令： kubectl config current-context
执行结果：

  1. docker-desktop

4) 查看 Kubernetes cluster 信息
命令： kubectl cluster-info
执行结果：

  1. Kubernetes control plane is running at https://kubernetes.docker.internal:6443
  2. CoreDNS is running at https://kubernetes.docker.internal:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
  4. To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

5) 查看 Kubernetes node 信息
命令： kubectl get nodes
执行结果：

  1. NAME             STATUS   ROLES           AGE   VERSION
  2. docker-desktop   Ready    control-plane   39m   v1.24.0

2.3 【失败，搁置】安装和访问 Kubernetes Dashboard

访问 Kubernetes 有命令行（kubectl）和 图形界面（Kubernetes Dashboard）2 种方式，为便于使用 Kubernetes，安装 Kubernetes Dashboard。

Kubernetes Dashboard is a general purpose, web-based UI for Kubernetes clusters. It allows users to manage applications running in the cluster and troubleshoot them, as well as manage the cluster itself.
1) 安装 Kubernetes Dashboard
命令：

  1. kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.5.1/aio/deploy/recommended.yaml

执行结果：

  1. namespace/kubernetes-dashboard created
  2. serviceaccount/kubernetes-dashboard created
  3. service/kubernetes-dashboard created
  4. secret/kubernetes-dashboard-certs created
  5. secret/kubernetes-dashboard-csrf created
  6. secret/kubernetes-dashboard-key-holder created
  7. configmap/kubernetes-dashboard-settings created
  8. role.rbac.authorization.k8s.io/kubernetes-dashboard created
  9. clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
  10. rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
  11. clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
  12. deployment.apps/kubernetes-dashboard created
  13. service/dashboard-metrics-scraper created
  14. deployment.apps/dashboard-metrics-scraper created

3 构建 Docker 镜像

  • mvn clean package -DskipTests
  • docker-compose build

  • docker images

    4 部署mysql数据库

  • staffjoy_account

  • staffjoy_company

  • 授予ip访问权限：

    1. use mysql;
    2. grant all privileges on *.* to root@10.9.30.122 identified by '12345678' with grant option;

    5 部署Staffjoy(k8s/test)

  • cd k8s/test

  • kubectl apply -f .\config\config.yaml
  • kubectl apply -f .

    6 端口转发

    1）查看 faraday-svc-deployment 的 pod 名称
    命令：kubectl get pods
    执行结果：
    1. NAME                                        READY   STATUS              RESTARTS        AGE
    2. account-svc-deployment-966756658-qcpsg      1/1     Running             22 (8m9s ago)   93m
    3. app-spa-deployment-64746d8ddf-s22mt         0/1     ErrImageNeverPull   0               4m46s
    4. bot-svc-deployment-d8979c94d-bgt7c          1/1     Running             0               4m46s
    5. company-svc-deployment-ff9cfdd78-kfm64      1/1     Running             3 (3m35s ago)   4m46s
    6. email-svc-deployment-88579d685-4tvsv        1/1     Running             0               4m46s
    7. faraday-svc-deployment-7d6798cbf7-8ntg7     1/1     Running             0               4m46s
    8. myaccount-spa-deployment-5c964c788c-hz5fd   0/1     ErrImageNeverPull   0               4m46s
    9. whoami-svc-deployment-8455464c58-p55tm      1/1     Running             0               4m46s
    10. www-web-deployment-745984b974-xbh99         1/1     Running             0               4m46s
    2）启用 faraday-svc-deployment 端口转发
    命令：kubectl port-forward faraday-svc-deployment-7d6798cbf7-8ntg7 80:80
    执行结果：
    1. Forwarding from 127.0.0.1:80 -> 80
    2. Forwarding from [::1]:80 -> 80

    7 启用 SwitchHosts

    image.png

    8 Kubernetes 命令行校验

    1) 查看 pods
    命令： kubectl get pods -o wide
    执行结果：
    1. NAME                                        READY   STATUS              RESTARTS       AGE    IP          NODE             NOMINATED NODE   READINESS GATES
    2. account-svc-deployment-966756658-qcpsg      1/1     Running             22 (33m ago)   118m   10.1.0.14   docker-desktop   <none>           <none>
    3. app-spa-deployment-64746d8ddf-s22mt         0/1     ErrImageNeverPull   0              29m    10.1.0.16   docker-desktop   <none>           <none>
    4. bot-svc-deployment-d8979c94d-bgt7c          1/1     Running             0              29m    10.1.0.15   docker-desktop   <none>           <none>
    5. company-svc-deployment-ff9cfdd78-kfm64      1/1     Running             3 (28m ago)    29m    10.1.0.18   docker-desktop   <none>           <none>
    6. email-svc-deployment-88579d685-4tvsv        1/1     Running             0              29m    10.1.0.17   docker-desktop   <none>           <none>
    7. faraday-svc-deployment-7d6798cbf7-8ntg7     1/1     Running             0              29m    10.1.0.20   docker-desktop   <none>           <none>
    8. myaccount-spa-deployment-5c964c788c-hz5fd   0/1     ErrImageNeverPull   0              29m    10.1.0.23   docker-desktop   <none>           <none>
    9. whoami-svc-deployment-8455464c58-p55tm      1/1     Running             0              29m    10.1.0.21   docker-desktop   <none>           <none>
    10. www-web-deployment-745984b974-xbh99         1/1     Running             0              29m    10.1.0.22   docker-desktop   <none>           <none>

可以看到 app-spa-deployment 和 myaccount-spa-deployment 2 个 deployment 没有部署成功。
2）查看 services
命令： kubectl get services
执行结果：

  1. NAME                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
  2. account-service     ClusterIP   10.108.154.240   <none>        80/TCP         3h8m
  3. app-service         ClusterIP   10.98.87.248     <none>        80/TCP         3h8m
  4. bot-service         ClusterIP   10.96.205.124    <none>        80/TCP         3h8m
  5. company-service     ClusterIP   10.106.6.147     <none>        80/TCP         3h8m
  6. email-service       ClusterIP   10.99.105.209    <none>        80/TCP         3h8m
  7. faraday-service     NodePort    10.110.85.46     <none>        80:30001/TCP   3h8m
  8. kubernetes          ClusterIP   10.96.0.1        <none>        443/TCP        5h51m
  9. myaccount-service   ClusterIP   10.98.23.240     <none>        80/TCP         3h8m
  10. mysql-svc           ClusterIP   10.101.216.158   <none>        3306/TCP       3h8m
  11. whoami-service      ClusterIP   10.98.48.102     <none>        80/TCP         3h8m
  12. www-service         ClusterIP   10.110.232.73    <none>        80/TCP         3h8m

3）查看 deployments
命令： kubectl get deployments
执行结果：

  1. NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
  2. account-svc-deployment     1/1     1            1           121m
  3. app-spa-deployment         0/1     1            0           33m
  4. bot-svc-deployment         1/1     1            1           33m
  5. company-svc-deployment     1/1     1            1           33m
  6. email-svc-deployment       1/1     1            1           33m
  7. faraday-svc-deployment     1/1     1            1           33m
  8. myaccount-spa-deployment   0/1     1            0           33m
  9. whoami-svc-deployment      1/1     1            1           33m
  10. www-web-deployment         1/1     1            1           33m

可以看到 app-spa-deployment 和 myaccount-spa-deployment 2 个 deployment 没有部署成功。

9 Kubetnetes Dashboard 校验

10 Staffjoy 校验

在浏览器访问 http://www.staffjoy-v2.local/（或 http://www.staffjoy-v2.local:30001/），输入邮箱 zixiu@jskillcloud.com，然后点击【SIGN UP FOR FREE】，跳转到注册确认页

image.png
在Staffjoy Web 主页（http://www.staffjoy-v2.local/）注册账号

image.png
注册确认页

查看 staffjoy_account 数据库，account 表，发现新增了 1 条记录，说明 Staffjoy 的 mysql-svc 正常运行。

image.png
注册账号后，staffjoy_account.account 表新增了 1 条记录

11 清理

1）kubectl delete deployments —all
执行结果：

  1. deployment.apps "account-svc-deployment" deleted
  2. deployment.apps "app-spa-deployment" deleted
  3. deployment.apps "bot-svc-deployment" deleted
  4. deployment.apps "company-svc-deployment" deleted
  5. deployment.apps "email-svc-deployment" deleted
  6. deployment.apps "faraday-svc-deployment" deleted
  7. deployment.apps "myaccount-spa-deployment" deleted
  8. deployment.apps "whoami-svc-deployment" deleted
  9. deployment.apps "www-web-deployment" deleted

2）kubectl delete services —all
执行结果：

  1. service "account-service" deleted
  2. service "app-service" deleted
  3. service "bot-service" deleted
  4. service "company-service" deleted
  5. service "email-service" deleted
  6. service "faraday-service" deleted
  7. service "kubernetes" deleted
  8. service "myaccount-service" deleted
  9. service "mysql-svc" deleted
  10. service "whoami-service" deleted
  11. service "www-service" deleted

3）kubectl delete configmaps —all
执行结果：

configmap "common-config" deleted
configmap "kube-root-ca.crt" deleted

12 除障记录

12.1 config/config.yaml 文件中数字类型属性值要用引号包围

12.1.1 问题描述

config/config.yaml 文件中，数字类型属性值若写成：
ACCOUNT_DATASOURCE_PASSWORD: 12345678

则，执行命令： kubectl apply -f .\config\config.yaml，会出错：

Error from server (BadRequest): error when creating ".\\config\\config.yaml": ConfigMap in version "v1" cannot be handled as a ConfigMap: json: cannot unmarshal number into Go struct field ConfigMap.data o
f type string

12.1.2 解法

将数字类型属性值用引号包围起来：
ACCOUNT_DATASOURCE_PASSWORD: “12345678”

12.2 mysql-svc Endpoints 的 ip 不能为环回地址

12.2.1 问题描述

mysql-svc.yaml 文件：

# https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-mapping-external-services
kind: Service
apiVersion: v1
metadata:
  name: mysql-svc
spec:
  type: ClusterIP
  ports:
    - port: 3306
      targetPort: 3306
---
# https://confluence.atlassian.com/jirakb/configuring-database-connection-results-in-error-host-xxxxxxx-is-not-allowed-to-connect-to-this-mysql-server-358908249.html
kind: Endpoints
apiVersion: v1
metadata:
  name: mysql-svc
subsets:
  - addresses:
      - ip: 127.0.0.1 # UPDATE TO YOUR MYSQL IP ADDRESS
    ports:
      - port: 3306

此时执行命令： “kubectl apply -f .”，会出错：

Error from server (Invalid): error when creating "mysql-svc.yaml": Endpoints "mysql-svc" is invalid: subsets[0].addresses[0].ip: Invalid value: "127.0.0.1": may not be in the loopback range (127.0.0.0/8, :
:1/128)

12.2.2 解法

用 ipconfig 命令查看本地 ip 地址，然后 Endpoints 的 ip 设置成本地 ip 地址。

image.png
用 ipconfig 命令查看本地 ipv4 地址

修改后的 mysql-svc.yaml 文件：

# https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-mapping-external-services
kind: Service
apiVersion: v1
metadata:
  name: mysql-svc
spec:
  type: ClusterIP
  ports:
    - port: 3306
      targetPort: 3306
---
# https://confluence.atlassian.com/jirakb/configuring-database-connection-results-in-error-host-xxxxxxx-is-not-allowed-to-connect-to-this-mysql-server-358908249.html
kind: Endpoints
apiVersion: v1
metadata:
  name: mysql-svc
subsets:
  - addresses:
      - ip: 10.9.30.122 # UPDATE TO YOUR MYSQL IP ADDRESS
    ports:
      - port: 3306

12.3 Deployment 配置与 Kubernetes API 版本不兼容

12.3.1 问题描述

以 account-svc.yaml 文件为例，其原始内容如下：

# https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-apiversion-definition-guide.html
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: account-svc-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: account-svc
        env: test
    spec:
      containers:
        - name: account-svc
          image: boboweike/account-svc
          imagePullPolicy: Never
          ports:
            - name: http-port
              containerPort: 80
          envFrom:
            - configMapRef:
                name: common-config
---
apiVersion: v1
kind: Service
metadata:
  name: account-service
spec:
  selector:
    app: account-svc
    env: test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

此时执行命令： “kubectl apply -f .”，会出错：

resource mapping not found for name: "account-svc-deployment" namespace: "" from "account-svc.yaml": no matches for kind "Deployment" in version "extensions/v1beta1"
ensure CRDs are installed first

12.3.2 原因

这个问题是 account-svc-deployment 这个 Deployment 的配置与当前运行的 Kubernetes API 版本不兼容导致的。

12.3.3 解法

1）查看 Deployment 对应的 API 版本
命令：kubectl api-resources
执行结果：
image.png

从执行结果中找到 KIND 列为 Deployment 的行，该行的 APIVERSION 列为 apps/v1，将 account-svc.yaml 文件中 account-svc-deployment 的 apiVersion 字段值改成 apps/v1。
2）查找指定 API 版本的 Deployment 的配置文档，按文档修改 Deployment 配置
对于 apiVersion apps/v1 的 Deployment 的配置，需要增加 spec.selector 字段
3）最终的 account-svc.yaml 文件
最终的 account-svc.yaml 文件如下：

# https://matthewpalmer.net/kubernetes-app-developer/articles/kubernetes-apiversion-definition-guide.html
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: account-svc-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: account-svc
      env: test
  template:
    metadata:
      labels:
        app: account-svc
        env: test
    spec:
      containers:
        - name: account-svc
          image: boboweike/account-svc
          imagePullPolicy: Never
          ports:
            - name: http-port
              containerPort: 80
          envFrom:
            - configMapRef:
                name: common-config
---
apiVersion: v1
kind: Service
metadata:
  name: account-service
spec:
  selector:
    app: account-svc
    env: test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

4) 其他服务配置文件参照 account-svc.yaml 文件修改