背景

服务器配置

节点 内网IP 公网IP 配置
ren 10.0.4.17 1.15.230.38 4C8G
yan 10.0.4.15 101.34.64.205 4C8G
bai 192.168.0.4 106.12.145.172 2C8G

参考kubesphere官方文档

https://kubesphere.io/zh/docs/v3.3/quick-start/minimal-kubesphere-on-k8s/

安装KubeSphere前置环境

提前安装k8s集群

脚本安装-跨云创建k8s集群

安装nfs-server

  1. # 在每个机器。
  2. yum install -y nfs-utils
  5. # 在master 执行以下命令 
  6. # 配置要暴露的目录
  7. echo "/nfs/data/ *(insecure,rw,sync,no_root_squash)" > /etc/exports
  10. # 执行以下命令，启动 nfs 服务;创建共享目录
  11. mkdir -p /nfs/data
  14. # 在master执行
  15. systemctl enable rpcbind
  16. systemctl enable nfs-server
  17. systemctl start rpcbind
  18. systemctl start nfs-server
  20. # 使配置生效
  21. exportfs -r
  24. #检查配置是否生效
  25. exportfs

配置nfs-client（worker节点执行）

worker节点执行，文件同步

  1. showmount -e 1.15.230.38
  3. mkdir -p /nfs/data
  5. mount -t nfs 1.15.230.38:/nfs/data /nfs/data

验证nfs

任意节点，/nfs/data 目录下操作文件，则各节点目录同步更新

配置默认存储

配置动态供应的默认存储类

  1. ## 创建了一个存储类
  2. apiVersion: storage.k8s.io/v1
  3. kind: StorageClass
  4. metadata:
  5.   name: nfs-storage
  6.   annotations:
  7.     storageclass.kubernetes.io/is-default-class: "true"
  8. provisioner: k8s-sigs.io/nfs-subdir-external-provisioner
  9. parameters:
  10.   archiveOnDelete: "true"  ## 删除pv的时候，pv的内容是否要备份
  12. ---
  13. apiVersion: apps/v1
  14. kind: Deployment
  15. metadata:
  16.   name: nfs-client-provisioner
  17.   labels:
  18.     app: nfs-client-provisioner
  19.   # replace with namespace where provisioner is deployed
  20.   namespace: default
  21. spec:
  22.   replicas: 1
  23.   strategy:
  24.     type: Recreate
  25.   selector:
  26.     matchLabels:
  27.       app: nfs-client-provisioner
  28.   template:
  29.     metadata:
  30.       labels:
  31.         app: nfs-client-provisioner
  32.     spec:
  33.       serviceAccountName: nfs-client-provisioner
  34.       containers:
  35.         - name: nfs-client-provisioner
  36.           image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/nfs-subdir-external-provisioner:v4.0.2
  37.           # resources:
  38.           #    limits:
  39.           #      cpu: 10m
  40.           #    requests:
  41.           #      cpu: 10m
  42.           volumeMounts:
  43.             - name: nfs-client-root
  44.               mountPath: /persistentvolumes
  45.           env:
  46.             - name: PROVISIONER_NAME
  47.               value: k8s-sigs.io/nfs-subdir-external-provisioner
  48.             - name: NFS_SERVER
  49.               value: 1.15.230.38 ## 指定自己nfs服务器地址
  50.             - name: NFS_PATH  
  51.               value: /nfs/data  ## nfs服务器共享的目录
  52.       volumes:
  53.         - name: nfs-client-root
  54.           nfs:
  55.             server: 1.15.230.38
  56.             path: /nfs/data
  57. ---
  58. apiVersion: v1
  59. kind: ServiceAccount
  60. metadata:
  61.   name: nfs-client-provisioner
  62.   # replace with namespace where provisioner is deployed
  63.   namespace: default
  64. ---
  65. kind: ClusterRole
  66. apiVersion: rbac.authorization.k8s.io/v1
  67. metadata:
  68.   name: nfs-client-provisioner-runner
  69. rules:
  70.   - apiGroups: [""]
  71.     resources: ["nodes"]
  72.     verbs: ["get", "list", "watch"]
  73.   - apiGroups: [""]
  74.     resources: ["persistentvolumes"]
  75.     verbs: ["get", "list", "watch", "create", "delete"]
  76.   - apiGroups: [""]
  77.     resources: ["persistentvolumeclaims"]
  78.     verbs: ["get", "list", "watch", "update"]
  79.   - apiGroups: ["storage.k8s.io"]
  80.     resources: ["storageclasses"]
  81.     verbs: ["get", "list", "watch"]
  82.   - apiGroups: [""]
  83.     resources: ["events"]
  84.     verbs: ["create", "update", "patch"]
  85. ---
  86. kind: ClusterRoleBinding
  87. apiVersion: rbac.authorization.k8s.io/v1
  88. metadata:
  89.   name: run-nfs-client-provisioner
  90. subjects:
  91.   - kind: ServiceAccount
  92.     name: nfs-client-provisioner
  93.     # replace with namespace where provisioner is deployed
  94.     namespace: default
  95. roleRef:
  96.   kind: ClusterRole
  97.   name: nfs-client-provisioner-runner
  98.   apiGroup: rbac.authorization.k8s.io
  99. ---
  100. kind: Role
  101. apiVersion: rbac.authorization.k8s.io/v1
  102. metadata:
  103.   name: leader-locking-nfs-client-provisioner
  104.   # replace with namespace where provisioner is deployed
  105.   namespace: default
  106. rules:
  107.   - apiGroups: [""]
  108.     resources: ["endpoints"]
  109.     verbs: ["get", "list", "watch", "create", "update", "patch"]
  110. ---
  111. kind: RoleBinding
  112. apiVersion: rbac.authorization.k8s.io/v1
  113. metadata:
  114.   name: leader-locking-nfs-client-provisioner
  115.   # replace with namespace where provisioner is deployed
  116.   namespace: default
  117. subjects:
  118.   - kind: ServiceAccount
  119.     name: nfs-client-provisioner
  120.     # replace with namespace where provisioner is deployed
  121.     namespace: default
  122. roleRef:
  123.   kind: Role
  124.   name: leader-locking-nfs-client-provisioner
  125.   apiGroup: rbac.authorization.k8s.io

安装

  1. kubectl apply -f sc.yaml

验证

  1. #确认配置是否生效
  2. kubectl get sc

image.png
image.png

pvc验证

  1. kind: PersistentVolumeClaim
  2. apiVersion: v1
  3. metadata:
  4.   name: nginx-pvc
  5. spec:
  6.   accessModes:
  7.     - ReadWriteMany
  8.   resources:
  9.     requests:
  10.       storage: 200Mi
  1. kubectl apply -f pvc.yaml

image.png

安装集群指标监控(metrics-server)

集群指标监控组件 kind: Deployment image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/metrics-server:v0.4.3

  1. apiVersion: v1
  2. kind: ServiceAccount
  3. metadata:
  4.   labels:
  5.     k8s-app: metrics-server
  6.   name: metrics-server
  7.   namespace: kube-system
  8. ---
  9. apiVersion: rbac.authorization.k8s.io/v1
  10. kind: ClusterRole
  11. metadata:
  12.   labels:
  13.     k8s-app: metrics-server
  14.     rbac.authorization.k8s.io/aggregate-to-admin: "true"
  15.     rbac.authorization.k8s.io/aggregate-to-edit: "true"
  16.     rbac.authorization.k8s.io/aggregate-to-view: "true"
  17.   name: system:aggregated-metrics-reader
  18. rules:
  19. - apiGroups:
  20.   - metrics.k8s.io
  21.   resources:
  22.   - pods
  23.   - nodes
  24.   verbs:
  25.   - get
  26.   - list
  27.   - watch
  28. ---
  29. apiVersion: rbac.authorization.k8s.io/v1
  30. kind: ClusterRole
  31. metadata:
  32.   labels:
  33.     k8s-app: metrics-server
  34.   name: system:metrics-server
  35. rules:
  36. - apiGroups:
  37.   - ""
  38.   resources:
  39.   - pods
  40.   - nodes
  41.   - nodes/stats
  42.   - namespaces
  43.   - configmaps
  44.   verbs:
  45.   - get
  46.   - list
  47.   - watch
  48. ---
  49. apiVersion: rbac.authorization.k8s.io/v1
  50. kind: RoleBinding
  51. metadata:
  52.   labels:
  53.     k8s-app: metrics-server
  54.   name: metrics-server-auth-reader
  55.   namespace: kube-system
  56. roleRef:
  57.   apiGroup: rbac.authorization.k8s.io
  58.   kind: Role
  59.   name: extension-apiserver-authentication-reader
  60. subjects:
  61. - kind: ServiceAccount
  62.   name: metrics-server
  63.   namespace: kube-system
  64. ---
  65. apiVersion: rbac.authorization.k8s.io/v1
  66. kind: ClusterRoleBinding
  67. metadata:
  68.   labels:
  69.     k8s-app: metrics-server
  70.   name: metrics-server:system:auth-delegator
  71. roleRef:
  72.   apiGroup: rbac.authorization.k8s.io
  73.   kind: ClusterRole
  74.   name: system:auth-delegator
  75. subjects:
  76. - kind: ServiceAccount
  77.   name: metrics-server
  78.   namespace: kube-system
  79. ---
  80. apiVersion: rbac.authorization.k8s.io/v1
  81. kind: ClusterRoleBinding
  82. metadata:
  83.   labels:
  84.     k8s-app: metrics-server
  85.   name: system:metrics-server
  86. roleRef:
  87.   apiGroup: rbac.authorization.k8s.io
  88.   kind: ClusterRole
  89.   name: system:metrics-server
  90. subjects:
  91. - kind: ServiceAccount
  92.   name: metrics-server
  93.   namespace: kube-system
  94. ---
  95. apiVersion: v1
  96. kind: Service
  97. metadata:
  98.   labels:
  99.     k8s-app: metrics-server
  100.   name: metrics-server
  101.   namespace: kube-system
  102. spec:
  103.   ports:
  104.   - name: https
  105.     port: 443
  106.     protocol: TCP
  107.     targetPort: https
  108.   selector:
  109.     k8s-app: metrics-server
  110. ---
  111. apiVersion: apps/v1
  112. kind: Deployment
  113. metadata:
  114.   labels:
  115.     k8s-app: metrics-server
  116.   name: metrics-server
  117.   namespace: kube-system
  118. spec:
  119.   selector:
  120.     matchLabels:
  121.       k8s-app: metrics-server
  122.   strategy:
  123.     rollingUpdate:
  124.       maxUnavailable: 0
  125.   template:
  126.     metadata:
  127.       labels:
  128.         k8s-app: metrics-server
  129.     spec:
  130.       containers:
  131.       - args:
  132.         - --cert-dir=/tmp
  133.         - --kubelet-insecure-tls
  134.         - --secure-port=4443
  135.         - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  136.         - --kubelet-use-node-status-port
  137.         image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/metrics-server:v0.4.3
  138.         imagePullPolicy: IfNotPresent
  139.         livenessProbe:
  140.           failureThreshold: 3
  141.           httpGet:
  142.             path: /livez
  143.             port: https
  144.             scheme: HTTPS
  145.           periodSeconds: 10
  146.         name: metrics-server
  147.         ports:
  148.         - containerPort: 4443
  149.           name: https
  150.           protocol: TCP
  151.         readinessProbe:
  152.           failureThreshold: 3
  153.           httpGet:
  154.             path: /readyz
  155.             port: https
  156.             scheme: HTTPS
  157.           periodSeconds: 10
  158.         securityContext:
  159.           readOnlyRootFilesystem: true
  160.           runAsNonRoot: true
  161.           runAsUser: 1000
  162.         volumeMounts:
  163.         - mountPath: /tmp
  164.           name: tmp-dir
  165.       nodeSelector:
  166.         kubernetes.io/os: linux
  167.       priorityClassName: system-cluster-critical
  168.       serviceAccountName: metrics-server
  169.       volumes:
  170.       - emptyDir: {}
  171.         name: tmp-dir
  172. ---
  173. apiVersion: apiregistration.k8s.io/v1
  174. kind: APIService
  175. metadata:
  176.   labels:
  177.     k8s-app: metrics-server
  178.   name: v1beta1.metrics.k8s.io
  179. spec:
  180.   group: metrics.k8s.io
  181.   groupPriorityMinimum: 100
  182.   insecureSkipTLSVerify: true
  183.   service:
  184.     name: metrics-server
  185.     namespace: kube-system
  186.   version: v1beta1
  187.   versionPriority: 100
  1. kubectl apply -f metrics.yaml

验证

image.png

  1. kubectl top nodes

image.png

  1. kubectl top pods -A

image.png

安装kubesphere

准备配置文件

3.1.1

  1. ---
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: clusterconfigurations.installer.kubesphere.io
  6. spec:
  7.   group: installer.kubesphere.io
  8.   versions:
  9.   - name: v1alpha1
  10.     served: true
  11.     storage: true
  12.   scope: Namespaced
  13.   names:
  14.     plural: clusterconfigurations
  15.     singular: clusterconfiguration
  16.     kind: ClusterConfiguration
  17.     shortNames:
  18.     - cc
  20. ---
  21. apiVersion: v1
  22. kind: Namespace
  23. metadata:
  24.   name: kubesphere-system
  26. ---
  27. apiVersion: v1
  28. kind: ServiceAccount
  29. metadata:
  30.   name: ks-installer
  31.   namespace: kubesphere-system
  33. ---
  34. apiVersion: rbac.authorization.k8s.io/v1
  35. kind: ClusterRole
  36. metadata:
  37.   name: ks-installer
  38. rules:
  39. - apiGroups:
  40.   - ""
  41.   resources:
  42.   - '*'
  43.   verbs:
  44.   - '*'
  45. - apiGroups:
  46.   - apps
  47.   resources:
  48.   - '*'
  49.   verbs:
  50.   - '*'
  51. - apiGroups:
  52.   - extensions
  53.   resources:
  54.   - '*'
  55.   verbs:
  56.   - '*'
  57. - apiGroups:
  58.   - batch
  59.   resources:
  60.   - '*'
  61.   verbs:
  62.   - '*'
  63. - apiGroups:
  64.   - rbac.authorization.k8s.io
  65.   resources:
  66.   - '*'
  67.   verbs:
  68.   - '*'
  69. - apiGroups:
  70.   - apiregistration.k8s.io
  71.   resources:
  72.   - '*'
  73.   verbs:
  74.   - '*'
  75. - apiGroups:
  76.   - apiextensions.k8s.io
  77.   resources:
  78.   - '*'
  79.   verbs:
  80.   - '*'
  81. - apiGroups:
  82.   - tenant.kubesphere.io
  83.   resources:
  84.   - '*'
  85.   verbs:
  86.   - '*'
  87. - apiGroups:
  88.   - certificates.k8s.io
  89.   resources:
  90.   - '*'
  91.   verbs:
  92.   - '*'
  93. - apiGroups:
  94.   - devops.kubesphere.io
  95.   resources:
  96.   - '*'
  97.   verbs:
  98.   - '*'
  99. - apiGroups:
  100.   - monitoring.coreos.com
  101.   resources:
  102.   - '*'
  103.   verbs:
  104.   - '*'
  105. - apiGroups:
  106.   - logging.kubesphere.io
  107.   resources:
  108.   - '*'
  109.   verbs:
  110.   - '*'
  111. - apiGroups:
  112.   - jaegertracing.io
  113.   resources:
  114.   - '*'
  115.   verbs:
  116.   - '*'
  117. - apiGroups:
  118.   - storage.k8s.io
  119.   resources:
  120.   - '*'
  121.   verbs:
  122.   - '*'
  123. - apiGroups:
  124.   - admissionregistration.k8s.io
  125.   resources:
  126.   - '*'
  127.   verbs:
  128.   - '*'
  129. - apiGroups:
  130.   - policy
  131.   resources:
  132.   - '*'
  133.   verbs:
  134.   - '*'
  135. - apiGroups:
  136.   - autoscaling
  137.   resources:
  138.   - '*'
  139.   verbs:
  140.   - '*'
  141. - apiGroups:
  142.   - networking.istio.io
  143.   resources:
  144.   - '*'
  145.   verbs:
  146.   - '*'
  147. - apiGroups:
  148.   - config.istio.io
  149.   resources:
  150.   - '*'
  151.   verbs:
  152.   - '*'
  153. - apiGroups:
  154.   - iam.kubesphere.io
  155.   resources:
  156.   - '*'
  157.   verbs:
  158.   - '*'
  159. - apiGroups:
  160.   - notification.kubesphere.io
  161.   resources:
  162.   - '*'
  163.   verbs:
  164.   - '*'
  165. - apiGroups:
  166.   - auditing.kubesphere.io
  167.   resources:
  168.   - '*'
  169.   verbs:
  170.   - '*'
  171. - apiGroups:
  172.   - events.kubesphere.io
  173.   resources:
  174.   - '*'
  175.   verbs:
  176.   - '*'
  177. - apiGroups:
  178.   - core.kubefed.io
  179.   resources:
  180.   - '*'
  181.   verbs:
  182.   - '*'
  183. - apiGroups:
  184.   - installer.kubesphere.io
  185.   resources:
  186.   - '*'
  187.   verbs:
  188.   - '*'
  189. - apiGroups:
  190.   - storage.kubesphere.io
  191.   resources:
  192.   - '*'
  193.   verbs:
  194.   - '*'
  195. - apiGroups:
  196.   - security.istio.io
  197.   resources:
  198.   - '*'
  199.   verbs:
  200.   - '*'
  201. - apiGroups:
  202.   - monitoring.kiali.io
  203.   resources:
  204.   - '*'
  205.   verbs:
  206.   - '*'
  207. - apiGroups:
  208.   - kiali.io
  209.   resources:
  210.   - '*'
  211.   verbs:
  212.   - '*'
  213. - apiGroups:
  214.   - networking.k8s.io
  215.   resources:
  216.   - '*'
  217.   verbs:
  218.   - '*'
  219. - apiGroups:
  220.   - kubeedge.kubesphere.io
  221.   resources:
  222.   - '*'
  223.   verbs:
  224.   - '*'
  225. - apiGroups:
  226.   - types.kubefed.io
  227.   resources:
  228.   - '*'
  229.   verbs:
  230.   - '*'
  232. ---
  233. kind: ClusterRoleBinding
  234. apiVersion: rbac.authorization.k8s.io/v1
  235. metadata:
  236.   name: ks-installer
  237. subjects:
  238. - kind: ServiceAccount
  239.   name: ks-installer
  240.   namespace: kubesphere-system
  241. roleRef:
  242.   kind: ClusterRole
  243.   name: ks-installer
  244.   apiGroup: rbac.authorization.k8s.io
  246. ---
  247. apiVersion: apps/v1
  248. kind: Deployment
  249. metadata:
  250.   name: ks-installer
  251.   namespace: kubesphere-system
  252.   labels:
  253.     app: ks-install
  254. spec:
  255.   replicas: 1
  256.   selector:
  257.     matchLabels:
  258.       app: ks-install
  259.   template:
  260.     metadata:
  261.       labels:
  262.         app: ks-install
  263.     spec:
  264.       serviceAccountName: ks-installer
  265.       containers:
  266.       - name: installer
  267.         image: kubesphere/ks-installer:v3.1.1
  268.         imagePullPolicy: "Always"
  269.         resources:
  270.           limits:
  271.             cpu: "1"
  272.             memory: 1Gi
  273.           requests:
  274.             cpu: 20m
  275.             memory: 100Mi
  276.         volumeMounts:
  277.         - mountPath: /etc/localtime
  278.           name: host-time
  279.       volumes:
  280.       - hostPath:
  281.           path: /etc/localtime
  282.           type: ""
  283.         name: host-time
  1. ---
  2. apiVersion: installer.kubesphere.io/v1alpha1
  3. kind: ClusterConfiguration
  4. metadata:
  5.   name: ks-installer
  6.   namespace: kubesphere-system
  7.   labels:
  8.     version: v3.1.1
  9. spec:
  10.   persistence:
  11.     storageClass: ""        # If there is no default StorageClass in your cluster, you need to specify an existing StorageClass here.
  12.   authentication:
  13.     jwtSecret: ""           # Keep the jwtSecret consistent with the Host Cluster. Retrieve the jwtSecret by executing "kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret" on the Host Cluster.
  14.   local_registry: ""        # Add your private registry address if it is needed.
  15.   etcd:
  16.     monitoring: true       # Enable or disable etcd monitoring dashboard installation. You have to create a Secret for etcd before you enable it.
  17.     endpointIps: 1.15.230.38  # etcd cluster EndpointIps. It can be a bunch of IPs here.
  18.     port: 2379              # etcd port.
  19.     tlsEnable: true
  20.   common:
  21.     redis:
  22.       enabled: true
  23.     openldap:
  24.       enabled: true
  25.     minioVolumeSize: 20Gi # Minio PVC size.
  26.     openldapVolumeSize: 2Gi   # openldap PVC size.
  27.     redisVolumSize: 2Gi # Redis PVC size.
  28.     monitoring:
  29.       # type: external   # Whether to specify the external prometheus stack, and need to modify the endpoint at the next line.
  30.       endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090 # Prometheus endpoint to get metrics data.
  31.     es:   # Storage backend for logging, events and auditing.
  32.       # elasticsearchMasterReplicas: 1   # The total number of master nodes. Even numbers are not allowed.
  33.       # elasticsearchDataReplicas: 1     # The total number of data nodes.
  34.       elasticsearchMasterVolumeSize: 4Gi   # The volume size of Elasticsearch master nodes.
  35.       elasticsearchDataVolumeSize: 20Gi    # The volume size of Elasticsearch data nodes.
  36.       logMaxAge: 7                     # Log retention time in built-in Elasticsearch. It is 7 days by default.
  37.       elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log.
  38.       basicAuth:
  39.         enabled: false
  40.         username: ""
  41.         password: ""
  42.       externalElasticsearchUrl: ""
  43.       externalElasticsearchPort: ""
  44.   console:
  45.     enableMultiLogin: true  # Enable or disable simultaneous logins. It allows different users to log in with the same account at the same time.
  46.     port: 30880
  47.   alerting:                # (CPU: 0.1 Core, Memory: 100 MiB) It enables users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.
  48.     enabled: true         # Enable or disable the KubeSphere Alerting System.
  49.     # thanosruler:
  50.     #   replicas: 1
  51.     #   resources: {}
  52.   auditing:                # Provide a security-relevant chronological set of records，recording the sequence of activities happening on the platform, initiated by different tenants.
  53.     enabled: true         # Enable or disable the KubeSphere Auditing Log System. 
  54.   devops:                  # (CPU: 0.47 Core, Memory: 8.6 G) Provide an out-of-the-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image.
  55.     enabled: true             # Enable or disable the KubeSphere DevOps System.
  56.     jenkinsMemoryLim: 2Gi      # Jenkins memory limit.
  57.     jenkinsMemoryReq: 1500Mi   # Jenkins memory request.
  58.     jenkinsVolumeSize: 8Gi     # Jenkins volume size.
  59.     jenkinsJavaOpts_Xms: 512m  # The following three fields are JVM parameters.
  60.     jenkinsJavaOpts_Xmx: 512m
  61.     jenkinsJavaOpts_MaxRAM: 2g
  62.   events:                  # Provide a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.
  63.     enabled: true         # Enable or disable the KubeSphere Events System.
  64.     ruler:
  65.       enabled: true
  66.       replicas: 2
  67.   logging:                 # (CPU: 57 m, Memory: 2.76 G) Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.
  68.     enabled: true         # Enable or disable the KubeSphere Logging System.
  69.     logsidecar:
  70.       enabled: true
  71.       replicas: 2
  72.   metrics_server:                    # (CPU: 56 m, Memory: 44.35 MiB) It enables HPA (Horizontal Pod Autoscaler).
  73.     enabled: false                   # Enable or disable metrics-server.
  74.   monitoring:
  75.     storageClass: ""                 # If there is an independent StorageClass you need for Prometheus, you can specify it here. The default StorageClass is used by default.
  76.     # prometheusReplicas: 1          # Prometheus replicas are responsible for monitoring different segments of data source and providing high availability.
  77.     prometheusMemoryRequest: 400Mi   # Prometheus request memory.
  78.     prometheusVolumeSize: 20Gi       # Prometheus PVC size.
  79.     # alertmanagerReplicas: 1          # AlertManager Replicas.
  80.   multicluster:
  81.     clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the Host or Member Cluster.
  82.   network:
  83.     networkpolicy: # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).
  84.       # Make sure that the CNI network plugin used by the cluster supports NetworkPolicy. There are a number of CNI network plugins that support NetworkPolicy, including Calico, Cilium, Kube-router, Romana and Weave Net.
  85.       enabled: true # Enable or disable network policies.
  86.     ippool: # Use Pod IP Pools to manage the Pod network address space. Pods to be created can be assigned IP addresses from a Pod IP Pool.
  87.       type: none # Specify "calico" for this field if Calico is used as your CNI plugin. "none" means that Pod IP Pools are disabled.
  88.     topology: # Use Service Topology to view Service-to-Service communication based on Weave Scope.
  89.       type: none # Specify "weave-scope" for this field to enable Service Topology. "none" means that Service Topology is disabled.
  90.   openpitrix: # An App Store that is accessible to all platform tenants. You can use it to manage apps across their entire lifecycle.
  91.     store:
  92.       enabled: true # Enable or disable the KubeSphere App Store.
  93.   servicemesh:         # (0.3 Core, 300 MiB) Provide fine-grained traffic management, observability and tracing, and visualized traffic topology.
  94.     enabled: true     # Base component (pilot). Enable or disable KubeSphere Service Mesh (Istio-based).
  95.   kubeedge:          # Add edge nodes to your cluster and deploy workloads on edge nodes.
  96.     enabled: false   # Enable or disable KubeEdge.
  97.     cloudCore:
  98.       nodeSelector: {"node-role.kubernetes.io/worker": ""}
  99.       tolerations: []
  100.       cloudhubPort: "10000"
  101.       cloudhubQuicPort: "10001"
  102.       cloudhubHttpsPort: "10002"
  103.       cloudstreamPort: "10003"
  104.       tunnelPort: "10004"
  105.       cloudHub:
  106.         advertiseAddress: # At least a public IP address or an IP address which can be accessed by edge nodes must be provided.
  107.           - ""            # Note that once KubeEdge is enabled, CloudCore will malfunction if the address is not provided.
  108.         nodeLimit: "100"
  109.       service:
  110.         cloudhubNodePort: "30000"
  111.         cloudhubQuicNodePort: "30001"
  112.         cloudhubHttpsNodePort: "30002"
  113.         cloudstreamNodePort: "30003"
  114.         tunnelNodePort: "30004"
  115.     edgeWatcher:
  116.       nodeSelector: {"node-role.kubernetes.io/worker": ""}
  117.       tolerations: []
  118.       edgeWatcherAgent:
  119.         nodeSelector: {"node-role.kubernetes.io/worker": ""}
  120.         tolerations: []

3.0.0

  1. ---
  2. apiVersion: apiextensions.k8s.io/v1beta1
  3. kind: CustomResourceDefinition
  4. metadata:
  5.   name: clusterconfigurations.installer.kubesphere.io
  6. spec:
  7.   group: installer.kubesphere.io
  8.   versions:
  9.   - name: v1alpha1
  10.     served: true
  11.     storage: true
  12.   scope: Namespaced
  13.   names:
  14.     plural: clusterconfigurations
  15.     singular: clusterconfiguration
  16.     kind: ClusterConfiguration
  17.     shortNames:
  18.     - cc
  20. ---
  21. apiVersion: v1
  22. kind: Namespace
  23. metadata:
  24.   name: kubesphere-system
  26. ---
  27. apiVersion: v1
  28. kind: ServiceAccount
  29. metadata:
  30.   name: ks-installer
  31.   namespace: kubesphere-system
  33. ---
  34. apiVersion: rbac.authorization.k8s.io/v1
  35. kind: ClusterRole
  36. metadata:
  37.   name: ks-installer
  38. rules:
  39. - apiGroups:
  40.   - ""
  41.   resources:
  42.   - '*'
  43.   verbs:
  44.   - '*'
  45. - apiGroups:
  46.   - apps
  47.   resources:
  48.   - '*'
  49.   verbs:
  50.   - '*'
  51. - apiGroups:
  52.   - extensions
  53.   resources:
  54.   - '*'
  55.   verbs:
  56.   - '*'
  57. - apiGroups:
  58.   - batch
  59.   resources:
  60.   - '*'
  61.   verbs:
  62.   - '*'
  63. - apiGroups:
  64.   - rbac.authorization.k8s.io
  65.   resources:
  66.   - '*'
  67.   verbs:
  68.   - '*'
  69. - apiGroups:
  70.   - apiregistration.k8s.io
  71.   resources:
  72.   - '*'
  73.   verbs:
  74.   - '*'
  75. - apiGroups:
  76.   - apiextensions.k8s.io
  77.   resources:
  78.   - '*'
  79.   verbs:
  80.   - '*'
  81. - apiGroups:
  82.   - tenant.kubesphere.io
  83.   resources:
  84.   - '*'
  85.   verbs:
  86.   - '*'
  87. - apiGroups:
  88.   - certificates.k8s.io
  89.   resources:
  90.   - '*'
  91.   verbs:
  92.   - '*'
  93. - apiGroups:
  94.   - devops.kubesphere.io
  95.   resources:
  96.   - '*'
  97.   verbs:
  98.   - '*'
  99. - apiGroups:
  100.   - monitoring.coreos.com
  101.   resources:
  102.   - '*'
  103.   verbs:
  104.   - '*'
  105. - apiGroups:
  106.   - logging.kubesphere.io
  107.   resources:
  108.   - '*'
  109.   verbs:
  110.   - '*'
  111. - apiGroups:
  112.   - jaegertracing.io
  113.   resources:
  114.   - '*'
  115.   verbs:
  116.   - '*'
  117. - apiGroups:
  118.   - storage.k8s.io
  119.   resources:
  120.   - '*'
  121.   verbs:
  122.   - '*'
  123. - apiGroups:
  124.   - admissionregistration.k8s.io
  125.   resources:
  126.   - '*'
  127.   verbs:
  128.   - '*'
  129. - apiGroups:
  130.   - policy
  131.   resources:
  132.   - '*'
  133.   verbs:
  134.   - '*'
  135. - apiGroups:
  136.   - autoscaling
  137.   resources:
  138.   - '*'
  139.   verbs:
  140.   - '*'
  141. - apiGroups:
  142.   - networking.istio.io
  143.   resources:
  144.   - '*'
  145.   verbs:
  146.   - '*'
  147. - apiGroups:
  148.   - config.istio.io
  149.   resources:
  150.   - '*'
  151.   verbs:
  152.   - '*'
  153. - apiGroups:
  154.   - iam.kubesphere.io
  155.   resources:
  156.   - '*'
  157.   verbs:
  158.   - '*'
  159. - apiGroups:
  160.   - notification.kubesphere.io
  161.   resources:
  162.   - '*'
  163.   verbs:
  164.   - '*'
  165. - apiGroups:
  166.   - auditing.kubesphere.io
  167.   resources:
  168.   - '*'
  169.   verbs:
  170.   - '*'
  171. - apiGroups:
  172.   - events.kubesphere.io
  173.   resources:
  174.   - '*'
  175.   verbs:
  176.   - '*'
  177. - apiGroups:
  178.   - core.kubefed.io
  179.   resources:
  180.   - '*'
  181.   verbs:
  182.   - '*'
  183. - apiGroups:
  184.   - installer.kubesphere.io
  185.   resources:
  186.   - '*'
  187.   verbs:
  188.   - '*'
  189. - apiGroups:
  190.   - storage.kubesphere.io
  191.   resources:
  192.   - '*'
  193.   verbs:
  194.   - '*'
  196. ---
  197. kind: ClusterRoleBinding
  198. apiVersion: rbac.authorization.k8s.io/v1
  199. metadata:
  200.   name: ks-installer
  201. subjects:
  202. - kind: ServiceAccount
  203.   name: ks-installer
  204.   namespace: kubesphere-system
  205. roleRef:
  206.   kind: ClusterRole
  207.   name: ks-installer
  208.   apiGroup: rbac.authorization.k8s.io
  210. ---
  211. apiVersion: apps/v1
  212. kind: Deployment
  213. metadata:
  214.   name: ks-installer
  215.   namespace: kubesphere-system
  216.   labels:
  217.     app: ks-install
  218. spec:
  219.   replicas: 1
  220.   selector:
  221.     matchLabels:
  222.       app: ks-install
  223.   template:
  224.     metadata:
  225.       labels:
  226.         app: ks-install
  227.     spec:
  228.       serviceAccountName: ks-installer
  229.       containers:
  230.       - name: installer
  231.         image: kubesphere/ks-installer:v3.0.0
  232.         imagePullPolicy: "Always"
  233.         volumeMounts:
  234.         - mountPath: /etc/localtime
  235.           name: host-time
  236.       volumes:
  237.       - hostPath:
  238.           path: /etc/localtime
  239.           type: ""
  240.         name: host-time
  1. ---
  2. apiVersion: installer.kubesphere.io/v1alpha1
  3. kind: ClusterConfiguration
  4. metadata:
  5.   name: ks-installer
  6.   namespace: kubesphere-system
  7.   labels:
  8.     version: v3.0.0
  9. spec:
  10.   persistence:
  11.     storageClass: ""        # If there is not a default StorageClass in your cluster, you need to specify an existing StorageClass here.
  12.   authentication:
  13.     jwtSecret: ""           # Keep the jwtSecret consistent with the host cluster. Retrive the jwtSecret by executing "kubectl -n kubesphere-system get cm kubesphere-config -o yaml | grep -v "apiVersion" | grep jwtSecret" on the host cluster.
  14.   etcd:
  15.     monitoring: true       # Whether to enable etcd monitoring dashboard installation. You have to create a secret for etcd before you enable it.
  16.     endpointIps: localhost  # etcd cluster EndpointIps, it can be a bunch of IPs here.
  17.     port: 2379              # etcd port
  18.     tlsEnable: true
  19.   common:
  20.     mysqlVolumeSize: 20Gi # MySQL PVC size.
  21.     minioVolumeSize: 20Gi # Minio PVC size.
  22.     etcdVolumeSize: 20Gi  # etcd PVC size.
  23.     openldapVolumeSize: 2Gi   # openldap PVC size.
  24.     redisVolumSize: 2Gi # Redis PVC size.
  25.     es:   # Storage backend for logging, events and auditing.
  26.       # elasticsearchMasterReplicas: 1   # total number of master nodes, it's not allowed to use even number
  27.       # elasticsearchDataReplicas: 1     # total number of data nodes.
  28.       elasticsearchMasterVolumeSize: 4Gi   # Volume size of Elasticsearch master nodes.
  29.       elasticsearchDataVolumeSize: 20Gi    # Volume size of Elasticsearch data nodes.
  30.       logMaxAge: 7                     # Log retention time in built-in Elasticsearch, it is 7 days by default.
  31.       elkPrefix: logstash              # The string making up index names. The index name will be formatted as ks-<elk_prefix>-log.
  32.   console:
  33.     enableMultiLogin: true  # enable/disable multiple sing on, it allows an account can be used by different users at the same time.
  34.     port: 30880
  35.   alerting:                # (CPU: 0.3 Core, Memory: 300 MiB) Whether to install KubeSphere alerting system. It enables Users to customize alerting policies to send messages to receivers in time with different time intervals and alerting levels to choose from.
  36.     enabled: true
  37.   auditing:                # Whether to install KubeSphere audit log system. It provides a security-relevant chronological set of records，recording the sequence of activities happened in platform, initiated by different tenants.
  38.     enabled: true
  39.   devops:                  # (CPU: 0.47 Core, Memory: 8.6 G) Whether to install KubeSphere DevOps System. It provides out-of-box CI/CD system based on Jenkins, and automated workflow tools including Source-to-Image & Binary-to-Image.
  40.     enabled: true
  41.     jenkinsMemoryLim: 2Gi      # Jenkins memory limit.
  42.     jenkinsMemoryReq: 1500Mi   # Jenkins memory request.
  43.     jenkinsVolumeSize: 8Gi     # Jenkins volume size.
  44.     jenkinsJavaOpts_Xms: 512m  # The following three fields are JVM parameters.
  45.     jenkinsJavaOpts_Xmx: 512m
  46.     jenkinsJavaOpts_MaxRAM: 2g
  47.   events:                  # Whether to install KubeSphere events system. It provides a graphical web console for Kubernetes Events exporting, filtering and alerting in multi-tenant Kubernetes clusters.
  48.     enabled: true
  49.     ruler:
  50.       enabled: true
  51.       replicas: 2
  52.   logging:                 # (CPU: 57 m, Memory: 2.76 G) Whether to install KubeSphere logging system. Flexible logging functions are provided for log query, collection and management in a unified console. Additional log collectors can be added, such as Elasticsearch, Kafka and Fluentd.
  53.     enabled: true
  54.     logsidecarReplicas: 2
  55.   metrics_server:                    # (CPU: 56 m, Memory: 44.35 MiB) Whether to install metrics-server. IT enables HPA (Horizontal Pod Autoscaler).
  56.     enabled: false
  57.   monitoring:
  58.     # prometheusReplicas: 1            # Prometheus replicas are responsible for monitoring different segments of data source and provide high availability as well.
  59.     prometheusMemoryRequest: 400Mi   # Prometheus request memory.
  60.     prometheusVolumeSize: 20Gi       # Prometheus PVC size.
  61.     # alertmanagerReplicas: 1          # AlertManager Replicas.
  62.   multicluster:
  63.     clusterRole: none  # host | member | none  # You can install a solo cluster, or specify it as the role of host or member cluster.
  64.   networkpolicy:       # Network policies allow network isolation within the same cluster, which means firewalls can be set up between certain instances (Pods).
  65.     # Make sure that the CNI network plugin used by the cluster supports NetworkPolicy. There are a number of CNI network plugins that support NetworkPolicy, including Calico, Cilium, Kube-router, Romana and Weave Net.
  66.     enabled: true
  67.   notification:        # Email Notification support for the legacy alerting system, should be enabled/disabled together with the above alerting option.
  68.     enabled: false
  69.   openpitrix:          # (2 Core, 3.6 G) Whether to install KubeSphere Application Store. It provides an application store for Helm-based applications, and offer application lifecycle management.
  70.     enabled: true
  71.   servicemesh:         # (0.3 Core, 300 MiB) Whether to install KubeSphere Service Mesh (Istio-based). It provides fine-grained traffic management, observability and tracing, and offer visualization for traffic topology.
  72.     enabled: true

安装

  1. kubectl apply -f kubesphere-installer.yml
  3. kubectl apply -f cluster-configuration.yml

image.png

查看进度

查看pod进度

  1. watch -n 1 kubectl get pods -o wide --all-namespaces

查看安装进度日志

  1. kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

问题排查
image.png
image.png

  1. kubectl describe pod -n kubesphere-monitoring-system   prometheus-k8s-0

解决

解决etcd监控证书找不到问题

  1. kubectl -n kubesphere-monitoring-system create secret generic kube-etcd-client-certs  --from-file=etcd-client-ca.crt=/etc/kubernetes/pki/etcd/ca.crt  --from-file=etcd-client.crt=/etc/kubernetes/pki/apiserver-etcd-client.crt  --from-file=etcd-client.key=/etc/kubernetes/pki/apiserver-etcd-client.key

验证