1. 监控 etcd 集群

1.1 查看接口信息

  1. curl --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem  https://10.0.0.11:2379/metrics -k 
  3. curl -L http://localhost:2379/metrics

1.2 创建 Service 和 Endpoints

  1. apiVersion: v1
  2. kind: Service 
  3. metadata:
  4.   labels:
  5.     app: etcd-k8s
  6.   name: etcd-k8s
  7.   namespace: kube-system
  8. spec:
  9.   ports:
  10.   - name: etcd-port
  11.     port: 2379
  12.     protocol: TCP
  13.     targetPort: 2379
  14.   type: ClusterIP
  15. ---
  16. apiVersion: v1
  17. kind: Endpoints
  18. metadata:
  19.   labels:
  20.     app: etcd-k8s
  21.   name: etcd-k8s
  22.   namespace: kube-system
  23. subsets:
  24. - addresses:     # etcd节点对应的主机ip，有几台就写几台
  25.   - ip: 10.0.0.11
  26.   - ip: 10.0.0.12
  27.   - ip: 10.0.0.21
  28.   ports:
  29.   - name: etcd-port
  30.     port: 2379
  31.     protocol: TCP

1.3 测试是否代理成功

  1. # 查看 svc 和 ep
  2. [root@k8s-master01 ~]# kubectl get svc,ep -n kube-system etcd-k8s
  3. NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
  4. service/etcd-k8s   ClusterIP   10.97.194.106   <none>        2379/TCP   46s
  6. NAME                 ENDPOINTS                                      AGE
  7. endpoints/etcd-k8s   10.0.0.11:2379,10.0.0.12:2379,10.0.0.21:2379   46s
  10. # 再次请求，将IP换成svc的IP测试
  11. [root@k8s-master01 ~]# curl --cert /etc/etcd/ssl/etcd.pem --key /etc/etcd/ssl/etcd-key.pem  https://10.97.194.106:2379/metrics -k

1.4 创建 secret

  1. # 创建etcd-ssl secret，注意证书路径
  2. kubectl create secret generic etcd-ssl --from-file=/etc/kubernetes/pki/etcd/etcd-ca.pem --from-file=/etc/kubernetes/pki/etcd/etcd.pem --from-file=/etc/kubernetes/pki/etcd/etcd-key.pem -n monitoring
  4. # 查看 secret
  5. [root@k8s-master01 ~]# kubectl describe secrets -n monitoring etcd-ssl
  6. Name:         etcd-ssl
  7. Namespace:    monitoring
  8. Labels:       <none>
  9. Annotations:  <none>
  11. Type:  Opaque
  13. Data
  14. ====
  15. etcd-ca.pem:          1330 bytes
  16. etcd-server-key.pem:  1679 bytes
  17. etcd-server.pem:      1460 bytes

1.5 编辑 prometheus 挂载证书

  1. # 编辑 prometheus
  2. [root@k8s-master01 ~]# kubectl edit prometheus k8s -n monitoring
  3. ...
  4. spec:
  5.   secrets:
  6.   - etcd-ssl # secret名称
  7. ...
  9. # 进入容器查看，证书是否已经挂载
  10. [root@k8s-master01 ~]# kubectl exec -it -n monitoring prometheus-k8s-0 -c prometheus -- ls /etc/prometheus/secrets/etcd-ssl/
  11. etcd-ca.pem          etcd-server-key.pem  etcd-server.pem

1.6 创建ServiceMonitor

  1. apiVersion: monitoring.coreos.com/v1
  2. kind: ServiceMonitor
  3. metadata:
  4.   name: etcd-k8s
  5.   namespace: monitoring
  6.   labels:
  7.     app: etcd-k8s
  8. spec:
  9.   jobLabel: app
  10.   endpoints:
  11.     - interval: 30s
  12.       port: etcd-port  # 对应 Service.spec.ports.name
  13.       scheme: https
  14.       tlsConfig:
  15.         caFile: /etc/prometheus/secrets/etcd-ssl/etcd-ca.pem # 证书路径，prometheus pod 里的路径
  16.         certFile: /etc/prometheus/secrets/etcd-ssl/etcd.pem
  17.         keyFile: /etc/prometheus/secrets/etcd-ssl/etcd-key.pem
  18.         insecureSkipVerify: true  # 关闭证书校验，证书serverName和etcd中签发的证书可能不匹配，添加此行后将不再对服务端的证书进行校验
  19.   selector:
  20.     matchLabels:
  21.       app: etcd-k8s  # 和scv的lables保持一致
  22.   namespaceSelector:
  23.     matchNames:
  24.     - kube-system    # 和svc所在namespace保持一致

查看ServiceMonitor：

  1. [root@k8s-master01 ~]# kubectl get servicemonitors etcd-k8s -n monitoring
  2. NAME       AGE
  3. etcd-k8s   6m32s

1.7 查看 Prometheus 页面

打开Prometheus Web端，按下面的路径Status -> targets -> monitoring/etcd-k8s寻找etcd监控信息。（获取**metrcis**需要一定时间，需要稍等一会）
image.png

1.8 Grafana 监控模板导入

打开仪表盘市场，找到etcd对应模板并导入。常用的模板有Etcd by Prometheus和Etcd-for-k8s-cn中文。然后打开Grafana Web页面，点击 **+** -> import -> load即可。

2. 监控 Kafka 集群