一.部署集群内prometheus监控节点
##安装node-exporter
kubectl appy -f node-exporter.yaml

  1. apiVersion: apps/v1
  2. kind: DaemonSet
  3. metadata:
  4.   name: node-exporter
  5.   namespace: kube-system
  6.   labels:
  7.     k8s-app: node-exporter
  8. spec:
  9.   selector:
  10.     matchLabels:
  11.       k8s-app: node-exporter
  12.   template:
  13.     metadata:
  14.       labels:
  15.         k8s-app: node-exporter
  16.     spec:
  17.       containers:
  18.       - image: prom/node-exporter
  19.         name: node-exporter
  20.         ports:
  21.         - containerPort: 9100
  22.           protocol: TCP
  23.           name: http
  24.       tolerations:
  25.       hostNetwork: true
  26.       hostPID: true
  27.       hostIPC: true
  28.       restartPolicy: Always

安装prometheus

kubectl apply -f configmap.yaml ##准备配置文件

  1. apiVersion: v1
  2. kind: ConfigMap
  3. metadata:
  4.   name: prometheus-config
  5.   namespace: kube-system
  6. data:
  7.   prometheus.yml: |
  8.     global:
  9.       scrape_interval:     15s
  10.       evaluation_interval: 15s
  11.     scrape_configs:
  13.     - job_name: 'kubernetes-apiservers'
  14.       kubernetes_sd_configs:
  15.       - role: endpoints
  16.       scheme: https
  17.       tls_config:
  18.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  19.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  20.       relabel_configs:
  21.       - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
  22.         action: keep
  23.         regex: default;kubernetes;https
  25.     - job_name: 'kubernetes-nodes'
  26.       kubernetes_sd_configs:
  27.       - role: node
  28.       scheme: https
  29.       tls_config:
  30.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  31.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  32.       relabel_configs:
  33.       - action: labelmap
  34.         regex: __meta_kubernetes_node_label_(.+)
  35.       - target_label: __address__
  36.         replacement: kubernetes.default.svc:443
  37.       - source_labels: [__meta_kubernetes_node_name]
  38.         regex: (.+)
  39.         target_label: __metrics_path__
  40.         replacement: /api/v1/nodes/${1}/proxy/metrics
  42.     - job_name: 'kubernetes-cadvisor'
  43.       kubernetes_sd_configs:
  44.       - role: node
  45.       scheme: https
  46.       tls_config:
  47.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  48.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  49.       relabel_configs:
  50.       - action: labelmap
  51.         regex: __meta_kubernetes_node_label_(.+)
  52.       - target_label: __address__
  53.         replacement: kubernetes.default.svc:443
  54.       - source_labels: [__meta_kubernetes_node_name]
  55.         regex: (.+)
  56.         target_label: __metrics_path__
  57.         replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
  59.     - job_name: 'kubernetes-service-endpoints'
  60.       kubernetes_sd_configs:
  61.       - role: endpoints
  62.       relabel_configs:
  63.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
  64.         action: keep
  65.         regex: true
  66.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
  67.         action: replace
  68.         target_label: __scheme__
  69.         regex: (https?)
  70.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
  71.         action: replace
  72.         target_label: __metrics_path__
  73.         regex: (.+)
  74.       - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
  75.         action: replace
  76.         target_label: __address__
  77.         regex: ([^:]+)(?::\d+)?;(\d+)
  78.         replacement: $1:$2
  79.       - action: labelmap
  80.         regex: __meta_kubernetes_service_label_(.+)
  81.       - source_labels: [__meta_kubernetes_namespace]
  82.         action: replace
  83.         target_label: kubernetes_namespace
  84.       - source_labels: [__meta_kubernetes_service_name]
  85.         action: replace
  86.         target_label: kubernetes_name
  88.     - job_name: 'kubernetes-services'
  89.       kubernetes_sd_configs:
  90.       - role: service
  91.       metrics_path: /probe
  92.       params:
  93.         module: [http_2xx]
  94.       relabel_configs:
  95.       - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
  96.         action: keep
  97.         regex: true
  98.       - source_labels: [__address__]
  99.         target_label: __param_target
  100.       - target_label: __address__
  101.         replacement: blackbox-exporter.example.com:9115
  102.       - source_labels: [__param_target]
  103.         target_label: instance
  104.       - action: labelmap
  105.         regex: __meta_kubernetes_service_label_(.+)
  106.       - source_labels: [__meta_kubernetes_namespace]
  107.         target_label: kubernetes_namespace
  108.       - source_labels: [__meta_kubernetes_service_name]
  109.         target_label: kubernetes_name
  111.     - job_name: 'kubernetes-ingresses'
  112.       kubernetes_sd_configs:
  113.       - role: ingress
  114.       relabel_configs:
  115.       - source_labels: [__meta_kubernetes_ingress_annotation_prometheus_io_probe]
  116.         action: keep
  117.         regex: true
  118.       - source_labels: [__meta_kubernetes_ingress_scheme,__address__,__meta_kubernetes_ingress_path]
  119.         regex: (.+);(.+);(.+)
  120.         replacement: ${1}://${2}${3}
  121.         target_label: __param_target
  122.       - target_label: __address__
  123.         replacement: blackbox-exporter.example.com:9115
  124.       - source_labels: [__param_target]
  125.         target_label: instance
  126.       - action: labelmap
  127.         regex: __meta_kubernetes_ingress_label_(.+)
  128.       - source_labels: [__meta_kubernetes_namespace]
  129.         target_label: kubernetes_namespace
  130.       - source_labels: [__meta_kubernetes_ingress_name]
  131.         target_label: kubernetes_name
  133.     - job_name: 'kubernetes-pods'
  134.       kubernetes_sd_configs:
  135.       - role: pod
  136.       relabel_configs:
  137.       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
  138.         action: keep
  139.         regex: true
  140.       - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
  141.         action: replace
  142.         target_label: __metrics_path__
  143.         regex: (.+)
  144.       - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
  145.         action: replace
  146.         regex: ([^:]+)(?::\d+)?;(\d+)
  147.         replacement: $1:$2
  148.         target_label: __address__
  149.       - action: labelmap
  150.         regex: __meta_kubernetes_pod_label_(.+)
  151.       - source_labels: [__meta_kubernetes_namespace]
  152.         action: replace
  153.         target_label: kubernetes_namespace
  154.       - source_labels: [__meta_kubernetes_pod_name]
  155.         action: replace
  156.         target_label: kubernetes_pod_name
  157.     - job_name: 'kubernetes-schedule'
  158.       scrape_interval: 5s
  159.       static_configs:
  160.         - targets: ['10.107.141.50:10251']
  162.     - job_name: 'kubernetes-control-manager'
  163.       scrape_interval: 5s
  164.       static_configs:
  165.         - targets: ['10.107.141.50:10252']
  167.     - job_name: 'kubernetes-kubelet'
  168.       scrape_interval: 5s
  169.       static_configs:
  170.         - targets: ['10.107.141.51:10255','10.107.141.52:10255']
  172.     - job_name: 'kubernetes-kube-proxy'
  173.       scrape_interval: 5s
  174.       static_configs:
  175.         - targets: ['10.107.141.51:10249','10.107.141.52:10249']

kubectl apply -f rbac.yaml ##准备deployment使用的rbac认证文件,给peomrtheus权限监控集群

  1. apiVersion: rbac.authorization.k8s.io/v1
  2. kind: ClusterRole
  3. metadata:
  4.   name: prometheus
  5. rules:
  6. - apiGroups: [""]
  7.   resources:
  8.   - nodes
  9.   - nodes/proxy
  10.   - services
  11.   - endpoints
  12.   - pods
  13.   verbs: ["get", "list", "watch"]
  14. - apiGroups:
  15.   - extensions
  16.   resources:
  17.   - ingresses
  18.   verbs: ["get", "list", "watch"]
  19. - nonResourceURLs: ["/metrics"]
  20.   verbs: ["get"]
  21. ---
  22. apiVersion: v1
  23. kind: ServiceAccount
  24. metadata:
  25.   name: prometheus
  26.   namespace: kube-system
  27. ---
  28. apiVersion: rbac.authorization.k8s.io/v1
  29. kind: ClusterRoleBinding
  30. metadata:
  31.   name: prometheus
  32. roleRef:
  33.   apiGroup: rbac.authorization.k8s.io
  34.   kind: ClusterRole
  35.   name: prometheus
  36. subjects:
  37. - kind: ServiceAccount
  38.   name: prometheus
  39.   namespace: kube-system

kubectl apply -f prometheus_deploy.yml

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   labels:
  5.     name: prometheus-deployment
  6.   name: prometheus
  7.   namespace: kube-system
  8. spec:
  9.   replicas: 1
  10.   selector:
  11.     matchLabels:
  12.       app: prometheus
  13.   template:
  14.     metadata:
  15.       labels:
  16.         app: prometheus
  17.     spec:
  18.       containers:
  19.       - image: prom/prometheus
  20.         name: prometheus
  21.         imagePullPolicy: IfNotPresent
  22.         command:
  23.         - "/bin/prometheus"
  24.         args:
  25.         - "--config.file=/etc/prometheus/prometheus.yml"
  26.         - "--storage.tsdb.path=/prometheus"
  27.         - "--storage.tsdb.retention=24h"
  28.         ports:
  29.         - containerPort: 9090
  30.           protocol: TCP
  31.         volumeMounts:
  32.         - mountPath: "/prometheus"
  33.           name: data
  34.         - mountPath: "/etc/prometheus"
  35.           name: config-volume
  36.         resources:
  37.           requests:
  38.             cpu: 100m
  39.             memory: 100Mi
  40.           limits:
  41.             cpu: 500m
  42.             memory: 2500Mi
  43.       serviceAccountName: prometheus
  44.       volumes:
  45.       - name: data
  46.         emptyDir: {}
  47.       - name: config-volume
  48.         configMap:
  49.           name: prometheus-config

kubectl apply -f prometheus_svc.yml

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   labels:
  5.     app: prometheus
  6.   name: prometheus
  7.   namespace: kube-system
  8. spec:
  9.   type: NodePort
  10.   ports:
  11.   - port: 9090
  12.     targetPort: 9090
  13.     nodePort: 30003
  14.   selector:
  15.     app: prometheus

kubectl get deployment -o wide -n kube-system
2B9`[U]BI1C8JHLTX@`8H4V.png
kubectl get svc -o wide -n kube-system
`Y(BE{LH[J}R]A4~9~7%RMG.png
##安装grafana
kubectl apply -f grafana_deploy.yaml

  1. apiVersion: apps/v1
  2. kind: Deployment
  3. metadata:
  4.   name: grafana-core
  5.   namespace: kube-system
  6.   labels:
  7.     app: grafana
  8.     component: core
  9. spec:
  10.   selector:
  11.     matchLabels:
  12.       app: grafana
  13.       component: core
  14.   replicas: 1
  15.   template:
  16.     metadata:
  17.       labels:
  18.         app: grafana
  19.         component: core
  20.     spec:
  21.       containers:
  22.       - image: grafana/grafana:4.2.0
  23.         name: grafana-core
  24.         imagePullPolicy: IfNotPresent
  25.         # env:
  26.         resources:
  27.           # keep request = limit to keep this container in guaranteed class
  28.           limits:
  29.             cpu: 100m
  30.             memory: 100Mi
  31.           requests:
  32.             cpu: 100m
  33.             memory: 100Mi
  34.         env:
  35.           # The following env variables set up basic auth twith the default admin user and admin password.
  36.           - name: GF_AUTH_BASIC_ENABLED
  37.             value: "true"
  38.           - name: GF_AUTH_ANONYMOUS_ENABLED
  39.             value: "false"
  40.           # - name: GF_AUTH_ANONYMOUS_ORG_ROLE
  41.           #   value: Admin
  42.           # does not really work, because of template variables in exported dashboards:
  43.           # - name: GF_DASHBOARDS_JSON_ENABLED
  44.           #   value: "true"
  45.         readinessProbe:
  46.           httpGet:
  47.             path: /login
  48.             port: 3000
  49.           # initialDelaySeconds: 30
  50.           # timeoutSeconds: 1
  51.         volumeMounts:
  52.         - name: grafana-persistent-storage
  53.           mountPath: /var
  54.       volumes:
  55.       - name: grafana-persistent-storage
  56.         emptyDir: {}

kubectl apply -f grafana_deploy.yaml

  1. apiVersion: v1
  2. kind: Service
  3. metadata:
  4.   name: grafana
  5.   namespace: kube-system
  6.   labels:
  7.     app: grafana
  8.     component: core
  9. spec:
  10.   type: NodePort
  11.   ports:
  12.     - port: 3000
  13.   selector:
  14.     app: grafana
  15.     component: core

测试访问：
~4H4VC@`~0{Y$]6DMP0)QVB.png
{VH$J`L(`6E~19@YR2G3Y3X.png
注：如果prometehsu采集不到kube-proxy节点，则有可能是kube-proxy的metrics数据走的
是127.0.0.1:10249端口，无法被外部采集到。可使用netstat -lntp | grep 10249查看,正确
结果如图所示
![AMR0~C9VI29T4`~ZKX]}C4.png
如采用二进制systemd安装方法可以去kube-proxy.conf文件，添加—metrics-bind-address=本机ip
N[0P0H9@[VSSV]GFCN~3QIJ.png
二.部署集群外prometheus的global节点

集群外直接用docker部署global节点: (此处监控的是集群内kube-proxy和kubelet组件，根据job_name指定）

  1. global:
  2.   scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  3.   evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  4.   # scrape_timeout is set to the global default (10s).
  6. alerting:
  7.   alertmanagers:
  8.   - static_configs:
  9.     - targets:
  10.       # - alertmanager:9093
  12. rule_files:
  13. scrape_configs:
  14.   - job_name: 'federate'
  15.     scrape_interval: 15s
  16.     honor_labels: true
  17.     metrics_path: '/federate'
  18.     params:
  19.       'match[]':
  20.         - '{job="kubernetes-kube-proxy"}'
  21.         - '{job="kubernetes-kubelet"}'
  22.     static_configs:
  23.       - targets:
  24.         - '10.107.141.51:30003'

运行集群外本地prometheus
docker run -d —name=prometheus -p 9090:9090 -v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

进入集群外prometheus查看
(0HSEZ{L]G4JIX]5V{(]W`W.png