说明

Kubernetes的kubelet组件内置了cadvisor，将Node上容器的指标以Prometheus支持的格式展示，可以通过这些指标计算得到更多有用的数据。

访问:10255/metrics/cadvisor，可以读取到以Prometheus支持的格式呈现的指标：
image.png
在prometheus的配置文件中配置相关的target之后，这些指标就可以从Prometheus中查询到

  1. ...
  2.     - job_name: kubernetes-apiservers
  3.       kubernetes_sd_configs:
  4.       - role: endpoints
  5.       relabel_configs:
  6.       - action: keep
  7.         regex: default;kubernetes;https
  8.         source_labels:
  9.         - __meta_kubernetes_namespace
  10.         - __meta_kubernetes_service_name
  11.         - __meta_kubernetes_endpoint_port_name
  12.       scheme: https
  13.       tls_config:
  14.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  15.         insecure_skip_verify: true
  16.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  18.     - job_name: kubernetes-nodes-kubelet
  19.       kubernetes_sd_configs:
  20.       - role: node
  21.       relabel_configs:
  22.       - action: labelmap
  23.         regex: __meta_kubernetes_node_label_(.+)
  24.       scheme: https
  25.       tls_config:
  26.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  27.         insecure_skip_verify: true
  28.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  30.     - job_name: kubernetes-nodes-cadvisor
  31.       kubernetes_sd_configs:
  32.       - role: node
  33.       relabel_configs:
  34.       - action: labelmap
  35.         regex: __meta_kubernetes_node_label_(.+)
  36.       - target_label: __metrics_path__
  37.         replacement: /metrics/cadvisor
  38.       scheme: https
  39.       tls_config:
  40.         ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
  41.         insecure_skip_verify: true
  42.       bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  44.     - job_name: kubernetes-service-endpoints
  45.       kubernetes_sd_configs:
  46.       - role: endpoints
  47.       relabel_configs:
  48.       - action: keep
  49.         regex: true
  50.         source_labels:
  51.         - __meta_kubernetes_service_annotation_prometheus_io_scrape
  52.       - action: replace
  53.         regex: (https?)
  54.         source_labels:
  55.         - __meta_kubernetes_service_annotation_prometheus_io_scheme
  56.         target_label: __scheme__
  57.       - action: replace
  58.         regex: (.+)
  59.         source_labels:
  60.         - __meta_kubernetes_service_annotation_prometheus_io_path
  61.         target_label: __metrics_path__
  62.       - action: replace
  63.         regex: ([^:]+)(?::\d+)?;(\d+)
  64.         replacement: $1:$2
  65.         source_labels:
  66.         - __address__
  67.         - __meta_kubernetes_service_annotation_prometheus_io_port
  68.         target_label: __address__
  69.       - action: labelmap
  70.         regex: __meta_kubernetes_service_label_(.+)
  71.       - action: replace
  72.         source_labels:
  73.         - __meta_kubernetes_namespace
  74.         target_label: kubernetes_namespace
  75.       - action: replace
  76.         source_labels:
  77.         - __meta_kubernetes_service_name
  78.         target_label: kubernetes_name
  80.     - job_name: kubernetes-services
  81.       kubernetes_sd_configs:
  82.       - role: service
  83.       metrics_path: /probe
  84.       params:
  85.         module:
  86.         - http_2xx
  87.       relabel_configs:
  88.       - action: keep
  89.         regex: true
  90.         source_labels:
  91.         - __meta_kubernetes_service_annotation_prometheus_io_probe
  92.       - source_labels:
  93.         - __address__
  94.         target_label: __param_target
  95.       - replacement: blackbox
  96.         target_label: __address__
  97.       - source_labels:
  98.         - __param_target
  99.         target_label: instance
  100.       - action: labelmap
  101.         regex: __meta_kubernetes_service_label_(.+)
  102.       - source_labels:
  103.         - __meta_kubernetes_namespace
  104.         target_label: kubernetes_namespace
  105.       - source_labels:
  106.         - __meta_kubernetes_service_name
  107.         target_label: kubernetes_name
  109.     - job_name: kubernetes-pods
  110.       kubernetes_sd_configs:
  111.       - role: pod
  112.       relabel_configs:
  113.       - action: keep
  114.         regex: true
  115.         source_labels:
  116.         - __meta_kubernetes_pod_annotation_prometheus_io_scrape
  117.       - action: replace
  118.         regex: (.+)
  119.         source_labels:
  120.         - __meta_kubernetes_pod_annotation_prometheus_io_path
  121.         target_label: __metrics_path__
  122.       - action: replace
  123.         regex: ([^:]+)(?::\d+)?;(\d+)
  124.         replacement: $1:$2
  125.         source_labels:
  126.         - __address__
  127.         - __meta_kubernetes_pod_annotation_prometheus_io_port
  128.         target_label: __address__
  129.       - action: labelmap
  130.         regex: __meta_kubernetes_pod_label_(.+)
  131.       - action: replace
  132.         source_labels:
  133.         - __meta_kubernetes_namespace
  134.         target_label: kubernetes_namespace
  135.       - action: replace
  136.         source_labels:
  137.         - __meta_kubernetes_pod_name
  138.         target_label: kubernetes_pod_name
  139. ...

容器CPU使用率的计算

从man top手册中找到了CPU使用率的定义：

  1. %CPU  --  CPU Usage
  2.   The task's share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time.
  4.   In a true SMP environment, if a process is multi-threaded and top is not operating in Threads mode, amounts greater
  5.   than 100% may be reported.  You toggle Threads mode with the `H' inter-active command.
  7.   Also for multi-processor environments, if Irix mode is Off, top will operate in Solaris mode where a task's cpu usage
  8.   will be divided by the total number of CPUs.  You toggle Irix/Solaris modes with the `I' interactive command.

即在过去的一段时间里进程占用的CPU时间与CPU总时间的比率，如果有多个CPU或者多核，需要将每个CPU的时间相加

kubelet中的cadvisor采集的指标，其中有一项是

  1. container_cpu_usage_seconds_total     Counter     Cumulative cpu time consumed     seconds

container_cpu_usage_seconds_total是container累计使用的CPU时间，用它除以CPU的总时间，就得到了容器的CPU使用率

  1. # 每分钟内的变化
  2. sum (rate (container_cpu_usage_seconds_total{image!="",name=~"^k8s_.*",io_kubernetes_container_name!="POD",pod_name=~"^$Deployment$Statefulset$Daemonset.*$",kubernetes_io_hostname=~"^$Node$"}[1m])) by (pod_name,kubernetes_io_hostname)

内存使用率

内存使用率可以使用container_memory_working_set_bytes来进行查看

  1. sum (container_memory_working_set_bytes{id!="/",pod_name=~"^$Deployment$Statefulset$Daemonset.*$",kubernetes_io_hostname=~"^$Node$"}) by (pod_name,kubernetes_io_hostname)

k8s的官方grafana

  • 3131 Kubernetes All Nodes


知识链接

  1. 通过Prometheus查询计算Kubernetes集群中的容器CPU、内存使用率等指标