监控目标

前言

prometheus-operator 将监控目标（即 prometheus.yaml 中的 job）抽象为 CRD，kube-prometheus-stack 安装包默认包含了 K8s 集群核心组件、节点资源的监控：

$ kubectl -n monitoring get servicemonitor
prometheus-kube-prometheus-alertmanager    18m
prometheus-kube-prometheus-node-exporter   18m
prometheus-kube-prometheus-operator        18m
prometheus-kube-prometheus-prometheus      18m
...

可以通过新建、更新、删除这些 CRD，实现对监控目标的 CRUD 操作。😊
对监控目标的 CURD 操作示例：

$ kubectl -n monitoring get servicemonitor prometheus-kube-prometheus-node-exporter -oyaml > ./sm-node-exporter.yaml
# 删除不需要的字段后的内容如下：
$ cat sm-node-exporter.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prometheus
    meta.helm.sh/release-namespace: monitoring
  labels:
    app: kube-prometheus-stack-node-exporter
    release: prometheus  ## required, prometheus-operator会依据这个发现&自动加载配置
  name: prometheus-kube-prometheus-node-exporter
  namespace: monitoring
spec:
  endpoints:
  - port: metrics
  jobLabel: jobLabel
  selector:  ## 基于 k8d service 的服务发现
    matchLabels:
      app: prometheus-node-exporter
      release: prometheus
## (1) 修改已有监控目标：
$ vim sm-node-exporter.yaml  # 编辑内容
$ kubectl -n monitoring apply -f sm-node-exporter.yaml # 使生效
## (2) 新增监控目标：
$ touch sm-libvirt-exporter.yaml  # 新增servicemonitor配置文件
$ vim sm-libvirt-exporter.yaml
$ kubectl -n monitoring apply -f sm-node-exporter.yaml # 使生效
## (3) 删除监控目标：
$ kubectl -n monitoring delete servicemonitor prometheus-kube-prometheus-node-exporter

下文内容即是对三种 CRD 以及静态监控配置的详细介绍。

servicemonitor

probe

podmonitor

静态监控目标

有时在测试阶段业务应用还未完善、或某些特殊的 exporter，没有 service，不适用于 servicemonitor，需要配置静态监控目标：
(1) 新建 secret 添加 additional scrape configs:

$ cat prometheus-additional.yaml
- job_name: blackbox
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - https://111.111.111.239:6443
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance
    - target_label: __address__
      replacement: blackbox-exporter-prometheus-blackbox-exporter.monitoring.svc.cluster.local:9115
# 新建 secret:
kubectl -n monitoring create secret generic additional-configs \
    --from-file=prometheus-additional.yaml
# 更新 secret:
kubectl -n monitoring create secret generic additional-configs \
    --from-file=prometheus-additional.yaml --dry-run=client -o yaml | kubectl apply -f -

(2) 在 values.yaml 中开启 additionalScrapeConfigsSecret：

## If additional scrape configurations are already deployed in a single secret file you can use this section.
## Expected values are the secret name and key
prometheus:
  ...
  prometheusSpec:
    ...
    additionalScrapeConfigsSecret:
      enabled: true
      name: additional-configs
      key: prometheus-additional.yaml

修改完 values.yaml 后，执行 helm upgrade 更新使生效。