说明

instance

  1. - 用Prometheus术语来说，可以抓取的端点称为实例 instance

job

  1. - 具有相同目的的实例的集合（例如，出于可伸缩性或可靠性而复制的过程）称为job
  3. ## 
  5. 举例
  6.   - job_name: 'pushgateway'
  7.     honor_timestamps: true
  8.     scrape_interval: 15s
  9.     scrape_timeout: 10s
  10.     metrics_path: /metrics
  11.     scheme: http
  12.     static_configs:
  13.     - targets:
  14.       - 172.20.70.205:9091
  15.       - 172.20.70.205:9092
  16.       - 172.20.70.215:9091

自动生成的标签和时间序列

  1. 当Prometheus抓取目标时，它会自动在抓取的时间序列上附加一些标签，以识别被抓取的目标：
  3. - job：目标所属的已配置作业名称。
  4. - instance：<host>:<port>抓取的目标网址的一部分。
  5. - up{job="<job-name>", instance="<instance-id>"}：1实例是否正常（即可达）或0刮取失败。
  7. - - - 设置告警查看采集失败的实例 `up==0`
  9. - scrape_duration_seconds{job="<job-name>", instance="<instance-id>"}：刮擦的耗时
  11. -

举例

  1. scrape_duration_seconds{instance="172.20.70.205", job="blackbox-ssh"} 0.001817932
  2. scrape_duration_seconds{instance="172.20.70.205:3000", job="single-targets"} 0.005416658
  3. scrape_duration_seconds{instance="172.20.70.205:9091", job="pushgateway"} 0.002726714
  4. scrape_duration_seconds{instance="172.20.70.205:9092", job="pushgateway"} 0.000506256
  5. scrape_duration_seconds{instance="172.20.70.205:9100", job="single-targets"} 0.012790691
  6. scrape_duration_seconds{instance="172.20.70.205:9104", job="single-targets"} 0.021421043
  7. scrape_duration_seconds{instance="172.20.70.205:9115", job="blackbox-http-targets"} 0.00427973
  1. 用途：统计job中采集比较耗时的instance ,
  4. - 为什么慢
  5.   - 网络质量
  6.   - metrics数据量太大
  7.   - prometheus采集端有瓶颈了，需要扩容
  8. - 上次采集最慢的五个 job+instance topk(5,scrape_duration_seconds)
  9. - 采集时间超过3秒的 scrape_duration_seconds > 3
  13. - scrape_samples_post_metric_relabeling{job="<job-name>", instance="<instance-id>"}：relabel之后剩余的重新标记后剩余的样本数
  14.   - 何为样本：简单理解就是 标签组唯一 
  15. - scrape_samples_scraped{job="<job-name>", instance="<instance-id>"}：目标暴露的样本数

举例 topk(5,scrape_samples_scraped)

  1. scrape_samples_scraped{instance="172.20.70.205:9256", job="single-targets"} 1691
  2. scrape_samples_scraped{instance="172.20.70.215:9256", job="single-targets"} 1010
  3. scrape_samples_scraped{instance="172.20.70.205:9104", job="single-targets"} 816
  4. scrape_samples_scraped{instance="172.20.70.215:9100", job="single-targets"} 500
  5. scrape_samples_scraped{instance="172.20.70.205:9100", job="single-targets"} 500

  • 用途： 统计样本数量按 job+instance分类

    按job排序 topk(5,sum(scrape_samples_scraped) by (job))

  1. {job="single-targets"} 4957
  2. {job="redis_exporter_targets"} 299
  3. {job="pushgateway"} 102
  4. {job="blackbox-http-targets"} 72
  5. {job="blackbox-ssh"} 6
  1. - scrape_series_added{job="<job-name>", instance="<instance-id>"}：此抓取中新系列的大概数量。v2.10的新功能
  2.   - 用途 统计新增的metrics，可以用来查看写峰
  3.   - 大部分情况应该都是旧的metrics append写入
  5. #

prometheus特殊tag说明

  1. - __address__ 采集endpoint的地址
  2. - __name__   metrics 的名称
  3. - instance   endpoint最后的tag
  4. - job         任务
  5. - __metrics_path__  采集的http path 如 /metrics  /cadvisor/metrics