基础设置

  1. 配置hosts
  3. 172.16.58.78   prome-master01
  4. 172.16.58.79   prome-node01
  7. master上生成ssh key 并拷贝到node上
  9. ssh-keygen
  10. ssh-copy-id prome_node_01
  12. # 测试ssh联通
  13. ssh prome_node_01

master安装ansible

  1. yum install -y ansible
  3. # 关闭hostcheck 
  4. vim /etc/ansible/ansible.cfg
  6. ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking＝no 
  11. playbook执行时需要设置机器文件 
  12. cat <<EOF > /opt/tgzs/host_file
  13. prome-master01
  14. prome-node01
  15. EOF
  18. 设置syslog 和logrotate服务
  19. ansible-playbook -i host_file init_syslog_logrotate.yaml
  22. 编写ansible 发布服务脚本
  24. ansible-playbook -i host_file  service_deploy.yaml  -e "tgz=node_exporter-1.1.2.linux-amd64.tar.gz" -e "app=node_exporter"
  27. 检查node_exporter服务状态
  28. ansible -i host_file all -m shell -a " ps -ef |grep node_exporter|grep -v grep "

init_syslog_logrotate.yaml
service_deploy.yaml

浏览器访问

  1. IP:9100/metrics
  2. IP:9100/metrics

本机访问数据

  1. [root@prome_master_01 tgzs]# curl  -s  localhost:9100/metrics |grep node_  |head -20
  2. # HELP node_arp_entries ARP entries by device
  3. # TYPE node_arp_entries gauge
  4. node_arp_entries{device="eth0"} 3
  5. # HELP node_boot_time_seconds Node boot time, in unixtime.
  6. # TYPE node_boot_time_seconds gauge
  7. node_boot_time_seconds 1.616987084e+09
  8. # HELP node_context_switches_total Total number of context switches.
  9. # TYPE node_context_switches_total counter
  10. node_context_switches_total 2.105979e+06
  11. # HELP node_cooling_device_cur_state Current throttle state of the cooling device
  12. # TYPE node_cooling_device_cur_state gauge
  13. node_cooling_device_cur_state{name="0",type="Processor"} 0
  14. node_cooling_device_cur_state{name="1",type="Processor"} 0
  15. node_cooling_device_cur_state{name="2",type="Processor"} 0
  16. node_cooling_device_cur_state{name="3",type="Processor"} 0
  17. # HELP node_cooling_device_max_state Maximum throttle state of the cooling device
  18. # TYPE node_cooling_device_max_state gauge
  19. node_cooling_device_max_state{name="0",type="Processor"} 0
  20. node_cooling_device_max_state{name="1",type="Processor"} 0
  21. node_cooling_device_max_state{name="2",type="Processor"} 0

默认开启的采集项目介绍

image.png

关闭默认开启采集项

  1. --no-collector.<name> flag
  3. # 未开启前
  4. [root@prome_master_01 node_exporter]# curl  -s  localhost:9100/metrics |grep node_cpu
  5. # HELP node_cpu_guest_seconds_total Seconds the CPUs spent in guests (VMs) for each mode.
  6. # TYPE node_cpu_guest_seconds_total counter
  7. node_cpu_guest_seconds_total{cpu="0",mode="nice"} 0
  8. node_cpu_guest_seconds_total{cpu="0",mode="user"} 0
  9. node_cpu_guest_seconds_total{cpu="1",mode="nice"} 0
  10. node_cpu_guest_seconds_total{cpu="1",mode="user"} 0
  11. node_cpu_guest_seconds_total{cpu="2",mode="nice"} 0
  12. node_cpu_guest_seconds_total{cpu="2",mode="user"} 0
  13. node_cpu_guest_seconds_total{cpu="3",mode="nice"} 0
  14. node_cpu_guest_seconds_total{cpu="3",mode="user"} 0
  15. # HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
  16. # TYPE node_cpu_seconds_total counter
  17. node_cpu_seconds_total{cpu="0",mode="idle"} 17691.27
  18. node_cpu_seconds_total{cpu="0",mode="iowait"} 8.9
  19. node_cpu_seconds_total{cpu="0",mode="irq"} 0
  20. node_cpu_seconds_total{cpu="0",mode="nice"} 0.32
  21. node_cpu_seconds_total{cpu="0",mode="softirq"} 0.28
  22. node_cpu_seconds_total{cpu="0",mode="steal"} 2.7
  24. # 关闭cpu采集
  25.  ./node_exporter --no-collector.cpu
  26. curl  -s  localhost:9100/metrics |grep node_cpu

关闭默认采集器项而开机器某些采集

  1.  --collector.disable-defaults --collector.<name> .
  3. # 只开启mem采集
  4.  ./node_exporter --collector.disable-defaults --collector.meminfo
  6. # 只开启mem 和cpu 采集
  7. ./node_exporter --collector.disable-defaults --collector.meminfo --collector.cpu

默认关闭的关闭原因

  • 太重
  • 太慢
  • 太多的开销

image.png

禁用golang sdk 指标

  • 使用 --web.disable-exporter-metrics
  • promhttp_ 代表访问/metrics 的http情况
    1. [root@prome_master_01 tgzs]# curl  -s  localhost:9100/metrics |grep promhttp_
    2. # HELP promhttp_metric_handler_errors_total Total number of internal errors encountered by the promhttp metric handler.
    3. # TYPE promhttp_metric_handler_errors_total counter
    4. promhttp_metric_handler_errors_total{cause="encoding"} 0
    5. promhttp_metric_handler_errors_total{cause="gathering"} 0
    6. # HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
    7. # TYPE promhttp_metric_handler_requests_in_flight gauge
    8. promhttp_metric_handler_requests_in_flight 1
    9. # HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
    10. # TYPE promhttp_metric_handler_requests_total counter
    11. promhttp_metric_handler_requests_total{code="200"} 8
    12. promhttp_metric_handler_requests_total{code="500"} 0
    13. promhttp_metric_handler_requests_total{code="503"} 0
    go_代表goruntime信息等 ```bash

    HELP go_goroutines Number of goroutines that currently exist.

    TYPE go_goroutines gauge

    go_goroutines 7

    HELP go_info Information about the Go environment.

    TYPE go_info gauge

    go_info{version=”go1.15.8”} 1

    HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.

    TYPE go_memstats_alloc_bytes gauge

    go_memstats_alloc_bytes 2.781752e+06
  3. - process 代表进程信息等
  4. ```bash
  5. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  6. # TYPE process_cpu_seconds_total counter
  7. process_cpu_seconds_total 0.54
  8. # HELP process_max_fds Maximum number of open file descriptors.
  9. # TYPE process_max_fds gauge
  10. process_max_fds 1024
  11. # HELP process_open_fds Number of open file descriptors.
  12. # TYPE process_open_fds gauge
  13. process_open_fds 9
  14. # HELP process_resident_memory_bytes Resident memory size in bytes.
  15. # TYPE process_resident_memory_bytes gauge
  16. process_resident_memory_bytes 1.5720448e+07

节点上自打点数据上报

  • —collector.textfile.directory=”” 配置本地采集目录
  • 在采集目录里创建.prom文件 ```bash

    创建目录

    mkdir ./text_file_dir

    准备 prom文件

    cat < ./text_file_dir/test.prom

    HELP nyy_test_metric just test

    TYPE nyy_test_metric gauge

    nyy_test_metric{method=”post”,code=”200”} 1027 EOF

启动服务

./node_exporter —collector.textfile.directory=./text_file_dir

curl查看数据

[root@prome_master_01 tgzs]# curl -s localhost:9100/metrics |grep nyy

HELP nyy_test_metric just test

TYPE nyy_test_metric gauge

nyy_test_metric{code=”200”,method=”post”} 1027

  1. http传入参数，按采集器过滤指标
  3. 原理：通过http请求参数过滤采集器
  4. ```go
  5. func (h *handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  6.     filters := r.URL.Query()["collect[]"]
  7.     level.Debug(h.logger).Log("msg", "collect query:", "filters", filters)
  9.     if len(filters) == 0 {
  10.         // No filters, use the prepared unfiltered handler.
  11.         h.unfilteredHandler.ServeHTTP(w, r)
  12.         return
  13.     }
  14.     // To serve filtered metrics, we create a filtering handler on the fly.
  15.     filteredHandler, err := h.innerHandler(filters...)
  16.     if err != nil {
  17.         level.Warn(h.logger).Log("msg", "Couldn't create filtered metrics handler:", "err", err)
  18.         w.WriteHeader(http.StatusBadRequest)
  19.         w.Write([]byte(fmt.Sprintf("Couldn't create filtered metrics handler: %s", err)))
  20.         return
  21.     }
  22.     filteredHandler.ServeHTTP(w, r)
  23. }

http访问

  1. # 只看cpu采集器的指标
  2. http://IP:9100/metrics?collect[]=cpu
  5. # 只看cpu和mem采集器的指标
  6. http://IP:9100/metrics?collect[]=cpu&collect[]=meminfo

prometheus配置

  1.   params:
  2.     collect[]:
  3.       - cpu
  4.       - meminfo
  5. 两种导入模式
  7. - url导入
  8. - json文件导入

两种导入模式
- url导入
- json文件导入
https://grafana.com/grafana/dashboards/8919

配置数据源

image.png

image.png

验证

image.png
image.png

手动安装node exporter

上传软件包到/server下

  1. tar xf node_exporter-1.1.2.linux-amd64.tar.gz -C /opt
  2. cd /opt/node_exporter-1.1.2.linux-amd64/
  3. mv node_exporter /usr/local/bin/
  5.  cat /etc/systemd/system/node_exporter.service
  6. [Unit]
  7. Description=Node Exporter
  8. After=network.target
  10. [Service]
  11. User=root
  12. Group=root
  13. Type=simple
  14. ExecStart=/usr/local/bin/node_exporter
  15. [Install]
  16. WantedBy=multi-user.target
  18.  1016  systemctl daemon-reload
  19.  1017  systemctl start node_exporter
  20.  1018  systemctl status node_exporter.service
  21.  1019  systemctl enable node_exporter
  22.  1020  curl localhost:9100/metrics

普罗米修斯端配置

  1. [root@Server-d0449e6d-d612-49ba-80cc-070b579955d6 prometheus-2.25.2.linux-amd64]# cat prometheus.yml
  2.   - job_name: node_exporter
  3.     honor_timestamps: true
  4.     scrape_interval: 15s
  5.     scrape_timeout: 10s 
  6.     metrics_path: /metrics 
  7.     scheme: http   采集方式默认http 
  8.     static_configs:
  9.     - targets:   采集两台node
  10.       - 192.168.0.107:7100
  11.       - 192.168.0.56:9100

image.png

targets解说

  1. - job 分组情况
  2. - endpoint 实例地址
  3. - state 采集是否成功
  4. - label 标签组
  5. - Last Scrape 上次采集到现在的间隔时间
  6. - Scrape Duration 上次采集耗时
  7. - Error 采集错误