安装Prometheus

首先从官方Repo下载最新版本。然后按照下列步骤安装Prometheus：

  1. useradd --no-create-home --shell /bin/false prometheus
  2. mkdir /etc/prometheus
  3. mkdir /var/lib/prometheus
  4. chown prometheus:prometheus /etc/prometheus
  5. chown prometheus:prometheus /var/lib/prometheus
  6. tar zxvf prometheus-2.20.1.linux-amd64.tar.gz
  7. mv prometheus-2.20.1.linux-amd64/prometheus /usr/bin/  # 当然使用cp复制也行，目录也可以变
  8. mv prometheus-2.20.1.linux-amd64/promtool /usr/bin/
  9. chown prometheus:prometheus /usr/bin/prometheus
  10. chown prometheus:prometheus /usr/bin/promtool
  11. cp -r prometheus-2.20.1.linux-amd64/consoles/ /etc/prometheus/
  12. cp -r prometheus-2.20.1.linux-amd64/console_libraries/ /etc/prometheus/
  13. chown -R prometheus:prometheus /etc/prometheus/consoles
  14. chown -R prometheus:prometheus /etc/prometheus/console_libraries
  15. cp prometheus-2.20.1.linux-amd64/prometheus.yml /etc/prometheus/
  16. chown prometheus:prometheus /etc/prometheus/prometheus.yml
  17. vi /etc/prometheus/prometheus.yml  # 编辑配置文件

由于使用Prometheus的目的是通过SNMP Exporter监控网络设备，因此这里配置主要针对snmp_exporter、blackbox_exporter。

  1. # my global config
  2. global:
  3.   scrape_interval:     10s # Set the scrape interval to every 15 seconds. Default is 
  5. rule_files:
  6.   # - "first_rules.yml"
  7.   # - "second_rules.yml"
  9. # A scrape configuration containing exactly one endpoint to scrape:
  10. # Here it's Prometheus itself.
  11. scrape_configs:
  12.   # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  13.   - job_name: 'prometheus'
  14.     # metrics_path defaults to '/metrics'
  15.     scrape_interval: 5s
  16.     static_configs:
  17.     - targets: ['localhost:9090']
  18.   - job_name: 'node_exporter_centos'
  19.     scrape_interval: 5s
  20.     static_configs:
  21.       - targets: ['10.X.X.X:9100']  # 填写node_exporter的主机IP
  22.   - job_name: 'blackbox'
  23.     metrics_path: /probe
  24.     params:
  25.       module: [icmp]
  26.     static_configs:
  27.       - targets: ['10.X.X.X', '10.X.X.X']  # 填写ICMP监控目的IP
  28.     relabel_configs:
  29.       - source_labels: [__address__]
  30.         target_label: __param_target
  31.       - source_labels: [__param_target]
  32.         target_label: instance
  33.       - target_label: __address__
  34.         replacement: 10.X.X.X:9115  # The blackbox exporter's real hostname:port.
  35.   - job_name: 'snmp'
  36.     scrape_interval: 30s
  37.     static_configs:
  38.       - targets: ['10.X.X.X', '10.X.X.X']
  39.     metrics_path: /snmp
  40.     params:
  41.       module: [h3c_switch]
  42.     relabel_configs:
  43.       - source_labels: [__address__]
  44.         target_label: __param_target
  45.       - source_labels: [__param_target]
  46.         target_label: instance
  47.       - target_label: __address__
  48.         replacement: 10.X.X.X:9116  # The SNMP exporter's real hostname:port.

使用vim /etc/systemd/system/prometheus.service编辑系统服务：

  1. [Unit]
  2. Description=Prometheus
  3. Wants=network-online.target
  4. After=network-online.target
  6. [Service]
  7. User=prometheus
  8. Group=prometheus
  9. Type=simple
  10. ExecStart=/usr/bin/prometheus \
  11. --config.file /etc/prometheus/prometheus.yml \
  12. --storage.tsdb.path /var/lib/prometheus/ \
  13. --storage.tsdb.retention.time=2y \
  14. --web.console.templates=/etc/prometheus/consoles \
  15. --web.console.libraries=/etc/prometheus/console_libraries
  16. ExecReload=/bin/kill -HUP $MAINPID
  17. Restart=on-failure
  18. [Install]
  19. WantedBy=multi-user.target

编辑保存后，加载服务：

  1. systemctl daemon-reload
  2. systemctl start prometheus
  3. systemctl status prometheus
  4. systemctl enable prometheus
  5. firewall-cmd --zone=public --add-port=9090/tcp --permanent --set-description="prometheus"
  6. firewall-cmd --reload

至此Prometheus安装完成。可以通过http://10.X.X.X:9090/targets查看每个Exporter的取值情况。

安装Node Exporter

直接贴步骤，依然可以从官方下载。

  1. tar zxvf node_exporter-1.1.2.linux-amd64.tar.gz
  2. mv node_exporter-1.1.2.linux-amd64/node_exporter /usr/local/bin/
  3. useradd -rs /bin/false nodeusr
  4. chown nodeusr:nodeusr /usr/local/bin/node_exporter

使用vi /etc/systemd/system/node_exporter.service编辑系统服务：

  1. [Unit]
  2. Description=Node Exporter
  3. After=network.target
  5. [Service]
  6. User=nodeusr
  7. Group=nodeusr
  8. Type=simple
  9. ExecStart=/usr/local/bin/node_exporter
  10. Restart=on-failure
  11. [Install]
  12. WantedBy=multi-user.target

编辑保存后，加载服务：

  1. systemctl daemon-reload
  2. systemctl start node_exporter
  3. systemctl status node_exporter
  4. systemctl enable node_exporter
  5. firewall-cmd --zone=public --add-port=9100/tcp --permanent --set-description="node_exporter"
  6. firewall-cmd --reload

至此Node Exporter安装完成。可以在Prometheus的Target里查看状态。因为前面在安装Prometheus的时候已经配置了node的job，因此这里略过配置步骤。

安装Blackbox Exporter

直接贴步骤，依然可以从官方下载。

  1. tar zxvf blackbox_exporter-0.18.0.linux-amd64.tar.gz
  2. mv blackbox_exporter-0.18.0.linux-amd64/blackbox_exporter /usr/local/bin/
  3. mkdir -p /etc/blackbox_exporter/
  4. mv blackbox_exporter-0.18.0.linux-amd64/blackbox.yml /etc/blackbox_exporter/
  5. useradd -rs /bin/false blackboxusr
  6. chown blackboxusr:blackboxusr /usr/local/bin/blackbox_exporter

使用vi /etc/blackbox_exporter/blackbox.yml编辑Blackbox Exporter的配置：

  1. modules:
  2.   icmp:
  3.     prober: icmp
  4.     timeout: 1s
  5.     icmp:
  6.       preferred_ip_protocol: "ip4"
  7.       # ip_protocol_fallback: true
  8.       source_ip_address: 10.4.92.6
  9.       # dont_fragment: false
  10.       # payload_size: 0

编辑保存后，使用chown -R blackboxusr:blackboxusr /etc/blackbox_exporter/修改文件及文件夹归属。
使用vi /etc/systemd/system/blackbox_exporter.service编辑系统服务：

  1. [Unit]
  2. Description=Blackbox Exporter
  3. After=network.target
  5. [Service]
  6. User=blackboxusr
  7. Group=blackboxusr
  8. Type=simple
  9. ExecStart=/usr/local/bin/blackbox_exporter \
  10. --config.file=/etc/blackbox_exporter/blackbox.yml
  11. Restart=on-failure
  12. [Install]
  13. WantedBy=multi-user.target

编辑保存后，加载服务：

  1. systemctl daemon-reload
  2. systemctl start blackbox_exporter
  3. systemctl status blackbox_exporter
  4. systemctl enable blackbox_exporter
  5. firewall-cmd --zone=public --add-port=9115/tcp --permanent --set-description="blackbox_exporter"
  6. firewall-cmd --reload

至此Blackbox Exporter安装完成。可以在Prometheus的Target里查看状态。因为前面在安装Prometheus的时候已经配置了blackbox的job，因此这里略过配置步骤。

安装SNMP Exporter

直接贴步骤，依然可以从官方下载。

  1. tar zxvf snmp_exporter-0.20.0.linux-amd64.tar.gz
  2. mv snmp_exporter-0.20.0.linux-amd64/snmp_exporter /usr/local/bin/
  3. mkdir -p /etc/snmp_exporter/
  4. mv snmp_exporter-0.20.0.linux-amd64/snmp.yml /etc/snmp_exporter/
  5. useradd -rs /bin/false snmpusr
  6. chown snmpusr:snmpusr /usr/local/bin/snmp_exporter

使用vi /etc/snmp_exporter/snmp.yml编辑SNMP Exporter的配置。默认的snmp.yml涉及的MIB节点过多，官方文档也有自己编译的方式，这里直接复制：

  1. h3c_switch:
  2.   walk:
  3.   - 1.3.6.1.2.1.2.2.1.2
  4.   - 1.3.6.1.2.1.2.2.1.3
  5.   - 1.3.6.1.2.1.31.1.1.1.1
  6.   - 1.3.6.1.2.1.31.1.1.1.10
  7.   - 1.3.6.1.2.1.31.1.1.1.18
  8.   - 1.3.6.1.2.1.31.1.1.1.6
  9.   get:
  10.   - 1.3.6.1.2.1.1.3.0
  11.   metrics:
  12.   - name: sysUpTime
  13.     oid: 1.3.6.1.2.1.1.3
  14.     type: gauge
  15.     help: The time (in hundredths of a second) since the network management portion
  16.       of the system was last re-initialized. - 1.3.6.1.2.1.1.3
  17.   - name: ifType
  18.     oid: 1.3.6.1.2.1.2.2.1.3
  19.     type: EnumAsInfo
  20.     help: The type of interface - 1.3.6.1.2.1.2.2.1.3
  21.     indexes:
  22.     - labelname: ifIndex
  23.       type: gauge
  24.     lookups:
  25.     - labels:
  26.       - ifIndex
  27.       labelname: ifAlias
  28.       oid: 1.3.6.1.2.1.31.1.1.1.18
  29.       type: DisplayString
  30.     - labels:
  31.       - ifIndex
  32.       labelname: ifDescr
  33.       oid: 1.3.6.1.2.1.2.2.1.2
  34.       type: DisplayString
  35.     - labels:
  36.       - ifIndex
  37.       labelname: ifName
  38.       oid: 1.3.6.1.2.1.31.1.1.1.1
  39.       type: DisplayString
  40.     enum_values:
  41.       1: other
  42.       2: regular1822
  43.       3: hdh1822
  44.       4: ddnX25
  45.       5: rfc877x25
  46.       6: ethernetCsmacd
  47.       7: iso88023Csmacd
  48.       8: iso88024TokenBus
  49.       9: iso88025TokenRing
  50.       10: iso88026Man
  51.       11: starLan
  52.       12: proteon10Mbit
  53.       13: proteon80Mbit
  54.       14: hyperchannel
  55.       15: fddi
  56.       16: lapb
  57.       17: sdlc
  58.       18: ds1
  59.       19: e1
  60.       20: basicISDN
  61.       21: primaryISDN
  62.       22: propPointToPointSerial
  63.       23: ppp
  64.       24: softwareLoopback
  65.       25: eon
  66.       26: ethernet3Mbit
  67.       27: nsip
  68.       28: slip
  69.       29: ultra
  70.       30: ds3
  71.       31: sip
  72.       32: frameRelay
  73.       33: rs232
  74.       34: para
  75.       35: arcnet
  76.       36: arcnetPlus
  77.       37: atm
  78.       38: miox25
  79.       39: sonet
  80.       40: x25ple
  81.       41: iso88022llc
  82.       42: localTalk
  83.       43: smdsDxi
  84.       44: frameRelayService
  85.       45: v35
  86.       46: hssi
  87.       47: hippi
  88.       48: modem
  89.       49: aal5
  90.       50: sonetPath
  91.       51: sonetVT
  92.       52: smdsIcip
  93.       53: propVirtual
  94.       54: propMultiplexor
  95.       55: ieee80212
  96.       56: fibreChannel
  97.       57: hippiInterface
  98.       58: frameRelayInterconnect
  99.       59: aflane8023
  100.       60: aflane8025
  101.       61: cctEmul
  102.       62: fastEther
  103.       63: isdn
  104.       64: v11
  105.       65: v36
  106.       66: g703at64k
  107.       67: g703at2mb
  108.       68: qllc
  109.       69: fastEtherFX
  110.       70: channel
  111.       71: ieee80211
  112.       72: ibm370parChan
  113.       73: escon
  114.       74: dlsw
  115.       75: isdns
  116.       76: isdnu
  117.       77: lapd
  118.       78: ipSwitch
  119.       79: rsrb
  120.       80: atmLogical
  121.       81: ds0
  122.       82: ds0Bundle
  123.       83: bsc
  124.       84: async
  125.       85: cnr
  126.       86: iso88025Dtr
  127.       87: eplrs
  128.       88: arap
  129.       89: propCnls
  130.       90: hostPad
  131.       91: termPad
  132.       92: frameRelayMPI
  133.       93: x213
  134.       94: adsl
  135.       95: radsl
  136.       96: sdsl
  137.       97: vdsl
  138.       98: iso88025CRFPInt
  139.       99: myrinet
  140.       100: voiceEM
  141.       101: voiceFXO
  142.       102: voiceFXS
  143.       103: voiceEncap
  144.       104: voiceOverIp
  145.       105: atmDxi
  146.       106: atmFuni
  147.       107: atmIma
  148.       108: pppMultilinkBundle
  149.       109: ipOverCdlc
  150.       110: ipOverClaw
  151.       111: stackToStack
  152.       112: virtualIpAddress
  153.       113: mpc
  154.       114: ipOverAtm
  155.       115: iso88025Fiber
  156.       116: tdlc
  157.       117: gigabitEthernet
  158.       118: hdlc
  159.       119: lapf
  160.       120: v37
  161.       121: x25mlp
  162.       122: x25huntGroup
  163.       123: transpHdlc
  164.       124: interleave
  165.       125: fast
  166.       126: ip
  167.       127: docsCableMaclayer
  168.       128: docsCableDownstream
  169.       129: docsCableUpstream
  170.       130: a12MppSwitch
  171.       131: tunnel
  172.       132: coffee
  173.       133: ces
  174.       134: atmSubInterface
  175.       135: l2vlan
  176.       136: l3ipvlan
  177.       137: l3ipxvlan
  178.       138: digitalPowerline
  179.       139: mediaMailOverIp
  180.       140: dtm
  181.       141: dcn
  182.       142: ipForward
  183.       143: msdsl
  184.       144: ieee1394
  185.       145: if-gsn
  186.       146: dvbRccMacLayer
  187.       147: dvbRccDownstream
  188.       148: dvbRccUpstream
  189.       149: atmVirtual
  190.       150: mplsTunnel
  191.       151: srp
  192.       152: voiceOverAtm
  193.       153: voiceOverFrameRelay
  194.       154: idsl
  195.       155: compositeLink
  196.       156: ss7SigLink
  197.       157: propWirelessP2P
  198.       158: frForward
  199.       159: rfc1483
  200.       160: usb
  201.       161: ieee8023adLag
  202.       162: bgppolicyaccounting
  203.       163: frf16MfrBundle
  204.       164: h323Gatekeeper
  205.       165: h323Proxy
  206.       166: mpls
  207.       167: mfSigLink
  208.       168: hdsl2
  209.       169: shdsl
  210.       170: ds1FDL
  211.       171: pos
  212.       172: dvbAsiIn
  213.       173: dvbAsiOut
  214.       174: plc
  215.       175: nfas
  216.       176: tr008
  217.       177: gr303RDT
  218.       178: gr303IDT
  219.       179: isup
  220.       180: propDocsWirelessMaclayer
  221.       181: propDocsWirelessDownstream
  222.       182: propDocsWirelessUpstream
  223.       183: hiperlan2
  224.       184: propBWAp2Mp
  225.       185: sonetOverheadChannel
  226.       186: digitalWrapperOverheadChannel
  227.       187: aal2
  228.       188: radioMAC
  229.       189: atmRadio
  230.       190: imt
  231.       191: mvl
  232.       192: reachDSL
  233.       193: frDlciEndPt
  234.       194: atmVciEndPt
  235.       195: opticalChannel
  236.       196: opticalTransport
  237.       197: propAtm
  238.       198: voiceOverCable
  239.       199: infiniband
  240.       200: teLink
  241.       201: q2931
  242.       202: virtualTg
  243.       203: sipTg
  244.       204: sipSig
  245.       205: docsCableUpstreamChannel
  246.       206: econet
  247.       207: pon155
  248.       208: pon622
  249.       209: bridge
  250.       210: linegroup
  251.       211: voiceEMFGD
  252.       212: voiceFGDEANA
  253.       213: voiceDID
  254.       214: mpegTransport
  255.       215: sixToFour
  256.       216: gtp
  257.       217: pdnEtherLoop1
  258.       218: pdnEtherLoop2
  259.       219: opticalChannelGroup
  260.       220: homepna
  261.       221: gfp
  262.       222: ciscoISLvlan
  263.       223: actelisMetaLOOP
  264.       224: fcipLink
  265.       225: rpr
  266.       226: qam
  267.       227: lmp
  268.       228: cblVectaStar
  269.       229: docsCableMCmtsDownstream
  270.       230: adsl2
  271.       231: macSecControlledIF
  272.       232: macSecUncontrolledIF
  273.       233: aviciOpticalEther
  274.       234: atmbond
  275.       235: voiceFGDOS
  276.       236: mocaVersion1
  277.       237: ieee80216WMAN
  278.       238: adsl2plus
  279.       239: dvbRcsMacLayer
  280.       240: dvbTdm
  281.       241: dvbRcsTdma
  282.       242: x86Laps
  283.       243: wwanPP
  284.       244: wwanPP2
  285.       245: voiceEBS
  286.       246: ifPwType
  287.       247: ilan
  288.       248: pip
  289.       249: aluELP
  290.       250: gpon
  291.       251: vdsl2
  292.       252: capwapDot11Profile
  293.       253: capwapDot11Bss
  294.       254: capwapWtpVirtualRadio
  295.       255: bits
  296.       256: docsCableUpstreamRfPort
  297.       257: cableDownstreamRfPort
  298.       258: vmwareVirtualNic
  299.       259: ieee802154
  300.       260: otnOdu
  301.       261: otnOtu
  302.       262: ifVfiType
  303.       263: g9981
  304.       264: g9982
  305.       265: g9983
  306.       266: aluEpon
  307.       267: aluEponOnu
  308.       268: aluEponPhysicalUni
  309.       269: aluEponLogicalLink
  310.       270: aluGponOnu
  311.       271: aluGponPhysicalUni
  312.       272: vmwareNicTeam
  313.   - name: ifHCOutOctets
  314.     oid: 1.3.6.1.2.1.31.1.1.1.10
  315.     type: counter
  316.     help: The total number of octets transmitted out of the interface, including framing
  317.       characters - 1.3.6.1.2.1.31.1.1.1.10
  318.     indexes:
  319.     - labelname: ifIndex
  320.       type: gauge
  321.     lookups:
  322.     - labels:
  323.       - ifIndex
  324.       labelname: ifAlias
  325.       oid: 1.3.6.1.2.1.31.1.1.1.18
  326.       type: DisplayString
  327.     - labels:
  328.       - ifIndex
  329.       labelname: ifDescr
  330.       oid: 1.3.6.1.2.1.2.2.1.2
  331.       type: DisplayString
  332.     - labels:
  333.       - ifIndex
  334.       labelname: ifName
  335.       oid: 1.3.6.1.2.1.31.1.1.1.1
  336.       type: DisplayString
  337.   - name: ifHCInOctets
  338.     oid: 1.3.6.1.2.1.31.1.1.1.6
  339.     type: counter
  340.     help: The total number of octets received on the interface, including framing
  341.       characters - 1.3.6.1.2.1.31.1.1.1.6
  342.     indexes:
  343.     - labelname: ifIndex
  344.       type: gauge
  345.     lookups:
  346.     - labels:
  347.       - ifIndex
  348.       labelname: ifAlias
  349.       oid: 1.3.6.1.2.1.31.1.1.1.18
  350.       type: DisplayString
  351.     - labels:
  352.       - ifIndex
  353.       labelname: ifDescr
  354.       oid: 1.3.6.1.2.1.2.2.1.2
  355.       type: DisplayString
  356.     - labels:
  357.       - ifIndex
  358.       labelname: ifName
  359.       oid: 1.3.6.1.2.1.31.1.1.1.1
  360.       type: DisplayString
  361.   version: 2
  362.   auth:
  363.     community: xxxxxx

编辑保存后，使用chown -R snmpusr:snmpusr /etc/snmp_exporter/修改文件及文件夹归属。
使用vi /etc/systemd/system/snmp_exporter.service编辑系统服务：

  1. [Unit]
  2. Description=Blackbox Exporter
  3. After=network.target
  5. [Service]
  6. User=blackboxusr
  7. Group=blackboxusr
  8. Type=simple
  9. ExecStart=/usr/local/bin/blackbox_exporter \
  10. --config.file=/etc/blackbox_exporter/blackbox.yml
  11. Restart=on-failure
  12. [Install]
  13. WantedBy=multi-user.target

编辑保存后，加载服务：

  1. systemctl daemon-reload
  2. systemctl start snmp_exporter
  3. systemctl status snmp_exporter
  4. systemctl enable snmp_exporter
  5. firewall-cmd --zone=public --add-port=9116/tcp --permanent --set-description="snmp_exporter"
  6. firewall-cmd --reload

至此SNMP Exporter安装完成。可以在Prometheus的Target里查看状态。因为前面在安装Prometheus的时候已经配置了snmp的job，因此这里略过配置步骤。

参考