通过实现Locust的prometheus的exporter，将数据导入到proemtheus，然后由grafana进行数据展示。
什么是proemtheus，请转prometheus
什么是Grafana，请转grafana

Exporter编写

  1. 需要拓展一个web端接口/export/prometheus ，让prometheus根据配置定时拉取Metric信息，这里需要用到Prometheus 官方提供的 client 库，prometheus_client，来生成符合 Prometheus 规范的 metrics 信息。

    1. pip install prometheus_client

  2. 编写locust版的Exporter，prometheus_exporter.py ```python

    -- coding: utf-8 --

    file: prometheus_exporter

    Author: ShunZhe

    Date: 2021/7/31

    import six from itertools import chain

from flask import request, Response from locust import stats as locust_stats, runners as locust_runners from locust import User, task, events from prometheus_client import Metric, REGISTRY, exposition

This locustfile adds an external web endpoint to the locust master, and makes it serve as a prometheus exporter.

Runs it as a normal locustfile, then points prometheus to it.

locust -f prometheus_exporter.py —master

Lots of code taken from mbolek’s locust_exporter, thx mbolek!

class LocustCollector(object): registry = REGISTRY

  1. def __init__(self, environment, runner):
  2.     self.environment = environment
  3.     self.runner = runner
  5. def collect(self):
  6.     # collect metrics only when locust runner is spawning or running.
  7.     runner = self.runner
  9.     if runner and runner.state in (locust_runners.STATE_SPAWNING, locust_runners.STATE_RUNNING):
  10.         stats = []
  11.         for s in chain(locust_stats.sort_stats(runner.stats.entries), [runner.stats.total]):
  12.             stats.append({
  13.                 "method": s.method,
  14.                 "name": s.name,
  15.                 "num_requests": s.num_requests,
  16.                 "num_failures": s.num_failures,
  17.                 "avg_response_time": s.avg_response_time,
  18.                 "min_response_time": s.min_response_time or 0,
  19.                 "max_response_time": s.max_response_time,
  20.                 "current_rps": s.current_rps,
  21.                 "median_response_time": s.median_response_time,
  22.                 "ninetieth_response_time": s.get_response_time_percentile(0.9),
  23.                 # only total stats can use current_response_time, so sad.
  24.                 # "current_response_time_percentile_95": s.get_current_response_time_percentile(0.95),
  25.                 "avg_content_length": s.avg_content_length,
  26.                 "current_fail_per_sec": s.current_fail_per_sec
  27.             })
  29.         # perhaps StatsError.parse_error in e.to_dict only works in python slave, take notices!
  30.         errors = [e.to_dict() for e in six.itervalues(runner.stats.errors)]
  32.         metric = Metric('locust_user_count', 'Swarmed users', 'gauge')
  33.         metric.add_sample('locust_user_count', value=runner.user_count, labels={})
  34.         yield metric
  36.         metric = Metric('locust_errors', 'Locust requests errors', 'gauge')
  37.         for err in errors:
  38.             metric.add_sample('locust_errors', value=err['occurrences'],
  39.                               labels={'path': err['name'], 'method': err['method'],
  40.                                       'error': err['error']})
  41.         yield metric
  43.         is_distributed = isinstance(runner, locust_runners.MasterRunner)
  44.         if is_distributed:
  45.             metric = Metric('locust_slave_count', 'Locust number of slaves', 'gauge')
  46.             metric.add_sample('locust_slave_count', value=len(runner.clients.values()), labels={})
  47.             yield metric
  49.         metric = Metric('locust_fail_ratio', 'Locust failure ratio', 'gauge')
  50.         metric.add_sample('locust_fail_ratio', value=runner.stats.total.fail_ratio, labels={})
  51.         yield metric
  53.         metric = Metric('locust_state', 'State of the locust swarm', 'gauge')
  54.         metric.add_sample('locust_state', value=1, labels={'state': runner.state})
  55.         yield metric
  57.         stats_metrics = ['avg_content_length', 'avg_response_time', 'current_rps', 'current_fail_per_sec',
  58.                          'max_response_time', 'ninetieth_response_time', 'median_response_time',
  59.                          'min_response_time',
  60.                          'num_failures', 'num_requests']
  62.         for mtr in stats_metrics:
  63.             mtype = 'gauge'
  64.             if mtr in ['num_requests', 'num_failures']:
  65.                 mtype = 'counter'
  66.             metric = Metric('locust_stats_' + mtr, 'Locust stats ' + mtr, mtype)
  67.             for stat in stats:
  68.                 # Aggregated stat's method label is None, so name it as Aggregated
  69.                 # locust has changed name Total to Aggregated since 0.12.1
  70.                 if 'Aggregated' != stat['name']:
  71.                     metric.add_sample('locust_stats_' + mtr, value=stat[mtr],
  72.                                       labels={'path': stat['name'], 'method': stat['method']})
  73.                 else:
  74.                     metric.add_sample('locust_stats_' + mtr, value=stat[mtr],
  75.                                       labels={'path': stat['name'], 'method': 'Aggregated'})
  76.             yield metric

@events.init.add_listener def locust_init(environment, runner, **kwargs): print(“locust init event received”) if environment.web_ui and runner: @environment.web_ui.app.route(“/export/prometheus”) def prometheus_exporter(): registry = REGISTRY encoder, content_type = exposition.choose_encoder(request.headers.get(‘Accept’)) if ‘name[]’ in request.args: registry = REGISTRY.restricted_registry(request.args.get(‘name[]’)) body = encoder(registry) return Response(body, content_type=content_type)

  1.     REGISTRY.register(LocustCollector(environment, runner))

class Dummy(User): @task(20) def hello(self): pass

  2. 3. master启动prometheus_exporter.py文件
  3. ```dockerfile
  4. version: '3'
  6. networks: # 自定义网络(默认桥接), 不使用links通信
  7.   coll_network:
  8.     driver: bridge
  10. services:
  11.   master:
  12.     build: ./compose
  13.     networks:
  14.       - coll_network
  15.     ports:
  16.     - "8089:8089"
  17.     volumes:
  18.       - ./coll:/mnt/locust
  19. #    command: locust -f test_op.py --master --web-host=0.0.0.0 --first_index 0 --last_index 1000
  20.     command: locust -f prometheus_exporter.py --master --master-host=192.168.10.206
  22.   worker_1:
  23.     build: ./compose
  24.     networks:
  25.       - coll_network
  26.     volumes:
  27.       - ./coll:/mnt/locust
  28.     command: locust -f test_op.py --worker --master-host master --first_index 0 --last_index 1000
  29.     depends_on:
  30.       - master
  1. # 开始压测并通过浏览器访问，可以看到数据采集的信息如下
  3. http://0.0.0.0:8089/export/prometheus
  6. # HELP python_gc_objects_collected_total Objects collected during gc
  7. # TYPE python_gc_objects_collected_total counter
  8. python_gc_objects_collected_total{generation="0"} 31281.0
  9. python_gc_objects_collected_total{generation="1"} 6190.0
  10. python_gc_objects_collected_total{generation="2"} 351.0
  11. # HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
  12. # TYPE python_gc_objects_uncollectable_total counter
  13. python_gc_objects_uncollectable_total{generation="0"} 0.0
  14. python_gc_objects_uncollectable_total{generation="1"} 0.0
  15. python_gc_objects_uncollectable_total{generation="2"} 0.0
  16. # HELP python_gc_collections_total Number of times this generation was collected
  17. # TYPE python_gc_collections_total counter
  18. python_gc_collections_total{generation="0"} 165.0
  19. python_gc_collections_total{generation="1"} 14.0
  20. python_gc_collections_total{generation="2"} 1.0
  21. # HELP python_info Python platform information
  22. # TYPE python_info gauge
  23. python_info{implementation="CPython",major="3",minor="7",patchlevel="11",version="3.7.11"} 1.0
  24. # HELP process_virtual_memory_bytes Virtual memory size in bytes.
  25. # TYPE process_virtual_memory_bytes gauge
  26. process_virtual_memory_bytes 2.83045888e+08
  27. # HELP process_resident_memory_bytes Resident memory size in bytes.
  28. # TYPE process_resident_memory_bytes gauge
  29. process_resident_memory_bytes 4.9606656e+07
  30. # HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
  31. # TYPE process_start_time_seconds gauge
  32. process_start_time_seconds 1.62781476304e+09
  33. # HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
  34. # TYPE process_cpu_seconds_total counter
  35. process_cpu_seconds_total 1.3900000000000001
  36. # HELP process_open_fds Number of open file descriptors.
  37. # TYPE process_open_fds gauge
  38. process_open_fds 26.0
  39. # HELP process_max_fds Maximum number of open file descriptors.
  40. # TYPE process_max_fds gauge
  41. process_max_fds 1.048576e+06
  42. # HELP locust_user_count Swarmed users
  43. # TYPE locust_user_count gauge
  44. locust_user_count 30.0
  45. # HELP locust_errors Locust requests errors
  46. # TYPE locust_errors gauge
  47. # HELP locust_slave_count Locust number of slaves
  48. # TYPE locust_slave_count gauge
  49. locust_slave_count 4.0
  50. # HELP locust_fail_ratio Locust failure ratio
  51. # TYPE locust_fail_ratio gauge
  52. locust_fail_ratio 0.0
  53. # HELP locust_state State of the locust swarm
  54. # TYPE locust_state gauge
  55. locust_state{state="spawning"} 1.0
  56. # HELP locust_stats_avg_content_length Locust stats avg_content_length
  57. # TYPE locust_stats_avg_content_length gauge
  58. locust_stats_avg_content_length{method="接收数据",path="0 open"} 114.0
  59. locust_stats_avg_content_length{method="接收数据",path="3 heartbeat"} 1.0
  60. locust_stats_avg_content_length{method="接收数据",path="40 message ok"} 2.0
  61. locust_stats_avg_content_length{method="接收数据",path="42 join_ack"} 8961.85
  62. locust_stats_avg_content_length{method="接收数据",path="42 operation_ack"} 69.0
  63. locust_stats_avg_content_length{method="接收数据",path="42 set_doc_ack"} 28.0
  64. locust_stats_avg_content_length{method="接收数据",path="42 svr_user_status_change"} 154.0
  65. locust_stats_avg_content_length{method="Aggregated",path="Aggregated"} 435.48864105741427
  66. # HELP locust_stats_avg_response_time Locust stats avg_response_time
  67. # TYPE locust_stats_avg_response_time gauge
  68. locust_stats_avg_response_time{method="接收数据",path="0 open"} 0.41
  69. locust_stats_avg_response_time{method="接收数据",path="3 heartbeat"} 38.403225806451616
  70. locust_stats_avg_response_time{method="接收数据",path="40 message ok"} 0.92
  71. locust_stats_avg_response_time{method="接收数据",path="42 join_ack"} 76.88
  72. locust_stats_avg_response_time{method="接收数据",path="42 operation_ack"} 44.30930607853685
  73. locust_stats_avg_response_time{method="接收数据",path="42 set_doc_ack"} 57.24
  74. locust_stats_avg_response_time{method="接收数据",path="42 svr_user_status_change"} 77.93
  75. locust_stats_avg_response_time{method="Aggregated",path="Aggregated"} 43.82073523337464
  76. # HELP locust_stats_current_rps Locust stats current_rps
  77. # TYPE locust_stats_current_rps gauge
  78. locust_stats_current_rps{method="接收数据",path="0 open"} 0.0
  79. locust_stats_current_rps{method="接收数据",path="3 heartbeat"} 0.7
  80. locust_stats_current_rps{method="接收数据",path="40 message ok"} 0.0
  81. locust_stats_current_rps{method="接收数据",path="42 join_ack"} 0.0
  82. locust_stats_current_rps{method="接收数据",path="42 operation_ack"} 11.7
  83. locust_stats_current_rps{method="接收数据",path="42 set_doc_ack"} 0.0
  84. locust_stats_current_rps{method="接收数据",path="42 svr_user_status_change"} 0.0
  85. locust_stats_current_rps{method="Aggregated",path="Aggregated"} 12.4
  86. # HELP locust_stats_current_fail_per_sec Locust stats current_fail_per_sec
  87. # TYPE locust_stats_current_fail_per_sec gauge
  88. locust_stats_current_fail_per_sec{method="接收数据",path="0 open"} 0.0
  89. locust_stats_current_fail_per_sec{method="接收数据",path="3 heartbeat"} 0.0
  90. locust_stats_current_fail_per_sec{method="接收数据",path="40 message ok"} 0.0
  91. locust_stats_current_fail_per_sec{method="接收数据",path="42 join_ack"} 0.0
  92. locust_stats_current_fail_per_sec{method="接收数据",path="42 operation_ack"} 0.0
  93. locust_stats_current_fail_per_sec{method="接收数据",path="42 set_doc_ack"} 0.0
  94. locust_stats_current_fail_per_sec{method="接收数据",path="42 svr_user_status_change"} 0.0
  95. locust_stats_current_fail_per_sec{method="Aggregated",path="Aggregated"} 0.0
  96. # HELP locust_stats_max_response_time Locust stats max_response_time
  97. # TYPE locust_stats_max_response_time gauge
  98. locust_stats_max_response_time{method="接收数据",path="0 open"} 2.0
  99. locust_stats_max_response_time{method="接收数据",path="3 heartbeat"} 159.0
  100. locust_stats_max_response_time{method="接收数据",path="40 message ok"} 5.0
  101. locust_stats_max_response_time{method="接收数据",path="42 join_ack"} 184.0
  102. locust_stats_max_response_time{method="接收数据",path="42 operation_ack"} 317.0
  103. locust_stats_max_response_time{method="接收数据",path="42 set_doc_ack"} 280.0
  104. locust_stats_max_response_time{method="接收数据",path="42 svr_user_status_change"} 185.0
  105. locust_stats_max_response_time{method="Aggregated",path="Aggregated"} 317.0
  106. # HELP locust_stats_ninetieth_response_time Locust stats ninetieth_response_time
  107. # TYPE locust_stats_ninetieth_response_time gauge
  108. locust_stats_ninetieth_response_time{method="接收数据",path="0 open"} 1.0
  109. locust_stats_ninetieth_response_time{method="接收数据",path="3 heartbeat"} 42.0
  110. locust_stats_ninetieth_response_time{method="接收数据",path="40 message ok"} 1.0
  111. locust_stats_ninetieth_response_time{method="接收数据",path="42 join_ack"} 150.0
  112. locust_stats_ninetieth_response_time{method="接收数据",path="42 operation_ack"} 51.0
  113. locust_stats_ninetieth_response_time{method="接收数据",path="42 set_doc_ack"} 67.0
  114. locust_stats_ninetieth_response_time{method="接收数据",path="42 svr_user_status_change"} 150.0
  115. locust_stats_ninetieth_response_time{method="Aggregated",path="Aggregated"} 60.0
  116. # HELP locust_stats_median_response_time Locust stats median_response_time
  117. # TYPE locust_stats_median_response_time gauge
  118. locust_stats_median_response_time{method="接收数据",path="0 open"} 0.0
  119. locust_stats_median_response_time{method="接收数据",path="3 heartbeat"} 34.0
  120. locust_stats_median_response_time{method="接收数据",path="40 message ok"} 1.0
  121. locust_stats_median_response_time{method="接收数据",path="42 join_ack"} 61.0
  122. locust_stats_median_response_time{method="接收数据",path="42 operation_ack"} 40.0
  123. locust_stats_median_response_time{method="接收数据",path="42 set_doc_ack"} 50.0
  124. locust_stats_median_response_time{method="接收数据",path="42 svr_user_status_change"} 62.0
  125. locust_stats_median_response_time{method="Aggregated",path="Aggregated"} 40.0
  126. # HELP locust_stats_min_response_time Locust stats min_response_time
  127. # TYPE locust_stats_min_response_time gauge
  128. locust_stats_min_response_time{method="接收数据",path="0 open"} 0.0
  129. locust_stats_min_response_time{method="接收数据",path="3 heartbeat"} 27.0
  130. locust_stats_min_response_time{method="接收数据",path="40 message ok"} 0.0
  131. locust_stats_min_response_time{method="接收数据",path="42 join_ack"} 50.0
  132. locust_stats_min_response_time{method="接收数据",path="42 operation_ack"} 31.0
  133. locust_stats_min_response_time{method="接收数据",path="42 set_doc_ack"} 42.0
  134. locust_stats_min_response_time{method="接收数据",path="42 svr_user_status_change"} 50.0
  135. locust_stats_min_response_time{method="Aggregated",path="Aggregated"} 0.0
  136. # HELP locust_stats_num_failures_total Locust stats num_failures
  137. # TYPE locust_stats_num_failures_total counter
  138. locust_stats_num_failures{method="接收数据",path="0 open"} 0.0
  139. locust_stats_num_failures{method="接收数据",path="3 heartbeat"} 0.0
  140. locust_stats_num_failures{method="接收数据",path="40 message ok"} 0.0
  141. locust_stats_num_failures{method="接收数据",path="42 join_ack"} 0.0
  142. locust_stats_num_failures{method="接收数据",path="42 operation_ack"} 0.0
  143. locust_stats_num_failures{method="接收数据",path="42 set_doc_ack"} 0.0
  144. locust_stats_num_failures{method="接收数据",path="42 svr_user_status_change"} 0.0
  145. locust_stats_num_failures{method="Aggregated",path="Aggregated"} 0.0
  146. # HELP locust_stats_num_requests_total Locust stats num_requests
  147. # TYPE locust_stats_num_requests_total counter
  148. locust_stats_num_requests{method="接收数据",path="0 open"} 100.0
  149. locust_stats_num_requests{method="接收数据",path="3 heartbeat"} 62.0
  150. locust_stats_num_requests{method="接收数据",path="40 message ok"} 100.0
  151. locust_stats_num_requests{method="接收数据",path="42 join_ack"} 100.0
  152. locust_stats_num_requests{method="接收数据",path="42 operation_ack"} 1859.0
  153. locust_stats_num_requests{method="接收数据",path="42 set_doc_ack"} 100.0
  154. locust_stats_num_requests{method="接收数据",path="42 svr_user_status_change"} 100.0
  155. locust_stats_num_requests{method="Aggregated",path="Aggregated"} 2421.0

Prometheus部署

  1. 部署Prometheus，拉取metric数据

    1. docker pull prom/prometheus

  2. 创建prometheus.yml配置文件（与docker-compose.yml同目录） ```yaml global: scrape_interval: 15s evaluation_interval: 15s

scrape_configs:

  • job_name: prometheus static_configs:

    • targets: [‘localhost:9090’] labels: instance: prometheus

  • job_name: locust metrics_path: ‘/export/prometheus’ static_configs:

    • targets: [‘192.168.10.206:8089’] # 地址修改为实际地址 labels: instance: locust ```
  1. 启动Prometheus，将9090端口映射出来 ```bash

    prometheus.yml目录下执行命令

docker run -itd -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

  2. 4. 访问http://<locust_master_ip>:9090/targets查看数据收集是否正常
  4. ![image.png](https://cdn.nlark.com/yuque/0/2021/png/1170640/1627817023918-7622d855-4f8e-4fb6-b678-fe83b8d06349.png#clientId=u429fe8ed-2727-4&from=paste&height=551&id=u6a658126&margin=%5Bobject%20Object%5D&name=image.png&originHeight=1102&originWidth=3490&originalType=binary&ratio=1&size=377984&status=done&style=none&taskId=ucc435c47-7afe-4ffa-b7fe-793379e1c69&width=1745)
  5. <a name="tFt5y"></a>
  6. #### Grafana部署&配置
  8. 1. 拉取grafana镜像
  9. ```bash
  10. docker pull grafana/grafana

  1. 启动grafana服务

    1. docker run -d --name grafana -p 3000:3000 grafana/grafana

  2. 访问http://:3000/进入grafana，默认账号密码为admin。

image.png

  1. 选择添加prometheus数据源

Configuration — Data sources — Add data source — 选择Prometheus
这里的URL 设置为 prometheus的地址：http://:9090
image.png

  1. 导入DashBoard

通过Dashboard 找到合适的导入
➕ — import — load，这里导入的dashboard id是12081
效果展示：
image.png

附项目目录：
image.png
Dockerfile文件如下：

  1. FROM python:3.7
  3. # 设置 python 环境变量
  4. ENV PYTHONUNBUFFERED 1
  5. ENV APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=DontWarn
  7. COPY pip.conf /root/.pip/pip.conf
  10. # 创建 locust 文件夹
  11. RUN mkdir -p /mnt/locust
  13. # 将 locust 文件夹为工作目录
  14. WORKDIR /mnt/locust
  16. # 将当前目录加入到工作目录中（. 表示当前目录）
  17. ADD . /mnt/locust
  19. # 更新pip版本
  20. RUN /usr/local/bin/python -m pip install --upgrade pip
  22. # 利用 pip 安装依赖
  23. RUN pip install -r requirements.txt
  24. RUN pip install websocket
  25. RUN pip install websocket-client
  26. RUN pip3 install prometheus-client

docker-compose.yml文件如下：

  1. version: '3'
  3. networks: # 自定义网络(默认桥接), 不使用links通信
  4.   coll_network:
  5.     driver: bridge
  7. services:
  8.   master:
  9.     build: ./compose
  10.     networks:
  11.       - coll_network
  12.     ports:
  13.     - "8089:8089"
  14.     volumes:
  15.       - ./coll:/mnt/locust
  16. #    command: locust -f test_op.py --master --web-host=0.0.0.0 --first_index 0 --last_index 1000
  17.     command: locust -f prometheus_exporter.py --master --master-host=192.168.10.206
  19.   worker_1:
  20.     build: ./compose
  21.     networks:
  22.       - coll_network
  23.     volumes:
  24.       - ./coll:/mnt/locust
  25.     command: locust -f test_op.py --worker --master-host master --first_index 0 --last_index 1000
  26.     depends_on:
  27.       - master
  29.   worker_2:
  30.     build: ./compose
  31.     networks:
  32.       - coll_network
  33.     volumes:
  34.       - ./coll:/mnt/locust
  35.     command: locust -f test_op.py --worker --master-host master --first_index 1000 --last_index 2000
  36.     depends_on:
  37.       - master
  39.   worker_3:
  40.     build: ./compose
  41.     networks:
  42.       - coll_network
  43.     volumes:
  44.       - ./coll:/mnt/locust
  45.     command: locust -f test_op.py --worker --master-host master --first_index 2000 --last_index 3000
  46.     depends_on:
  47.       - master
  49.   worker_4:
  50.     build: ./compose
  51.     networks:
  52.       - coll_network
  53.     volumes:
  54.       - ./coll:/mnt/locust
  55.     command: locust -f test_op.py --worker --master-host master --first_index 3000 --last_index 4000
  56.     depends_on:
  57.       - master