一、压测目的
thanos-sidecar验证采集2000/4000/6000个节点对性能影响
**
二、压测准备
1、采集源机器配置
CPU:24核
MEM:125G
2、export数量
4000个节点
3、单条metrics大小
总大小为2100字节
# HELP requests_total HTTP requests total
# TYPE requests_total counter
requests_total{clientip="127.0.0.1",dc="dx",ent="xfime1",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.2",dc="dx",ent="xfime2",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.3",dc="gz",ent="xfime3",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.4",dc="hu",ent="xfime4",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.5",dc="da",ent="xfime5",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.17",dc="azx",ent="xfi6me",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.14",dc="dax",ent="xfime6",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.121",dc="dfxx",ent="xfime7",method="get",sub="iat",type="percent"} 3.0
requests_total{clientip="127.0.0.11",dc="dxa",ent="xfime8",method="get",sub="iat",type="percent"} 3.0
# HELP requests_created HTTP requests total
# TYPE requests_created gauge
requests_created{clientip="127.0.0.1",dc="dx",ent="xfime1",method="get",sub="iat",type="percent"} 1.621652127910199e+09
requests_created{clientip="127.0.0.2",dc="dx",ent="xfime2",method="get",sub="iat",type="percent"} 1.6216521279102333e+09
requests_created{clientip="127.0.0.3",dc="gz",ent="xfime3",method="get",sub="iat",type="percent"} 1.6216521279102564e+09
requests_created{clientip="127.0.0.4",dc="hu",ent="xfime4",method="get",sub="iat",type="percent"} 1.6216521279102776e+09
requests_created{clientip="127.0.0.5",dc="da",ent="xfime5",method="get",sub="iat",type="percent"} 1.621652127910298e+09
requests_created{clientip="127.0.0.17",dc="azx",ent="xfi6me",method="get",sub="iat",type="percent"} 1.6216521279103184e+09
requests_created{clientip="127.0.0.14",dc="dax",ent="xfime6",method="get",sub="iat",type="percent"} 1.6216521279103408e+09
requests_created{clientip="127.0.0.121",dc="dfxx",ent="xfime7",method="get",sub="iat",type="percent"} 1.6216521279103608e+09
requests_created{clientip="127.0.0.11",dc="dxa",ent="xfime8",method="get",sub="iat",type="percent"} 1.6216521279103804e+09
三、压测过程
1、压测思路
- thanos-sidecar采集2000/4000/6000个节点数据源的性能指标、CPU、mem等指标结果对比。
2、压测准备
采集4000个节点的metrics数据,每秒采集一次,单条metrics数据大小在2100个字节。
3、prometheus+sidecar+S3架构图
四、压测结果
1)原生prometheus+thanos-sidecar性能延迟数据
计算方法:采集4000个节点采集性能指标的平均值,采集一个小时+数据
结论:
- 2000节点延迟在1.5ms。
- 4000节点延迟在3ms。
- 6000节点延迟在6ms。
2)原生prometheus+thanos-sidecar机器资源使用
结论:
- 6000个节点使用CPU个数在2个。
- 6000个节点使用内存在1.4G
五、压测结论
- 采集6000个节点性能延迟在6ms
- 6000个节点使用内存在1.4G左右,CPU使用数量在2核
- 当前每个机房的节点数在1000个节点左右,可以满足我们的性能需求