Redis大key热key慢key分析

热key

命令：—hotkeys

  1. redis-cli -h 10.241.80.237 -p 7100 -c -a Bonree@365  --hotkeys -i 0.1

-i 参数为每个SCAN命令指定秒数

执行结果

  1. [BONREE bonree@br-server-010:/data/br/base/redis]$ redis-cli -h br-server-010 -p 7100 -a Bonree@365 --hotkeys -i 0.1
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  4. # Scanning the entire keyspace to find hot keys as well as
  5. # average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
  6. # per 100 SCAN commands (not usually needed).
  8. [00.00%] Hot key 'pf_1_spring:session:sessions:05bf0382-2a52-4050-ad4b-9857a8fed8fe' found so far with counter 1
  9. [00.00%] Hot key 'APMAGENTSTAT21' found so far with counter 6
  10. [00.00%] Hot key 'APMconfigure:t_srv_cfg_agent:agent_0' found so far with counter 6
  11. [00.00%] Hot key 'APMconfigure:t_srv_cfg_acc_snapshot_gat:cluster:38522' found so far with counter 3
  12. [00.00%] Hot key 'APMagent:146c70b3-db59-9a57-ac57-02b6c047543d' found so far with counter 6
  13. [05.42%] Hot key 'APMpf_auth_server_api' found so far with counter 4
  14. [05.42%] Hot key 'APMServiceConfigInfo:agent_id:42704' found so far with counter 5
  15. [05.42%] Hot key 'APMconfigure:t_srv_cfg_acc_log:global:1157' found so far with counter 8
  16. [10.34%] Hot key 'APMmachine_id:2645:proc' found so far with counter 4
  17. [10.34%] Hot key 'pf_1_spring:session:sessions:expires:cb5d46ca-6f01-4c27-96d9-2dc562bf9177' found so far with counter 1
  18. [10.34%] Hot key 'pf_1_spring:session:sessions:expires:80d8d114-88d6-47a0-9914-1810c692d976' found so far with counter 3
  19. [15.27%] Hot key 'APMconfigure:t_srv_cfg_appprotection:global:1157' found so far with counter 6
  20. [15.27%] Hot key 'pf_1_spring:session:sessions:a0905aaa-d6ff-4adb-966a-c593c732392c' found so far with counter 5
  21. [15.27%] Hot key 'APMONLINE_MACHINE' found so far with counter 5
  22. [15.27%] Hot key 'APMuseGlobalConfig' found so far with counter 9
  23. [15.27%] Hot key 'pf_1_spring:session:sessions:expires:68de5685-0772-4b4c-b040-a9374d095561' found so far with counter 2
  24. [15.27%] Hot key 'APMAGENTSTAT25' found so far with counter 6
  25. [15.27%] Hot key 'APMMACHINESTAT1' found so far with counter 6
  26. [21.67%] Hot key 'pf_1_spring:session:expirations:1638428040000' found so far with counter 6
  27. [21.67%] Hot key 'APMMACHINESTAT0' found so far with counter 6
  28. [21.67%] Hot key 'APMMACHINESTAT28' found so far with counter 6
  29. [27.59%] Hot key 'APMagent:42704' found so far with counter 5
  30. [27.59%] Hot key 'APMMACHINESTAT5' found so far with counter 6
  31. [32.51%] Hot key 'APMconfigure:t_srv_cfg_acc_cluster:cluster:38522' found so far with counter 6
  32. [37.44%] Hot key 'APMconfigure:t_srv_bt_naming_cfg:global:1157' found so far with counter 6
  33. [37.44%] Hot key 'APMconfigure:t_srv_cfg_agent:agent_42700' found so far with counter 6
  34. [42.36%] Hot key 'APMplatform_user_mapping_key' found so far with counter 6
  35. [72.91%] Hot key 'APMAGENTSTAT18' found so far with counter 7
  36. [87.68%] Hot key 'APMmachine_id:2646:proc' found so far with counter 7
  38. -------- summary -------
  40. Sampled 203 keys in the keyspace!
  41. hot key found with counter: 9    keyname: APMuseGlobalConfig
  42. hot key found with counter: 8    keyname: APMconfigure:t_srv_cfg_acc_log:global:1157
  43. hot key found with counter: 7    keyname: APMAGENTSTAT18
  44. hot key found with counter: 7    keyname: APMmachine_id:2646:proc
  45. hot key found with counter: 6    keyname: APMAGENTSTAT21
  46. hot key found with counter: 6    keyname: APMconfigure:t_srv_cfg_agent:agent_0
  47. hot key found with counter: 6    keyname: APMagent:146c70b3-db59-9a57-ac57-02b6c047543d
  48. hot key found with counter: 6    keyname: APMconfigure:t_srv_cfg_appprotection:global:1157
  49. hot key found with counter: 6    keyname: APMAGENTSTAT25
  50. hot key found with counter: 6    keyname: APMMACHINESTAT1
  51. hot key found with counter: 6    keyname: pf_1_spring:session:expirations:1638428040000
  52. hot key found with counter: 6    keyname: APMMACHINESTAT0
  53. hot key found with counter: 6    keyname: APMMACHINESTAT28
  54. hot key found with counter: 6    keyname: APMMACHINESTAT5
  55. hot key found with counter: 6    keyname: APMconfigure:t_srv_cfg_acc_cluster:cluster:38522
  56. hot key found with counter: 6    keyname: APMconfigure:t_srv_bt_naming_cfg:global:1157

结果分析

summary部分会将访问次数最多的key列出来，counter: 9表示访问的次数

  1. Sampled 203 keys in the keyspace!
  2. hot key found with counter: 9    keyname: APMuseGlobalConfig
  3. hot key found with counter: 8    keyname: APMconfigure:t_srv_cfg_acc_log:global:1157
  4. hot key found with counter: 7    keyname: APMAGENTSTAT18
  5. hot key found with counter: 7    keyname: APMmachine_id:2646:proc
  6. hot key found with counter: 6    keyname: APMAGENTSTAT21
  7. hot key found with counter: 6    keyname: APMconfigure:t_srv_cfg_agent:agent_0
  8. hot key found with counter: 6    keyname: APMagent:146c70b3-db59-9a57-ac57-02b6c047543d
  9. hot key found with counter: 6    keyname: APMconfigure:t_srv_cfg_appprotection:global:1157
  10. hot key found with counter: 6    keyname: APMAGENTSTAT25
  11. hot key found with counter: 6    keyname: APMMACHINESTAT1
  12. hot key found with counter: 6    keyname: pf_1_spring:session:expirations:1638428040000
  13. hot key found with counter: 6    keyname: APMMACHINESTAT0
  14. hot key found with counter: 6    keyname: APMMACHINESTAT28
  15. hot key found with counter: 6    keyname: APMMACHINESTAT5
  16. hot key found with counter: 6    keyname: APMconfigure:t_srv_cfg_acc_cluster:cluster:38522
  17. hot key found with counter: 6    keyname: APMconfigure:t_srv_bt_naming_cfg:global:1157

问题

热key扫描是基于LFU算法发现机制，所以如果要记录热key，内存策略必须配置成allkeys-lfu和volatile-lfu

  1. [BONREE bonree@br-sdk-base01:/data/br]$ redis-cli -h 192.168.0.11 -p 7000 -c --hotkeys -i 0.1
  3. # Scanning the entire keyspace to find hot keys as well as
  4. # average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
  5. # per 100 SCAN commands (not usually needed).
  7. Error: ERR An LFU maxmemory policy is not selected, access frequency not tracked. Please note that when switching between policies at runtime LRU and LFU data will take some time to adjust.

大key

命令：—bigkeys

  1. redis-cli -h 192.168.0.11 -p 7000 -c --bigkeys -i 0.1

关键参数

执行结果

  1. [BONREE bonree@br-sdk-base01:/data/br]$ redis-cli -h 192.168.0.11 -p 7000 -c --bigkeys -i 0.1
  3. # Scanning the entire keyspace to find biggest keys as well as
  4. # average sizes per key type.  You can use -i 0.1 to sleep 0.1 sec
  5. # per 100 SCAN commands (not usually needed).
  7. [00.55%] Biggest hash   found so far 'APMmachine_id:33:net_adapter' with 2 fields
  8. [00.66%] Biggest string found so far '1_9_ALERT_T_202112021322' with 1 bytes
  9. [00.67%] Biggest string found so far 'APMAgentIdentity:128' with 92 bytes
  10. [00.74%] Biggest hash   found so far 'APMETL_ENTITY_CACHE_sql_47' with 230 fields
  11. [01.11%] Biggest hash   found so far 'APMETL_ENTITY_CACHE_bt_27' with 260 fields
  12. [01.60%] Biggest hash   found so far 'APMmethodinfo:6' with 718 fields
  13. [02.79%] Biggest hash   found so far 'APMETL_ENTITY_CACHE_becd_76' with 4841 fields
  14. [06.71%] Biggest string found so far '1_B_IPSTORAGE_V6_8' with 4334766 bytes
  15. [07.16%] Biggest hash   found so far 'APMapp_id:12:bt' with 6447 fields
  16. [09.41%] Biggest set    found so far 'APMapp_id:6:bt_becd_v621' with 7999 members
  17. [17.70%] Biggest hash   found so far '1_9_SDK_VISIT_1638374400000_4' with 7403 fields
  18. [17.96%] Biggest string found so far '1_8_MAU_KEY_MAIN2' with 16774953 bytes
  19. [23.02%] Biggest set    found so far 'APMapp_id:3:bt_becd_v621' with 99479 members
  20. [87.10%] Biggest string found so far '1_8_MAU_KEY_MAIN6' with 16775451 bytes
  22. -------- summary -------
  24. Sampled 268959 keys in the keyspace!
  25. Total key length in bytes is 7366007 (avg len 27.39)
  27. Biggest string found '1_8_MAU_KEY_MAIN6' has 16775451 bytes
  28. Biggest    set found 'APMapp_id:3:bt_becd_v621' has 99479 members
  29. Biggest   hash found '1_9_SDK_VISIT_1638374400000_4' has 7403 fields
  31. 268504 strings with 69254134 bytes (99.83% of keys, avg size 257.93)
  32. 0 lists with 0 items (00.00% of keys, avg size 0.00)
  33. 4 sets with 193105 members (00.00% of keys, avg size 48276.25)
  34. 451 hashs with 120050 fields (00.17% of keys, avg size 266.19)
  35. 0 zsets with 0 members (00.00% of keys, avg size 0.00)

结果分析

主要看summary部分信息，String类型控制在1mb以 内, hash、list、set、zset等元素个数要控制在5000以内

所以1_8_MAU_KEY_MAIN6和APMapp_id:3:bt_becd_v621，1_9_SDK_VISIT_1638374400000_4这三个key都是比较大的key

  1. -------- summary -------
  3. Sampled 268959 keys in the keyspace!
  4. Total key length in bytes is 7366007 (avg len 27.39)
  6. Biggest string found '1_8_MAU_KEY_MAIN6' has 16775451 bytes          #string类型的key ，1_8_MAU_KEY_MAIN6，占用16775451字节
  7. Biggest    set found 'APMapp_id:3:bt_becd_v621' has 99479 members    #set类型的key，APMapp_id:3:bt_becd_v621有99479个成员
  8. Biggest   hash found '1_9_SDK_VISIT_1638374400000_4' has 7403 fields #hash类型的key，1_9_SDK_VISIT_1638374400000_4有7403字段

慢key

命令：slowlog get len

  1. redis-cli -c -p 7100 -a Bonree@365 slowlog get 5

slowlog get 5 获取最近5条慢日志

关键参数

  1. slowlog-log-slower-than 50000      #执行时间大于多少微秒的查询进行记录 
  2. slowlog-max-len         1000       #最大保存多少条慢日志

执行结果

  1. [BONREE bonree@stresskafka01:/data/br/base/redis]$ redis-cli -c -p 7100 -a Bonree@365 slowlog get 5
  2. Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
  3. 1) 1) (integer) 0                          #slowlog 唯一标识符
  4.    2) (integer) 1638340843                 #命令执行时间点，unix时间戳
  5.    3) (integer) 8045318                    #命令执行耗时，单位：微妙
  6.    4) 1) "DEL"                             #执行的命令
  7.       2) "counter:{06S}:__rand_int__"
  8.       3) "key:{06S}:__rand_int__"
  9.       4) "myhash:{06S}"
  10.       5) "mylist:{06S}"
  11.    5) "10.241.90.34:54930"                #客户端ip
  12.    6) ""                                  #客户端名称

结果分析

1、客户端10.241.90.34:54930在2021-12-01 14:40:43执行DEL 命令，执行DEL命令，耗时8045318微妙

2、slowlog-max-len不要设置的太大，因为慢日志是直接写在内存中的，设置太大会占用redis一定内存