Bigkey and hotspot key of redis principle

BigKey

bigKey Refer to key Corresponding value The memory space occupied is relatively large , For example, a string type value It can be stored to 512MB, A list type value Up to storage 2（32）-1 Elements .bigKey Both spatial and temporal complexity are not very friendly , Next we will introduce its harm .

bigKey The harm of is reflected in five aspects

• Uneven memory space （ Balance ）： For example, in Redis  Cluster in ,bigKey The memory space of the node will be unevenly used .
• Time out blocking ： because Redis Single thread features , operation bigKey More time-consuming , It means blocking Redis The possibility increases .
• Network congestion ： Every time to get bigKey The generated network traffic is large , Suppose a bigKey by 1MB, Accesses per second are 1000, So every second produces 1000MB Of traffic , For ordinary gigabit network card （ An byte is 128MB/S） It's a disaster for our servers .
• Active delete , Passive expiration delete , Data migration and other operations , Due to processing Key Too long , Blocking can also occur .
• list , hash And so on , a large number of elements , It needs to be traversed many times , Multiple system calls also take a lot of time .

Judge a key Size , have access to debug object key command see serializedlength  size , as follows ：

127.0.0.1:6379> set name "11111111111111111111111111111111111"
OK
127.0.0.1:6379> debug object name
Value at:0x7f9563606630 refcount:1 encoding:embstr serializedlength:12 lru:12109803 lru_seconds_idle:11
127.0.0.1:6379> strlen name
(integer) 35

Redis Scan The command iterates over the database keys in the database

SCAN Command is a cursor based iterator , After each call , Will return a new cursor to the user , The user needs to use this new cursor as the SCAN Cursor parameters for the command , To continue the previous iteration process .

SCAN Returns an array of two elements , The first element is a new cursor for the next iteration , The second element is an array , This array contains all the elements that are iterated . If the new cursor returns 0 Indicates that the iteration is over .

127.0.0.1:6379> scan 0
1) "0"
2) 1) "cities"
   2) "2021-06-04_users"
   3) "name"
   4) "2021-06-03_users"
   5) "key"
127.0.0.1:6379>

Looking for hot spots key

Hot news events or goods Usually Will give the system Bring huge traffic , For storing this information Redis Come on But it is a huge challenge . With Redis Cluster As an example , It will cause the overall flow imbalance , Individual nodes appear OPS Too big a situation , In extreme cases, hot spots Key

Even more than Redis What it can bear OPS, So look for hot spots Key It is very important for development and operation and maintenance personnel .

hotspot Key Hazards of ：

• Requests are too focused on several slices , This leads to excessive pressure in these segments , Unable to function as a cluster .
• When the pressure of a single slice is too high , The problem cannot be solved by adding slices .

ops What is it? ？

redis Medium OPS namely operation per second Number of operations per second . Means every second to Redis Persistent operation of

• agent

        image Twemproxy  Codis These agent-based Redis Distributed architecture , All client requests are completed through the proxy , This architect is best suited Do hot spots Key The statistical , Because agents are all Redis client and Server side The bridge .

• Redis  Server side

      Use monitor Command statistics hotspot Key yes Many developers and operation and maintenance personnel first thought ,monitor command Can be monitored Redis All commands executed

ZBMAC-2f32839f6:redis-master yangyanping$ ./src/redis-cli 

127.0.0.1:6379> set myname yangyanping

OK

127.0.0.1:6379> get myname

"yangyanping"

127.0.0.1:6379> set user:1 30

OK

127.0.0.1:6379> get user:1

"30"

 monitor The result of the execution of the order

127.0.0.1:6379> monitor

OK

1622770113.161698 [0 127.0.0.1:54652] "COMMAND"

1622770129.910122 [0 127.0.0.1:54652] "set" "myname" "yangyanping"

1622770142.613021 [0 127.0.0.1:54652] "get" "myname"

1622770156.695975 [0 127.0.0.1:54652] "set" "user:1" "30"

1622770161.058709 [0 127.0.0.1:54652] "get" "user:1"

utilize monitor Results of command You can count the hot spots in a period of time Key

  monitor When the command is high and issued, it will affect Redis performance

• machine

Redis Client side usage TCP The protocol communicates with the server , The communication protocol is RESP. If you stand at the angle of the machine , You can do this by checking all the Redis Port of TCP Data packets To grab Finish hotspot Key The statistics of .

Redis Server and client through RESP（REdis Serialization Protocol） Protocol communication . It's an intuitive text protocol , The advantage is that it's very easy to implement , Excellent parsing performance

Why design such a text protocol that wastes traffic ？

Redis The designers of think that the bottleneck of database generally does not lie in network traffic , But the internal logic processing of the database itself .Redis Put all the data in memory , Use a single thread to provide external services , A single node is running full of CPU The core case can achieve 10w/s The superelevation of QPS.