Data distribution optimization: how to deal with data skew?

In a slice cluster , The data will be distributed to different instances according to certain distribution rules . such as , In the use of Redis Cluster or Codis when , The data will first follow CRC The calculated value of the algorithm is right Slot（ Logic slot ） modulus , meanwhile , be-all Slot It will be assigned to different instances by the operation and maintenance administrator . such , The data is saved to the corresponding instance .

Although this method is relatively simple to implement , But it can easily lead to a problem ： Data skew .

There are two types of data skew .

• The amount of data is skewed ： In some cases , The data on the instance is unevenly distributed , There is a lot of data on an instance .
• Data access ： Although the amount of data on each cluster instance varies little , But the data on an instance is hot data , Very frequently visited .

If data skew occurs , So a lot of data is saved , Or the processing pressure of instances with hot data will increase , Slow down , It may even cause the instance to run out of memory resources , To collapse . This is what we should avoid when applying slicing clusters .

Today's lesson , I'll talk to you , How do these two kinds of data skew happen , How should we deal with .

Causes and Countermeasures of data skew

First , Let's look at the causes of data skew and Countermeasures .

When data skew occurs , The data is unevenly distributed on multiple instances of the slice cluster , A large amount of data is concentrated on one or several instances , As shown in the figure below ：

that , How does data skew come about ？ There are three main reasons , They are saved on an instance bi