redis(4) breakdown through avalanche

1. redis breakdown

It turned out to be there, but it became invalid. Breakdown

When redis is used as a cache, when the key expires due to the expiration time or the LRU LFU algorithm makes the total key of redis invalid, there is suddenly high concurrency. At this time, there is no such key in redis, and the data will be sent to the database at that time. It is equivalent to breaking down redis and going to the database


Method 1: setnx to set the lock

Assume that 3 threads arrive at the same time.

1. Go to get key first to get the target key. Nli is obtained at this time

2. Set the lock, setnx lockkey lockvalue. Setnx is called. It will be created successfully when there is no lockkey. If there is, it cannot be created. In other words, only after setnx is successful, can you access the database

It is possible that all three threads did not get the get key. Since redis is single-threaded, only one thread may succeed in setnx. Others are waiting. After the first thread releases the lock, the subsequent threads can setnx successfully.

3. Only the thread that has acquired the lock can access the database.

The above method will also have new problems:

What if the thread hangs suddenly after the first thread acquires the lock? At this time, the following threads cannot get the lock.

At this point, we can set an expiration time for the lock. The lock can also be released when the time is up.

But if the thread is not hung up, the expiration time is up, and the lock is almost released, but querying the database has not ended, at this time the next thread will be stuck at the step of querying data

Solution: One thread fetches and queries the database

Another thread monitors whether the data has been retrieved, and if the lock has not expired, the expiration time of the lock is updated.

Method 2: Set hotspot data without expiration

Method Three: Use Bloom Filter

Bloom's principle:

It is to use the bitmap bit to mark all the elements on the bitmap bit through a certain algorithm.

The elements in the request are also marked on the bitmap through a certain algorithm. But this may be mislabeled, but it can reduce the number of breakdowns.


eg: 1. Add all data to redis. BF.add all data

2. By judging whether it exists. If it returns 1, it may exist and can be queried in the database. BF.exist judges whether it exists, 1 may be there, and 0 must not be.

If it returns to 0, it must not exist, and just put it back directly.

Breakdown graph

2. Penetration

The data itself does not exist, it penetrates directly

1. That is, the key obtained by the request call is not in redis or the database. It is the data that the business needs to query, and there is no data in the database at all, it is empty.

Solve the problem:

Bloom filter. Disadvantages, cannot be deleted. Use cuckoo, you can delete

Do some verification at the interface layer, such as user authentication, or some range verification, and intercept part of it

Or for the data that cannot be obtained, cache some nulls, but the set expiration time should be small.

3. Avalanche

A large number of keys expire at the same time, and a large number of keys fail. A large number of visits reach the database DB. Equivalent to exchange blood.

Avalanches and breakdowns are a bit similar.

But there are differences: breakdown can be understood as high concurrency of individual data

Avalanche is the high concurrency of large amounts of data.

Solution: Random expiration time. Applicable when time is irrelevant

Another scenario is that the key must be changed at zero (a fixed point in time). At this time, the random expiration time is not applicable, and the breakdown scheme is strongly relied on at this time.