Redis advanced learning summary (Docker build environment)

Article Directory

One, master-slave replication

One master server master, multiple slave servers slave. Main functions: data redundancy, failure recovery, load balancing, and high-availability clusters.

The master can write, but the slave node cannot write and can only read. The information in the master will be saved by the slave node. The data of the slave node is still retained when the master node is down, and the identity of the slave node remains unchanged. After the master node recovers, the slave node Can continue to connect

xxxx
xxx

View the basic information of master-slave replication

127.0.0.1:6379> info replication
# Replication
role:master   # 当前角色master
connected_slaves:0  # 从机0个
master_failover_state:no-failover
master_replid:f298b23a9266dd0860de5be3d776ada013e40e86
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

Environment construction (one area N structure)

1. Start three Redis containers first

[email protected]:~# docker run -p 6379:6379 -d --name="redis00" bc8d70f9ef6c 
2e76a9ca22c00e027d55d971b0d5bc265d16ad33f7891733b87fcbfcda592eef

[email protected]:~# docker run -p 6389:6389 -d --name="redis01" bc8d70f9ef6c 
51610b97d4cbdf745d1514800f3a00f5ffc7ea1d4feb23a4e53781bfb50777d0

[email protected]:~# docker run -p 6399:6399 -d --name="redis02" bc8d70f9ef6c 
1fcda7dd77cba51aa036591fe1a17dfb672f168e7b4d06ca15bbee17414329af

[email protected]:~# docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED         STATUS         PORTS                              NAMES
1fcda7dd77cb   bc8d70f9ef6c   "docker-entrypoint.s…"   3 minutes ago   Up 3 minutes   6379/tcp, 0.0.0.0:6399->6399/tcp   redis02
51610b97d4cb   bc8d70f9ef6c   "docker-entrypoint.s…"   4 minutes ago   Up 3 minutes   6379/tcp, 0.0.0.0:6389->6389/tcp   redis01
2e76a9ca22c0   bc8d70f9ef6c   "docker-entrypoint.s…"   7 seconds ago   Up 5 seconds   0.0.0.0:6379->6379/tcp             redis00

2. Copy the official redis.conf file from the host to the three containers

[email protected]:~# docker cp ./redis.conf 2e76a9ca22c0:/usr/local/bin/
[email protected]:~# docker cp ./redis.conf 51610b97d4cb:/usr/local/bin/
[email protected]:~# docker cp ./redis.conf 1fcda7dd77cb:/usr/local/bin/

3. Enter the container to modify the configuration file, taking the redis00 machine as an example:

[email protected]:~# docker exec -it 2e76a9ca22c0 /bin/bash
[email protected]:/data/# cd /usr/local/bin
[email protected]:/usr/local/bin# ls
docker-entrypoint.sh  redis-check-aof  redis-sentinel
gosu                  redis-check-rdb  redis-server
redis-benchmark       redis-cli        redis.conf
# 为了好辨识各个机器,配置文件名为:redis+机器编号+端口号.conf
# 所以三个机器各名为:redis0079.conf、redis0189.conf、redis0299.conf
[email protected]:/usr/local/bin# mv redis.conf redis0079.conf

In order to edit files, view ip information and ping the test network in the container, you also need to install tools. All three machines execute this command

[email protected]:/usr/local/bin# apt update && apt install -y vim && apt install -y iproute2 && apt install -y iputils-ping

The configuration file redis0079.conf of the machine redis00 is modified as follows

port 6379   ==改为==>   port 6379
daemonize no   ==改为==>   daemonize yes
pidfile /var/run/redis_6379.pid   ==改为==>   pidfile /var/run/redis_6379.pid
logfile ""   ==改为==>   logfile "6379.log"
dbfilename dump.rdb   ==改为==>   dbfilename dump6379.rdb

The configuration file redis0189.conf of the machine redis01 is modified as follows

port 6389   ==改为==>   port 6389
daemonize no   ==改为==>   daemonize yes
pidfile /var/run/redis_6389.pid   ==改为==>   pidfile /var/run/redis_6389.pid
logfile ""   ==改为==>   logfile "6389.log"
dbfilename dump.rdb   ==改为==>   dbfilename dump6389.rdb

The configuration file redis0299.conf of the machine redis02 is modified as follows

port 6399   ==改为==>   port 6399
daemonize no   ==改为==>   daemonize yes
pidfile /var/run/redis_6399.pid   ==改为==>   pidfile /var/run/redis_6399.pid
logfile ""   ==改为==>   logfile "6399.log"
dbfilename dump.rdb   ==改为==>   dbfilename dump6399.rdb

4. Start redis

redis00

[email protected]:/usr/local/bin# redis-server /usr/local/bin/redis0079.conf 

redis01

[email protected]:/usr/local/bin# redis-server /usr/local/bin/redis0189.conf

redis02

[email protected]:/usr/local/bin# redis-server /usr/local/bin/redis0299.conf

View process information on the host

xxx

At this time, each redis service is the master node

5. Configure the slave node

The master node does not need to be configured, and the slave node configures the master node address

Get the ip address of the master node first

xxx

Then use the ** slaveofhost port ** command to set the address of the master node

xxx

The above configuration using commands is only temporary, and the modification of the configuration file is permanent

In the redis0189.conf file of redis01 and the redis0299.conf file of redis02

# replicaof <masterip> <masterport>改为replicaof 172.18.0.4 6379

Then restart the redis service of redis01 and redis02, you can

If you want to cancel the slave identity, just comment out the configuration file mode, and you can use the command in the command modeslaveof no one


Second, the sentinel mode

In master-slave replication, once the master node goes down, other nodes will not change, so the entire cluster cannot provide external services. The sentinel mode is to increase N sentinels, monitor the hosts of the entire cluster, and elect a master node. Once the master node goes down, another master node is newly elected.

The redis-sentinel tool in the redis installation directory acts as a sentinel. Generally, there are multiple sentinels, because if one sentinel is down, the cluster will also be down.

Environment setup (a sentinel)

Still use the environment of master-slave replication, one master node, two slave nodes, and then continue the following configuration

1. Modify the redis0079.conf, redis0189.conf, redis0299.conf files, and the modified content is the same

protected-mode yes# 改为,关闭安全模式,允许无密码访问protected-mode no
bind 127.0.0.1 -::1# 改为,监听所有地址bind * -::*

2. Add a configuration file on any node. sentinel.confAnd then configure

# 哨兵 监视 监视名 地址 端口 票数sentinel monitor sentinel01 172.18.0.4 6379 1

3. Run the sentry tool

[email protected]:/usr/local/bin# redis-sentinel sentinel.conf

4. Test

Shut down the master, a few seconds later, one of the slave nodes will be elected as the master node

xxx

Even if the host that just went down is restored, it can only be used as a slave node

3. Cache penetration, cache breakdown, cache avalanche

1. Cache penetration

concept

We generally use the cache because the database query is slow and consumes more resources, which means that the data is stored in the memory, but because there is too much data or other reasons, not all the data in the database is put in the memory, so we will Set up a strategy, if you can't find the data in the cache, look for it in the database. Now there is a hacker who forges a large number of requests, and the requested data is not in the cache, so the background programs will all look in the database, the cache loses its meaning, and the database will be under a lot of pressure.

The request passes through the cache and reaches the database. This is cache breakdown.

solution

There are many types, two of which are most commonly used:

  • Because of forged requests or due to code bugs, requests generally have the same key multiple times and the value is null, and the same key must be queried in the database every time, but the partial check is meaningless because it always returns null. So we can set the requested key to null and store it in the cache
  • Use Bloom filter (commonly used)

Principle of Bloom filter :

First, there is a bitmap, we can imagine it as an array, it has a length, a subscript, and the stored value can only be 0 or 1.

Then use different hash functions to calculate the key of our data, which can be 3 hash functions

Then calculate the remainder of the obtained hash value to the length of the bitmap, and the result of the remainder is the subscript of the bitmap

Then, after the remainder of the three results, the value of the corresponding bitmap subscript is set to 1, which means that the key exists

Then as the key increases, some positions in the bitmap may overlap, but it doesn’t matter

When the last request comes, calculate the hash value of the requested key, find the corresponding bit subscript, determine whether the key exists, and access the cache if it exists

If the bitmap subscript value corresponding to the result of a requested key calculation is 1, then the key may or may not exist, because there is a probability that different keys have the same hash value, and 3 hash values ​​are still calculated
If the bitmap subscript value corresponding to the result of a requested key calculation is 0, then the key must not exist

Therefore, the Bloom filter has a certain probability of misjudgment, but we can accept it.

2. Cache breakdown

concept

When we store data, we will take into account that some keys may not have a large amount of access, and they will waste space if they are kept, and all keys have an expiration time. Even if they expire, they can be stored in the cache after the request is processed again. in. If a key suddenly becomes hot data and tens of millions of requests are aimed at this key, and the key just reaches the expiration time, then tens of millions of requests will access the database at the same time because they cannot get the results in the cache. After the database crashes, the keys in the cache gradually become invalid over time, but the database cannot get the value, then the keys gradually become invalid, and finally the service ends.

At the moment when a key expires, a large number of requests hit the database, causing the database to go down, and then the same applies to each key, which is an avalanche.

solution

There is no good solution:

  • Lock the key resource in the program so that only one thread or process can access it at a time. Of course, queues can also be used.

3. Cache avalanche

concept

Basically similar to cache breakdown, but cache breakdown is for the instant when a key fails, and cache avalanche is for the moment when multiple keys fail at the same time

solution

  • Fuse mechanism