redis002,-transaction-publish and subscribe, master-slave replication

Article Directory

1. Affairs

The note comes from-the
redis single command of mad god video saves the atomicity, but the redis transaction does not guarantee the atomicity.

Transaction essence: a collection of a set of commands. Commands are placed in the queue, all commands of a transaction will be serialized, and will be executed in order during the execution of the transaction.
One-time, sequential, exclusive, execute a series of commands.

There is also no concept of isolation level.
All commands are not directly executed in the transaction, and are only executed when the execution command is initiated. Exec

Redis transaction:
open transaction: (multi)
command enqueue: ()
execute transaction: (exec)

Lock: redis can realize optimistic lock, watch

127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> multi # 开启事务
OK
127.0.0.1:6379> set k1 v1
QUEUED
127.0.0.1:6379> set k2 v2
QUEUED
127.0.0.1:6379> get k2
QUEUED
127.0.0.1:6379> set k3 v3
QUEUED
127.0.0.1:6379> exec # 执行事务
1) OK
2) OK
3) "v2"
4) OK

############################################################
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set k1 v1
QUEUED
127.0.0.1:6379> set k3 v3
QUEUED
127.0.0.1:6379> set k4 v4
QUEUED
127.0.0.1:6379> discard  # 放弃事务
OK
127.0.0.1:6379> get k4
(nil)


Compilation type exception (the code is wrong, the command is wrong!) All the commands in the food are not executed

127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set k1 v1
QUEUED
127.0.0.1:6379> set k2 v3
QUEUED
127.0.0.1:6379> getset k3 # 错误命令
(error) ERR wrong number of arguments for 'getset' command
127.0.0.1:6379> set k4 v5
QUEUED
127.0.0.1:6379> exec
(error) EXECABORT Transaction discarded because of previous errors.

Runtime exception (1/0) If the transaction queue has grammaticality, other commands can be executed when the command is executed, and the wrong command will throw an exception

An incorrect command in the transaction reports an error, and subsequent commands continue to execute, so redis transactions are not atomic

127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> set k1 "v1"
OK
127.0.0.1:6379> multi 
OK
127.0.0.1:6379> incr k1 # 不能自加1,而后面的命令继续执行
QUEUED
127.0.0.1:6379> set k2 v2
QUEUED
127.0.0.1:6379> set k3 v3
QUEUED
127.0.0.1:6379> exec
1) (error) ERR value is not an integer or out of range
2) OK
3) OK
127.0.0.1:6379> get k3
"v3"
127.0.0.1:6379> 

This is more like batch processing, not like transactions.

2. Optimistic lock, pessimistic lock

2.1, pessimistic lock

Very pessimistic, thinking that there will be problems anytime

2.2, optimistic lock

No problem at any time
1. Get version
2. Compare version when updating

watch is optimistic lock

127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> set money 100
OK
127.0.0.1:6379> set out 0
OK
127.0.0.1:6379> watch money
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> decrby money 20
QUEUED
127.0.0.1:6379> incrby out 20
QUEUED
127.0.0.1:6379> exec
1) (integer) 80
2) (integer) 20

Test multi-threading, modify the value, use watch as an optimistic lock operation of redis

127.0.0.1:6379> watch money # 监视money
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> decrby money 20
QUEUED
127.0.0.1:6379> incrby money 20
QUEUED
127.0.0.1:6379> exec # 执行之前,另外一个线程修改了我们的值,这个时候,就会导致事务执行失败。
(nil)


################################
然后怎么搞?
先解锁:unwatch
再加锁:watch money
执行失败也会自动解锁

如果修改失败,去再次获取最新的值就好,
自旋锁可以很好的完成这个。
redis秒杀系统就是乐观锁,watch实现

3.jedis

What is it?
Redis officially recommended java connection development tools, using java to operate redis middleware.
1. Import the corresponding dependency
2. Code and test a
new jedis object, and then call the method corresponding to yesterday's command,
setex sets the expiration time and
setnx will process if it does not exist, (distributed lock)

Affairs:

4. SpringBoot integration

5. Detailed explanation of Redis.conf

Own virtual machine: the
configuration file is in
sudo vi /etc/redis/redis.conf
1. The unit is not case sensitive

units are case insensitive so 1GB 1Gb 1gB are all the same

2. You can merge other configurations into the redis configuration

Insert picture description here


3. Network

bind 127.0.0.1 ::1  #绑定的ip

protected-mode yes  # 保护模式

# Accept connections on the specified port, default is 6379 (IANA #815344).
# If port 0 is specified Redis will not listen on a TCP socket.
port 6379   # 默认端口

4. General configuration

daemonize yes #以守护进程方式运行

pidfile /var/run/redis/redis-server.pid # 如果以后台方式运行,就需要指定一个pid进程文件
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)  # 生产环境
# warning (only very important / critical messages are logged)
loglevel notice  # 日志级别

logfile /var/log/redis/redis-server.log # 生成的日志文件名,如果为“”空,就直接输出
databases 16 # 数据库的数量
always-show-logo yes  # 是否总是显示logo


5. Snapshot SNAPSHOTTING is
persisted. In the specified time, how many operations are performed will be persisted to the file .rdb file. The .aof file
redis is an in-memory database. If it is not persisted, it will be lost

save 900 1 # 如果900秒内,至少有一个key进行了修改,我们就持久化操作
save 300 10 # 如果300秒内,至少有一个key进行了10次修改,我们就持久化操作
save 60 10000# 如果60秒内,至少有一个key进行了10000次修改,我们就持久化操作

之后我们会自己定义自己的持久化测试。



# 持久化错误之后还要不要继续工作
stop-writes-on-bgsave-error yes 

#是否压缩rdb文件,会消耗cpu资源
rdbcompression yes

# 保存rdb文件的时候,是否校验rdb文件的错误
rdbchecksum yes

# rdb保存的文件目录
dir /var/lib/redis

6, master-slave replication related REPLICATION

7. Security SECURITY

进入数据库之后:---> config get requirepass  # 获取密码

配置文件中设置密码:需要重启才能生效
# requirepass foobared
requirepass 123456

命令行设置密码:
config set requirepass "123456"

下次操作需要验证:
6379----> auth 123456


8. Client limit CLIENTS

# maxclients 10000

# maxmemory <bytes> # 最大内存设置


# maxmemory-policy noeviction # 内存达到上限之后的处理策略
1、volatile-lru:只对设置了过期时间的key进行lru(默认)
2、allkeys-lru:所有的key进行lru
3、volatile-random:随机删除即将过期的key
4、allkeys-random:随即删除
5、volatile-ttl:删除即将过期的
6、noeviction:永不过期,返回错误

9. Aof configuration APPEND ONLY MODE

appendonly no # 默认不开启aof模式,默认使用rdb持久化,在大部分所有的情况下rdb够用了

# 持久化文件的名字
appendfilename "appendonly.aof"


# appendfsync always # 每次写入都会同步,速度慢
appendfsync everysec # 每秒都执行一次同步,可能会丢失这一秒的数据
# appendfsync no # 不执行同步,操作系统自己执行同步,速度最快

6, redis persistence

Redis is an in-memory database. If you don't save the database state to disk, once the server process exits, the database state in the server will disappear, so it provides a persistence function.

6.1, RDB (redis DataBase)

Configure in 5. Snapshot SNAPSHOTTING, and some default persistence rules.

Insert picture description here

The snapshot of the data set in the memory is written to the disk within the specified time interval, that is, the Snapshot snapshot. When it is restored, the snapshot file is read directly into the memory.
Redis will separately create (fork) a child process for persistence, first write the data to a temporary file, and after the persistence process is over, replace the last persisted file with this temporary file. During the entire process, the main process does not perform any IO operations to ensure extremely high performance. If large-scale data recovery is required, and the integrity of data recovery is not very sensitive, RDB is more efficient than AOF.
RDB disadvantages: Finally A persistent data may be lost, the default RDB, generally do not modify this configuration.
The file saved by rdb is: dump.rdb file. They are all configured in a snapshot in the configuration file.

The save command can directly save the configuration file

RDB trigger mechanism
1. When the save rules are met, the RDB rules will be automatically triggered.
2. The flushing command will also be triggered.
3. Exit redis and generate RDB files.

How to restore the RDB file
1. Just put the RDB file in our redis startup directory. When redis starts, it will automatically check the dump.rdb file and restore the files in it.

config get dir


127.0.0.1:6379> config get dir
1) "dir"
2) "/etc/redis"  # 在这个文件夹下有dump.rdb文件,启动时候就会恢复了

In the production environment, this file will be backed up,

Advantages
1. Suitable for large-scale data recovery
2. Low requirements for data integrity
Disadvantages
1. Process operations at a certain time interval are required. If redis is unexpectedly down, the last modification cannot be recorded.
2. When the fork process is occupied, it will be occupied A certain content space,

6.2, AOF (Append Only File)

Record all our commands. History is similar. When restoring, all commands of this file will be executed again.
What is it?

Insert picture description here


Record each write operation in the form of a log, record all the instructions executed by redis, (read operations are not recorded), only append files, not modify, redis will read the file to rebuild the data at the beginning of the startup, that is : If redis restarts, the write instruction will be executed once according to the content of the log file to complete the data recovery.
AOF saves the appendonly.aof file

It is not turned on by default.
Time interval to
write Rewrite configuration not to rewrite, rewrite rule, time, size

If the aof file is damaged, redis will not start up. You can use the redis-check-aof program to check the file to repair this .aof configuration file.

redis-check-aof --fix appendonly.aof

Repair operation:

Insert picture description here


advantages:
1. Every modification is synchronized, and the integrity of the file is better.
2. The default synchronization is once every second, and one second of data may be lost.
3. Never synchronize, the highest efficiency,

Disadvantages:
1. Compared with data files, AOF is much larger than RDB, and the repair speed is slow.
2. Aof operation efficiency is also slower than RDB, (io operation). So the default is rdb

When the file reaches 64MB, a file is overwritten. Fork a new process to rewrite our files.
Unlimited appending of aof default files, the files will get bigger and bigger.

The interview asks you about the persistence mechanism?
Talk about the parameters,


Let's talk about the basic principles: aof is to read and write files, and
rdb is a fork child process that records files.

The default rdb.
Can withstand data loss within a few minutes, then you can just use RDB persistence.
In master-slave replication, rdb is the standby, on the slave. And AOF is almost not used.
Extension

Insert picture description here


2

Insert picture description here

7. Redis publish and subscribe

Insert picture description here


It is widely used to build instant messaging applications, online chat rooms, and real-time broadcasts, real-time reminders
1, real-time messaging system
2, real-time chat, the channel is used as a chat room, and the information is echoed back to everyone.
3. You can subscribe to the follow system.
4. Use Message Middleware (MQ) Kafka for slightly more complicated scenarios

Insert picture description here
Insert picture description here

Subscriber:

$ redis-cli
127.0.0.1:6379> subscribe kuangshenshuo # 订阅一个频道,
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "kuangshenshuo"
3) (integer) 1   # 等待读取信息

1) "message" # 消息
2) "kuangshenshuo"  # 频道
3) "hello world" # 内容
1) "message"
2) "kuangshenshuo"
3) " game over"

Release side:

~$ redis-cli
127.0.0.1:6379> publish kuangshenshuo "hello world"  # 发布消息
(integer) 1
127.0.0.1:6379> publish kuangshenshuo " game over"
(integer) 1
127.0.0.1:6379> 



8. Redis master-slave replication

Mad God video link
Copy the data of one redis server to other redis servers, master/leader, data replication is one-way. The master writes primarily, and the slave reads primarily.
Master-slave replication, separation of reads and writes, 80% of them are read. Reduce server pressure. The minimum is 1 master and 2 slaves, sentry.
By default, each redis is the master node, and a master node can have multiple slave nodes, and a slave node can only have one master node (a father can have multiple sons, and a son can only have one father)

Interview questions
The role of master-slave replication:
1. Data redundancy: Master-slave replication realizes hot backup of data, which is a data redundancy method besides persistence.
2. Failure recovery: When the master node has a problem. Services can be provided by slave nodes to achieve rapid stock recovery.
3. Load balancing: On the basis of master-slave replication, with read-write separation, master node writes, slave node reads, (when writing redis data, the application connects to the master node, Connect to the slave node when reading data) to share the server load. Especially when writing less and reading more, sharing the read load by multiple slave nodes can greatly increase the amount of concurrency.
4. The cornerstone of high availability (cluster): master-slave replication is still the basis for the implementation of sentinels and clusters, so master-slave replication is the basis for redis high availability.

Generally, one redis is not enough:
1. Structurally, a single redis service will have a single point of failure, and it will deal with all requests, which is under great pressure.
2. In terms of capacity, the capacity of a single redis server is limited, and the memory used by a single redis server should not exceed 20G
. The goods on the e-commerce website are uploaded once and viewed numerous times, "read more and write less"

8.2, environment configuration

Only configure the slave library,

$ redis-cli
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:0
master_replid:dc5d56bca2403bb682490ed37aeaf2a177573757
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

Copy 3 configuration files
1, modify the port, 6379
2, run in the background
3, pidfile /var/run/redis_6379.pid
4, logfile "6379.log"
5, dbfilename dump6379.rdb

Open:
redis-server kconfig/redis6379.conf
redis-server kconfig/redis6380.conf
redis-server kconfig/redis6381.conf

Insert picture description here


at least one master and two slaves,
only need to configure the slave.
Recognize the boss

slaveof 127.0.0.1 6379

# 查看: info replication

If there is a password, write the password of the masterauth master node in the slave node configuration file.

Insert picture description here


Here is the command configuration, temporary. If you need permanent configuration, configure it in the configuration file. For the replication, configure the ip and port and password.

Understand the details. The
master node is responsible for writing, and the slave node can only read. All information and data in the master will be automatically saved by the slave.

Insert picture description here


If the master node is broken, when there is no sentinel, the slave node is still a slave node, unless you actively modify the slave node to become the master. After the disconnected host connects back again, the slave node automatically connects to the master node.

If the slave is disconnected, if it is a master-slave relationship configured using commands, if the slave is restarted at this time, the slave will automatically become the master. Once you turn yourself into the slave of the master again, you can go to the master again to get the information.

Full copy, incremental copy: the slave connects to the master again.

Insert picture description here
Insert picture description here

Seeking to usurp the throne:
If the master is disconnected, the slave becomes the master by itself. Before the sentry came out, it was manual. If the original boss comes back at this time, he is just a polished commander at this time. Can only be reconfigured

slaveof no one

The sentinel mode is automatically called the host. It
can monitor whether the host is faulty in the background. If it fails, it will automatically switch from the library to the main library according to the number of votes . The
sentinel mode is a special mode, and the sentinel is an independent process. It runs independently. The principle is that the sentinel waits for the response of the redis server by sending commands to monitor multiple redis instances running.
The sentry sent a message, and if he didn't reply, he would be dead.

Insert picture description here


Sentinel clusters, the sentries will also monitor each other to prevent the sentries from dying.

Insert picture description here


Then it becomes: the
projection algorithm, to study the source code.

Insert picture description here

1. Under the current path of the configuration file: configure the sentinel's configuration file

vim sentinel.conf
在打开的文件中:
# sentinel monitor host port 1
sentinel monitor myredis 127.0.0.1 6379 1   # 使这个哨兵监控主机

The following 1 means that as long as a sentinel believes that the main service has been offline, it is judged to be objectively offline.

2. Start the sentry

redis-sentinel kconfig/sentinel.conf

可以通过 info replication 来显示投票换主机的日志信息

If the master node is disconnected, a server will be randomly selected from the slaves at this time. (Here is a voting algorithm)

If the host comes back later, he can only be the younger brother. This is the rule of sentinel mode

Sentinel advantages:
1. Sentinel cluster, based on the master-slave replication model, has all the advantages of the master-slave replication model.
2. The master-slave can be switched, and the failure can be transferred, and the availability of the system will be better.
3. The sentinel mode is an upgrade of the master-slave mode. It is more robust from manual to automatic.
Disadvantages:
1. Redis is not good for online expansion. Once the cluster capacity reaches the upper limit, online expansion is very troublesome.
2. Realizing the configuration of sentinel mode is actually very troublesome, there are many options inside.

All configurations of sentinel mode:
1. If there is a sentinel cluster, you need to configure the port of each sentinel.
2. Define the working directory of the sentinel.
3. Set the default host node, ip+port
4. Password for each host.
5. The default delay time. (The default is 30 seconds)
6. Failover time
7. If there is a problem, execute the script, notify the staff, and configure the operation and maintenance.

9. Redis cache penetration and avalanche

9.1. Cache penetration (not found)

The user wanted to query a piece of data, and found that there was no data in the redis memory database, that is, the cache did not hit, so he wanted to query the persistence layer database, but found no, so this query failed. When the user many times, the cache did not hit, so all went Requesting the persistence layer database will put a lot of pressure on the persistence layer database, which is equivalent to cache penetration.

Insert picture description here

9.1. Bloom filter (solution 1)

It is a data structure that stores all possible query parameters in a hash form, verifies them at the control layer first, and discards them if they do not match, thereby avoiding the query pressure on the underlying storage system.

Insert picture description here

9.2. Cache empty objects (solution 2)

When the storage layer misses, even the returned empty object is cached, and an expiration time is set at the same time, and then the data is accessed later, the empty object will be obtained from the cache, which protects the back-end data source.

Insert picture description here


There will be two problems:
1. If the empty value is cached, it means that the cache needs more space to store more keys, because there may be many keys with empty values,
2. Even if the empty value is set to expire Over time, there will still be inconsistencies between the cache layer and the storage layer data for a period of time, which has an impact on businesses that need to maintain consistency. (The cache layer does not exist, but the storage layer already exists)

9.3. Cache breakdown (too much)

Weibo server is down, (hot search)

Solution:
1. Set hotspot data to never expire: From the perspective of cache, no expiration time is set, so there will be no problems after hotspot key expires.

2. Add mutex locks (setnx) to
use distributed locks: ensure that there is only one thread for each key colleague to query the back-end service, and other threads do not have the authority to obtain distributed locks, so just wait. This method will be higher The pressure of concurrency has shifted to distributed locks,

Insert picture description here

9.2, cache avalanche

Refers to a certain period of time, the cache collection expires and becomes invalid. Redis is down and the power goes out.
One of the reasons for the avalanche: For example, when writing a text, at midnight on Double Eleven, a wave of rush purchases will soon be ushered in. This wave of merchandise time is relatively concentrated and put into the cache. Assuming that it is cached for an hour, it will be early in the morning. At 1 o'clock, the cache of this batch of commodities expired, and the access queries to this batch of commodities fell on the database. For the database, periodic pressure peaks would occur, and all requests reached the storage. The call of the storage layer has skyrocketed, causing the storage layer to also hang up.

Insert picture description here


Stop some services, (guarantee that the main service is available) not allow refunds,

Solution:
Redis availability: redis since it is possible to hang up, then set up more than a few redis, so hang a other, in fact, it is to build clusters (off-site live)
limiting relegation
after a cache miss, by adding Lock or queue to control the number of threads reading the database write cache. For example, only one thread is allowed to query data and write cache for a certain key, and other threads wait.
Data warm-up
Before the data is officially deployed, the possible data is accessed in advance, and a large number of accessed data will be loaded into the cache. Before a large concurrent access is about to happen, manually trigger the loading of different keys in the cache, and set different expiration times. Make the time points of cache invalidation as even as possible.

Mad god video