In-depth understanding of Redis (3) Redis advanced features

1. Redis transaction

Redis transactions: one-time, sequential, exclusive

  • Redis transaction: A set of commands. All commands in a transaction will be serialized and executed in order during execution.
  • Redis single command guarantees atomicity, but transaction does not guarantee atomicity.
  • There is no isolation level for Redis transactions, and the execution command will be executed only when it is initiated. exec
Redis executes transactions:
  • Open transaction (multi)
  • Order to enqueue()
  • Execution transaction (exec)
# 执行事务
127.0.0.1:6379> multi	# 开启事务
OK
127.0.0.1:6379> set k1 v1	# 命令入队
QUEUED
127.0.0.1:6379> set k2 v1
QUEUED
127.0.0.1:6379> get k2
QUEUED
127.0.0.1:6379> set k3 v3
QUEUED
127.0.0.1:6379> exec	# 执行事务
1) OK
2) OK
3) "v1"
4) OK
###########################################
# 放弃事务执行
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set k1 v1
QUEUED
127.0.0.1:6379> set k2 v1
QUEUED
127.0.0.1:6379> set k4 v4
QUEUED
127.0.0.1:6379> DISCARD	# 取消事务
OK
127.0.0.1:6379> get k4
(nil)
###########################################
# 编译异常:事务中所有命令都不会执行
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set k1 v1
QUEUED
127.0.0.1:6379> set k2 v2
QUEUED
127.0.0.1:6379> set k3 v3
QUEUED
127.0.0.1:6379> getset k3
(error) ERR wrong number of arguments for 'getset' command
127.0.0.1:6379> set k4 v4
QUEUED
127.0.0.1:6379> set k6 v6
QUEUED
127.0.0.1:6379> exec	# 执行事务报错
(error) EXECABORT Transaction discarded because of previous errors.
127.0.0.1:6379> get k5	# 所有命令都不会执行
(nil)
127.0.0.1:6379> get k4
(nil)
# 运行时异常,队列中存在语法性,执行命令时,其他命令可以正常执行。错误命令抛出异常
127.0.0.1:6379> multi
OK
127.0.0.1:6379> incr k1
QUEUED
127.0.0.1:6379> DISCARD
OK
127.0.0.1:6379> flushdb
OK
127.0.0.1:6379> set k1 v1
OK
127.0.0.1:6379> mulit
(error) ERR unknown command 'mulit'
127.0.0.1:6379> multi
OK
127.0.0.1:6379> incr k1
QUEUED
127.0.0.1:6379> set k2 v2
QUEUED
127.0.0.1:6379> set k3 v3
QUEUED
127.0.0.1:6379> set k4 v4
QUEUED
127.0.0.1:6379> get k3
QUEUED
127.0.0.1:6379> exec
1) (error) ERR value is not an integer or out of range	# 执行事务报错但依旧执行
2) OK
3) OK
4) OK
5) "v3"

2. Redis implements optimistic locking

Monitoring: watch

Pessimistic lock:

No matter what you do, it will be locked

Optimistic lock

It will not be locked. When updating the data, judge whether anyone has modified the data during this period. Compare version when
getting version
update

Redis monitoring test

Successful execution:

127.0.0.1:6379> set money 1000
OK
127.0.0.1:6379> set out 0
OK
127.0.0.1:6379> watch money	# 监视 money 对象
OK
127.0.0.1:6379> multi	# 事务正常执行成功
OK
127.0.0.1:6379> DECRBY money 200
QUEUED
127.0.0.1:6379> INCRBY out 200
QUEUED
127.0.0.1:6379> exec
1) (integer) 800
2) (integer) 200

Multi-threaded modification value: use watch to do Redis optimistic lock operation

127.0.0.1:6379> watch money	# 监视 money
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> DECRBY money 100
QUEUED
127.0.0.1:6379> INCRBY out 100
QUEUED
127.0.0.1:6379> exec	# 执行之前,另外一个线程修改了 money 的值
(nil)
#####################################
127.0.0.1:6379> get money
"800"
127.0.0.1:6379> set money 1000
OK

After the transaction fails

127.0.0.1:6379> unwatch	# 先解锁
OK
127.0.0.1:6379> watch money	# 再次监视
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> DECRBY money 100
QUEUED
127.0.0.1:6379> INCRBY out 100
QUEUED
127.0.0.1:6379> exec	# 比对监视的值是否发生变化,未变化可以执行成功
1) (integer) 900
2) (integer) 300

3. Redis.conf configuration file

Redis is started through the configuration file

  • Redis has 16 databases by default
Insert picture description here


Redis 6 was single-threaded before. Redis is based on memory operations, and the bottleneck of Redis is based on the memory and network bandwidth of the original machine.

Redis is written in C language, and the official data is 100,000 + QPS, which is not worse than the key-value Menmecache!

3.1. Unit:

Configuration files are not sensitive to case units

# Redis configuration file example

# Note on units: when memory size is needed, it is possible to specify
# it in the usual form of 1k 5GB 4M and so forth:
#
# 1k => 1000 bytes
# 1kb => 1024 bytes
# 1m => 1000000 bytes
# 1mb => 1024*1024 bytes
# 1g => 1000000000 bytes
# 1gb => 1024*1024*1024 bytes
#
# units are case insensitive so 1GB 1Gb 1gB are all the same.

3.2. Multiple configuration files can be included

################################## INCLUDES ###################################

# Include one or more other config files here.  This is useful if you
# have a standard template that goes to all Redis servers but also need
# to customize a few per-server settings.  Include files can include
# other files, so use this wisely.
#
# Notice option "include" won't be rewritten by command "CONFIG REWRITE"
# from admin or Redis Sentinel. Since Redis always uses the last processed
# line as value of a configuration directive, you'd better put includes
# at the beginning of this file to avoid overwriting config change at runtime.
#
# If instead you are interested in using includes to override configuration
# options, it is better to use include as the last line.
#
# include .\path\to\local.conf
# include c:\path\to\other.conf

3.3, network

bind 127.0.0.1	# 绑定的 IP
protected-mode # 保护模式
port # 端口

3.4, general arrangement

daemonize yes # 以守护方式运行,默认为 no

pidfile /var/run/redis_6379.pid # 如果以后台方式运行,需要制定一个 pid文件

# 日志
# Specify the server verbosity level.
# This can be one of:
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (moderately verbose, what you want in production probably)
# warning (only very important / critical messages are logged)
loglevel notice
logfile "" # 文件名
database 16 # 数据库数量,默认16个数据库
always-show-logo yes # 是否显示 logo

3.5, snapshot

Persistence, how many operations performed within the specified time will be persisted to the file. RDB. Aof
Redis is stored in memory and needs to be persisted

# 900 秒内,至少有一个 key 进行了修改,就进行持久化操作
save 900 1
# 300 秒内,至少10 个 key 进行修改,进行持久化操作
save 300 10
# 60 秒内,至少 10000 个 key 进行修改,进行持久阿虎操作。
save 60 10000

stop-writes-on-bgsave-error yes # 默认持久化操作出错继续进行工作

rdbcompression yes # 是否压缩 rdb 文件

rdbchecksumyes # 保存文件进行校验

dir ./ # rdb 文件保存目录

3.6, REPLICATION replication

3.7, SECURITY security

Redis password can be set, no password by default

requirepass root
127.0.0.1:6379> config set requirepass "root" # 设置 Redis 密码
OK
127.0.0.1:6379> config get requirepass # 获取 Redis 密码
1) "requirepass"
2) "root"

3.8, CLIENTS client

maxclients 10000 # 设置 Redis 最大客户端连接数

maxmemory <bytes> # Redis 配置最大内存容量

maxmemory-policy noeviction # 内存达到上限的拒绝策略
	# volatile-lru 删除 lru 算法的 key
	# allkeys-lru 删除 lru 算法的 key
	# volatile-random 随机删除即将过期的 key
	# volatile-ttl 删除即将过期的
	# noeviction 永不过期,返回错误

3.9, APPEND ONLY mode AOF

appendonly no # 默认不开启 AOF,默认使用 RDB 持久化,大部分情况下,RDB足够使用。
appendfilename "appendonly.aof" # 持久化文件的名字
appendsync always # 每次修改都会 sync
appendsync everysec # 每秒执行一次,可能会丢失 1 秒数据
appendsync no # 不执行 sync

4. Redis persistence

Redis persistence method

4.1 RDB (Redis Database)

The file saved by RDB is dump.rdb, which is configured in the snapshot in the configuration file. dbfilename dump.rdb
save 60 5
The save rule is satisfied, and the rdb rule is triggered.
Execute the fluxhall command, and the rdb rule will also be triggered.
Exit redis, and the rdb rule will also be generated

Restore rdb file

Put the rdb file in the redis startup directory, it will automatically check the dump.rdb file at startup and restore the data in it

advantage:

A large amount of data recovery
does not have high data integrity

Disadvantage

When the
fork process needs to be operated at intervals , it will take up a certain amount of memory space.

4.2 AOF

Each write operation is recorded in the form of a log. If redis is restarted, the write command will be executed from front to back according to the content of the log file to complete data recovery.

AOF save appendonly.aof file
appendonly yesis not opened by default, manually open it and
restart redis to take effect, if there is an error in the aof file, redis cannot be started, and the aof file needs to be repaired

Tools provided by redis: redis-ckeck-aof --fix
If the file is normal, it can be restored directly

Rewrite rules:

aof defaults to unlimited appending of files, and the files are getting bigger and bigger

auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
If the aof file is larger than 64mb, fork a new process to rewrite the file

advantage:

Every modification is synchronized, file integrity is better

Disadvantages:

Compared with data files, AOF is much larger than RDB files, and the repair speed is slower.
AOF runs slower than RDB.
  1. The RDB persistence method can perform snapshot storage of data at specified time intervals
  2. The AOF persistence method records each write operation to the server. When the server restarts, the command is executed to restore the original data. The AOF command saves each write operation to the end of the file with the Redis protocol. Redis can rewrite the AOF file that reaches the file size. , So that the AOF file will not be too large
  3. If the data only exists while the server is running, persistence can be omitted
  4. Two persistence methods are enabled at the same time. Redis first loads the AOF file to restore the original data, and the data saved in the AOF file is more complete than the RDB data. RDB files are not real-time, and AOF files will be found when used at the same time. RDB is suitable for backing up the database, restarting quickly, and there will be no possible AOF bugs.

5. Redis publish and subscribe

Redis publish and subscribe (pub/sub) is a message communication mode. The sender pub sends messages, and the subscriber sub receives messages. Redis clients can subscribe to channels with any amount of data.

  1. News publisher
  2. Channel
  3. News subscriber

principle

Redis uses C implementation, pubsub.c file, the underlying implementation of
publish and subscribe. Redis implements publish and subscribe functions through the publish, subscribe, and psunscribe commands.
After subscribing to the channel through the subscribe command, a dictionary is maintained in redis-server, and the key of the dictionary corresponds to one A channel, the value of the dictionary is a linked list, which stores all the clients that have subscribed to the channel. The subscribe command is to add the client to the subscription linked list of the channel, and send a message to the subscriber through the publish command. The redis-server uses the specified channel as the key , Look up the linked list of all clients subscribed to this channel in the maintained channel dictionary, traverse the linked list and send the message to the subscriber.
publish and subscribe in redis can be set to publish and subscribe to a key value. When a key value is published, all subscribed clients will receive the corresponding message. Can be used as a chat function.

Sender

127.0.0.1:6379> PUBLISH channel1 "hello,redis" # 发布消息到指定频道 channel1
(integer) 1
127.0.0.1:6379> PUBLISH channel1 "hello,world"
(integer) 1

Subscriber

127.0.0.1:6379> SUBSCRIBE channel1 # 订阅频道 channel1
Reading messages... (press Ctrl-C to quit)
1) "subscribe"
2) "channel1"
3) (integer) 1
1) "message" # 收到订阅频道消息
2) "channel1"
3) "hello,redis"
1) "message"
2) "channel1"
3) "hello,world"

5. Redis master-slave replication

Master-slave replication refers to copying the data of one Redis server (master/leader) to other Redis servers (slave/follower). Data replication is one-way, and can only be from the master node to the slave node. The master is mainly writing and the slave is mainly reading.

5.1 The role of master-slave replication

  1. Data redundancy: data backup, a data redundancy method other than persistence
  2. Failure recovery: If the master node has a problem, the slave node can provide services to achieve failure recovery and service redundancy.
  3. Load balancing: On the basis of master-slave replication, with the separation of reads and writes, the master node can provide write services, slave nodes provide read services, read and write connections to different nodes, share the server load, in the case of less writes and more reads, through multiple slaves Nodes share the load and increase concurrency
  4. Master-slave replication is the cornerstone of the implementation of sentinels and clusters, and the foundation of redis high availability.

5.2 Environment configuration

Only configure the slave library, not the main library

View master and slave information

127.0.0.1:6379> info replication # 查看当前库信息
# Replication
role:master # 角色 master
connected_slaves:0 # 从机
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

Copy 3 configuration files and modify corresponding information

  1. The port number
  2. pid name
  3. log file name
  4. dump.rdb name

After modification, start 3 redis services. 默认情况下,每个 Redis 都是主节点, Generally only need to configure the slave

Configure in the configuration file
Use the command line to configure, it will automatically become the master after restart, as long as it becomes the slave, the value will be obtained from the master immediately.

# 暂时性配置
127.0.0.1:6380> SLAVEOF 127.0.0.1 6379 # 配置主机信息
OK

The host writes and the slave reads. All information and data in the master will be automatically saved by the slave

5.3 Copy principle

After the slave starts successfully and connects to the master, it will send a sync
command. After receiving the command, the master will start the background save process and collect all the commands used to modify the data. After the background process is executed, the master will transfer the entire data file to the slave to complete once. Synchronize.
Full copy : After the slave service receives the database file data, it saves it and loads it into the memory.
Incremental replication : The master continues to pass all collected modification commands to the slave in turn to complete the synchronization.
Reconnect to the master and automatically perform an incremental replication and complete synchronization.

5.4 Manually configure the host after downtime

Host is disconnected, using SLAVEOF no oneso changed from the host machine, after reconnecting the original host needs to be reconfigured

6. Sentinel mode sentinel

The master-slave switch is changed from manual to automatic switch. After monitoring the host failure, the slave library is automatically changed to the master library according to the number of votes.
Redis provides the sentinel command. The sentinel is a separate process that runs independently. The sentinel sends commands and waits for the response from the Redis server to monitor multiple Redis instances running.

The main service is down, and the system will not immediately perform the fallover process. Sentinel 1 thinks that the main server is unavailable, which is called subjective offline.
Multiple sentries detect that the main server is unavailable. After voting, perform the fallover operation. After the switch is successful, pass pub/ sub, switch from server to host, called objective offline

6.1 Configure Sentry Profile

Default port: 26379

1. Sentinel mode configuration file: sentinel.conf

sentinel monitor 监控的名称 host port num(下线票数的临界点)

2. Start the sentry

redis-sentinel konfig/哨兵配置文件

If the master node is disconnected, select the server from the slave as the master. After the failover (failover), the
master is reconnected and merged into the new master as the slave.
Advantages

  1. Sentinel cluster, based on master-slave replication mode, all master-slave configuration advantages have
  2. Master and slave can be switched, failure can be transferred, system availability is better
  3. The sentinel mode is an upgrade of the master-slave mode, from manual to automatic, which is more robust.

Disadvantage

  1. The cluster reaches the upper limit, it is troublesome to expand
  2. Trouble with sentinel mode configuration

7. Redis cache penetration and avalanche

Cache penetration (not found)

The user queries a piece of data, the data is not queried in Redis, and the cache misses, so the query to the persistence layer database, and the data is not queried, the query fails. When the amount of concurrency is large, it will put a lot of pressure on the database and cause cache penetration

Bloom filter

A data structure that stores the parameters that may be queried in the form of hash, and verifies them at the control layer first, and discards them if they do not meet the requirements, avoiding query pressure on the underlying storage system
Cache empty objects. If the database is not queried, return empty objects and cache them, set the expiration time, and then access the data to read from the cache, which
may cause problems. The database and the cache are inconsistent!

Cache breakdown (large amount)

A key is used as hot data, and a key is accessed with high concurrency. When the key fails, a large number of requests are made to the database.

Hot data never expires
plus mutex lock

Distributed locks ensure that each key has only one thread to query the back-end service at the same time, and other threads do not have the authority to acquire distributed locks.

Cache avalanche

In a certain period of time, a large number of caches have expired and become invalid

Redis cluster, high availability,
current limit degradation,
data warm-up,