Redis topic (5): Redis persistence (RDB and AOF)

This series will continue to output Redis-related knowledge and sort out, and while improving myself, I hope to help friends in need, continuous output...continuous output...continuous output...

Thematic directory: directory transmission

Preface

In the previous article, we briefly introduced the use of Redis data structure and simple application scenarios, hoping to help some people in need.

Introduction

Before introducing persistence, let's talk about why Redis is so high-performance? Why do you need a persistent operation?

Because all its data is in memory, all operations are memory-level operations. Everyone knows that the data stored in memory is extremely insecure and receives too much interference from external factors, so we need persistence. operating.

Let's briefly introduce the persistence operation:

There are two ways to persist redis: RDB snapshot (snapshot) and AOF (append-only file)

RDB snapshot (snapshot)

By default, Redis saves in-memory database snapshots in a binary file named dump.rdb.
You can set Redis to automatically save the
data set once the condition of "at least M changes in the data set within N seconds" is met .
For example, the following settings will make Redis automatically save once when the condition of "at least 1000 keys have been changed within 60 seconds" is met.
For example: to save 60 1000
turn off RDB, you only need to comment out all save save policies or you
Insert picture description here

can manually Execute commands to generate RDB snapshots. Enter the redis client and execute commands save or bgsave to generate dump.rdb files.
Each command execution will snapshot all redis memory to a new RDB file and overwrite the original RDB snapshot file.
save is a synchronous command, bgsave is an asynchronous command, bgsave will fork from the main redis process (fork() is a linux function) to create a child process
specifically used to generate RDB snapshot files

Let's take a look at the rbd snapshot file.

Insert picture description here


He will compress and store our data.

Comparison of save and bgsave:

commandsavebgsave
IO typeSynchronizeasynchronous
Whether to block other redis commandsYesNo, child process will be spawned
the complexityO(n)O(n)
advantageDoes not consume additional memoryDo not block client commands
DisadvantageBlock client commandsNeed to fork child process, consume memory

AOF (append-only file)

The snapshot function is not very durable: if Redis fails due to some reasons, the server will lose the data that was recently written to Redis but not saved in the snapshot.
For example save 60 1000, it happened at the 70th second after taking a snapshot. Failure, then we will lose the middle 10S of data. Here, Redis has added a completely durable persistence method: AOF persistence, which records every modified instruction to the file appendonly.aof.
  • Open AOF: comment # appendonly yes in the configuration file
From now on, whenever Redis executes a command to change the data set, this command will be appended to the end of the AOF file. In this way, when Redis fails and restarts, the program can rebuild data by executing the commands in the AOF file.
You can configure how often Redis will fsync data to the disk:
-appendfsync always: fsync is executed every time a new command is appended to the AOF file, which is very slow and very safe.
-appendfsync everysec: fsync once per second, fast enough (similar to using RDB persistence), and only 1 second of data will be lost in the event of a failure.
-appendfsync no: Never fsync, and hand over the data to the operating system for processing. A faster and less secure option.
-------------------------------------------------- --------------------------------------- The
recommended (and default) measure is every second Fsync once , this fsync strategy can balance speed and safety.

Let's do it. First, open aof, release the comment described above and restart it.

Connect the client to set a record: set test1 test1
view the AOF file:vi appendonly.aof

Insert picture description here
Someone may use $5 to mean, these things are used by Redis to calculate storage space. The actual length of test1 we have just now is 5, so it will also be recorded here.

AOF rewrite

There may be too many unused specifications in the AOF file, so AOF will generate the AOF file based on the latest data in the memory. For example, the operations such as get we performed will also be recorded in it. In addition, we can also specify the frequency of AOF automatic rewriting. The
following two configurations can control the frequency of AOF automatic rewriting
1). auto-aof-rewrite-min-size 64mb //aof file must be at least 64M before it will be automatically rewritten. Too small, the recovery speed is already very fast, the meaning of rewriting is not great
2).auto-aof-rewrite-percentage 100 //The file size of the aof file has increased by 100% since the last rewrite, and the rewrite
will be triggered again. Perform received rewrite: execute command: bgrewriteaof
Here, AOF rewrite Redis will also fork a child process to perform it, which does not have much impact on the normal Redis command.
  • RDB and AOF, contrast
commandRDBAOF
Start prioritylowhigh
volumesmallBig
Recovery speedfastslow
Data securityEasy to lose dataDecide according to strategy

It should be noted that when redis is started, if there are both rdb files and aof files, the aof file is preferred to restore data, because aof generally has more complete data.

Hybrid persistence

In Redis 4.0, hybrid persistence was introduced. When restarting Redis, we rarely use RDB to restore the memory state, because a large amount of data will be lost. We usually use AOF log replay, but the performance of replaying AOF log is relative to RDB. It is much slower, so in the case of a large Redis instance, it takes a long time to start. Redis4.0 therefore introduced hybrid persistence.
  • Open the configuration: # aof-use-rdb-preamble yes
If hybrid persistence is enabled, when AOF is rewritten, it will not simply convert the memory time into a command format and write it into the AOF file, but will process the RDB snapshot of the memory before the rewrite, and add the contents of the RDB snapshot. A large number of AOF commands to modify memory data are stored in a mixed manner, and they are all written into the new AOF file and the new AOF file to complete the replacement of the old and new AOF files.
Therefore, when Redis restarts, you can record the RDB content first, and then replay the incremental AOF log to completely replace the previous AOF full file replay, and the efficiency is greatly improved.
  • Mixed storage data format
Insert picture description here


Examples:

1. Open the configuration file according to the above, release the comment and change it to yes
2. Connect to the client to execute the command bgrewriteaof
3. Insert a piece of test data in progress.

After completing the above, we are looking at the data of the aof file:

Insert picture description here
This is the end of this article. If there is something wrong, I also ask you to correct me, thank you again!