Redis-shake for Redis-Cluster data migration (cluster to cluster)

1. Background
-During the implementation of the project, due to the dev environment and pro environment, there will be two redis clusters at this time, but part of the data is obtained from the third party database of Party A and stored in the redis cluster of the production environment, for convenience For testing and data verification, it is necessary to synchronize the data of each node of the production redis cluster to the redis cluster of the test environment, and it is best to synchronize incremental data in real time.
2. Scheme

  • Write a redis project with multiple data sources, read from one redis cluster and write to another redis cluster. PASS (so stupid)
  • AOF, RDB directly COPY. PASS (What about incremental data? Have to stop?)
  • The redis-shake introduced in this article https://github.com/alibaba/RedisShake

3. Condition preparation

  • First we must have two redis clusters (source:master[192.168.18.129:7001;192.168.18.130:7003;192.168.18.130:7005] target:master[192.168.1.71:4001;192.168.1.71:4002;192.168.1.71 :4003]) Why only master? Will explain later~
  • Download the latest version of redis-shake according to the path above

4. Doubts

  • To migrate production data to the test environment, the data in my production redis cluster must not be lost. It's over if you lose it.
  • Will the data in my current redis cluster be overwritten?
  • Can the updated data in the production environment be incrementally synchronized to the test environment?

5. Hands-on

Download: https://github.com/alibaba/RedisShake/tree/release-v2.0.3-20200724, my current latest tags version is release-v2.0.3-20200724.

Upload to liunx environment, of course, you can also decompress it on windows.

The liunx environment uses tar -zxvf redis-shake-v2.0.3.tar.gz

Insert picture description here

Go to the redis-shake-v2.0.3 folder and find redis-shake.conf to modify the configuration file

# 阿里的这个配置文件的注释特别详细,我只摘出我们需要修改的项目来进行修改。

# source redis configuration.
# used in `dump`, `sync` and `rump`.
# source redis type, e.g. "standalone" (default), "sentinel" or "cluster".
#   1. "standalone": standalone db mode.
#   2. "sentinel": the redis address is read from sentinel.
#   3. "cluster": the source redis has several db.
#   4. "proxy": the proxy address, currently, only used in "rump" mode.
# 源端redis的类型,支持standalone,sentinel,cluster和proxy四种模式,注意:目前proxy只用于rump模式。
source.type = cluster
# ip:port
# the source address can be the following:
#   1. single db address. for "standalone" type.
#   2. ${sentinel_master_name}:${master or slave}@sentinel single/cluster address, e.g., mymaster:[email protected]:26379;127.0.0.1:26380, or @127.0.0.1:26379;127.0.0.1:26380. for "sentinel" type.
#   3. cluster that has several db nodes split by semicolon(;). for "cluster" type. e.g., 10.1.1.1:20331;10.1.1.2:20441.
#   4. proxy address(used in "rump" mode only). for "proxy" type.
# 源redis地址。对于sentinel或者开源cluster模式,输入格式为"master名字:拉取角色为master或者[email protected]的地址",别的cluster
# 架构,比如codis, twemproxy, aliyun proxy等需要配置所有master或者slave的db地址。
source.address = 192.168.18.129:7001;192.168.18.130:7003;192.168.18.130:7005
# password of db/proxy. even if type is sentinel.
source.password_raw = 123123yu
# auth type, don't modify it
source.auth_type = auth


# target redis configuration. used in `restore`, `sync` and `rump`.
# the type of target redis can be "standalone", "proxy" or "cluster".
#   1. "standalone": standalone db mode.
#   2. "sentinel": the redis address is read from sentinel.
#   3. "cluster": open source cluster (not supported currently).
#   4. "proxy": proxy layer ahead redis. Data will be inserted in a round-robin way if more than 1 proxy given.
# 目的redis的类型,支持standalone,sentinel,cluster和proxy四种模式。
target.type = cluster
# ip:port
# the target address can be the following:
#   1. single db address. for "standalone" type.
#   2. ${sentinel_master_name}:${master or slave}@sentinel single/cluster address, e.g., mymaster:[email protected]:26379;127.0.0.1:26380, or @127.0.0.1:26379;127.0.0.1:26380. for "sentinel" type.
#   3. cluster that has several db nodes split by semicolon(;). for "cluster" type.
#   4. proxy address. for "proxy" type.
target.address = 192.168.1.71:4001;192.168.1.71:4002;192.168.1.71:4003
# password of db/proxy. even if type is sentinel.
target.password_raw = 123123yu

target.auth_type = auth

The above part is the most confusing place for users. In order to meet the flexible configuration of users, in addition to the migration of the cluster version, redis-shake also has single-node/master-slave migration and RDB migration. There are currently many configuration items open, but if I mention that we are focusing on the cluster version, so the configuration of the cluster version is as shown above, and the content is explained as follows:

source.type: The type of source redis, supports the following 4 types
standalone:: Single db node/master-slave version mode. If the source is pulled from multiple db nodes, choose this mode, even if it is an open source proxy-db architecture such as codis.
sentinel: sentinelMode.

cluster: Cluster mode. Open source cluster. For Alibaba Cloud, users cannot pull the address of db at present, so here can only be proxy.
proxy: Proxy mode. If it is the cluster version of Alibaba Cloud redis, please select proxy for pull/write from proxy, and cluster for pull from db. For normal cluster to cluster synchronization source end, please select cluster mode. Proxy mode is currently only used for Rump.

source.address: The address of the source redis. Starting from version 1.6, we support the cluster version. Different types correspond to different addresses: in the
standalonemode, you need to fill in the address of a single db node, and the master or slave version needs to enter the address of the master or slave.
sentinelIn the mode, it needs to be filled in sentinel_master_name:[email protected]_cluster_address. sentinel_master_nameRepresents the name of the master under the sentinel configuration, master_or_slaverepresents whether the db selected from the sentinel is master or slave, sentinel_cluster_addressrepresents the single node or cluster address of the sentinel, and the cluster address is separated by a semicolon (;). For mymaster:[email protected]:26379;127.0.0.1:26380example: . Note that if it is in sentinel mode, only one master or slave information can be pulled at present. If you need to pull multiple nodes, you need to start multiple shakes.
clusterIn the mode, you need to fill in the cluster address, separated by a semicolon (;). For example: 10.1.1.1:20331;10.1.1.2:20441. When using sync, only write the IP and port of the master, refer to the configuration file I am facing, otherwise an error will be reported when redis-shake is started, and you can modify it.
proxyIn the mode, you need to fill in the address of a single proxy. This mode is currently only used rump.

source.password_raw: The password of the source redis.

target.type: The type of the purpose redis, source.typeconsistent with it. Note that if the destination is the cluster version of Alibaba Cloud, please fill in proxy for the type, and fill in cluster only to synchronize db0.

target.address: The address of the destination redis. Starting from version 1.6, we support the cluster version, and different types correspond to different addresses. standaloneMode, see source.address.
sentinelMode, you need to fill in [email protected]_cluster_address. sentinel_master_nameRepresents the name of the master under the sentinel configuration, and sentinel_cluster_addressrepresents the single node or cluster address of the sentinel, where the cluster address is separated by a semicolon (;). For example: [email protected]:26379;127.0.0.1:26380
clustermode, see source.address.
proxyIn the mode, fill in the address of the proxy. If it is more than one proxy, the round-robin load balances the connection to ensure that one source db connection will only correspond to one proxy. If it is the cluster version of Alibaba Cloud, please choose this mode.

target.password_raw: The password of the destination redis.

After configuring the user profile, and then you can start in different modes: ./redis-shake -conf=redis-shake.conf -type=sync.

6. Verification

  • The source data has not been cleared, and the target data has not been overwritten
  • Set qhyu 123456789 in the source cluster, there is a real-time incremental update in the target
Insert picture description here
Insert picture description here