docker canal-server canal-adapter mysql

This article will introduce the use of docker to install and configure canal-server, canal-adapter, and synchronize MySQL data to MySQL.
The canal-mode in this tutorial is tcp, if it is replaced with other such as kafka, rocketMQ, it will be similar.

1.canal introduction

The working principle of canal is to pretend to be a MySQL slave and simulate the interaction protocol of MySQL slave to send a dump protocol to MySQL Mater. After receiving the dump request from canal, MySQL mater starts to push the binary log to canal, and then canal parses the binary log, and then Send to storage destination, such as MySQL, Kafka, Elastic Search, etc.

2. Preliminary preparation

1. MySQL installation

2. Add users to canal-server

After installing MySQL, add a user for canal-server use.

-- 创建用户 用户名:canal 密码:[email protected]
create user 'canal'@'%' identified by '[email protected]';
-- 授权 *.*表示所有库
grant SELECT, REPLICATION SLAVE, REPLICATION CLIENT on *.* to 'canal'@'%' identified by '[email protected]';

3. Open MySQL binlog

Next, set the following information in the MySQL configuration file my.cnf:

[mysqld]
# 打开binlog
log-bin=mysql-bin
# 选择ROW(行)模式
binlog-format=ROW
# 配置MySQL replaction需要定义,不要和canal的slaveId重复
server_id=1

After changing the configuration file, restart MySQL and use the command to check whether the binlog mode is turned on:

Insert picture description here

4. Create two database tables for this data synchronization test

Descriptiondatabasetable
Source databasemytest1user
Target databasemytest2user

The user table structure is: id, name

At this point, the MySQL server is ready to be completed.

3.Docker install canal-server

Go to the docker hub to find the canal-server mirror, I use canal/canal-server:latest here

#拉取镜像
docker pull canal/canal-server:latest
#第一次运行
docker run --name canal-server -p 11111:11111 -d canal/canal-server:latest

The purpose of this run is to find the directory of the configuration file instance.properties, and then mount the configuration file for later modification outside the container.

[email protected]:/opt/canal-adapter/conf# docker exec -it canal-server bash
[[email protected] admin]# ls
app.sh  bin  canal-server  health.sh  node_exporter  node_exporter-0.18.1.linux-arm64
[[email protected] admin]# cd canal-server/
[[email protected] canal-server]# ls
bin  conf  lib  logs  plugin
[[email protected] canal-server]# cd conf/
[[email protected] conf]# ls
canal_local.properties  canal.properties  example  logback.xml  metrics  spring
[[email protected] conf]# cd example/
[[email protected] example]# ls
h2.mv.db  instance.properties  meta.dat
[[email protected] example]# pwd
/home/admin/canal-server/conf/example

Create a folder, my path here is /opt/canal-server

#创建文件夹
mkdir /opt/canal-server
#进入目录
cd /opt/canal-server
#将instance.properties从容器中拷贝出来
docker cp canal-server:/home/admin/canal-server/conf/example/instance.properties ./

Modify the copied instance.properties to

Insert picture description here


stop the running canal-server, delete the container, and re-run canal-server

#停止容器
docker stop canal-server
#删除容器
docker rm canal-server
#重新运行canal-server,使用挂载出来的配置文件
docker run --name canal-server -p 11111:11111 -v /opt/canal-server:/home/admin/canal-server/conf/example -d canal/canal-server:latest

Check whether canal-server is running successfully.

Insert picture description here


At this point, docker installation canal-server is complete

4.Docker install canal-adapter

Also go to the docker hub to find the canal-adapter mirror, I use slpcat/canal-adapter:v1.1.5 here

#拉取镜像
docker pull slpcat/canal-adapter:v1.1.5
#第一次运行
docker run --name canal-adapter -p 8081:8081 -d slpcat/canal-adapter:v1.1.5

The purpose of this run is to find the directory of the configuration files application.yml and mytest_user.yml, and then mount the configuration files for later modification outside the container.

[email protected]:/opt/canal-server# docker exec -it canal-adapter bash
[email protected]:/opt/canal-adapter# ls
bin  conf  lib	logs  plugin
[email protected]:/opt/canal-adapter# cd conf/
[email protected]:/opt/canal-adapter/conf# ls
application.yml  rdb
[email protected]:/opt/canal-adapter/conf# pwd
/opt/canal-adapter/conf
[email protected]:/opt/canal-adapter/conf# cd rdb/
[email protected]:/opt/canal-adapter/conf/rdb# ls
mytest_user.yml
[email protected]:/opt/canal-adapter/conf/rdb# pwd
/opt/canal-adapter/conf/rdb
[email protected]:/opt/canal-adapter/conf/rdb# 

Create a folder, my path here is /opt/canal-adapter/conf, /opt/canal-adapter/conf/rdb

#创建conf文件夹
mkdir /opt/canal-adapter/conf
#创建rdb文件夹
mkdir /opt/canal-adapter/conf/rdb
#进入conf目录
cd /opt/canal-adapter/conf
#将 application.yml 从容器中拷贝到conf文件夹
docker cp canal-adapter:/opt/canal-adapter/conf/application.yml ./
#进入rdb文件夹
cd  /opt/canal-adapter/conf/rdb
#将 mytest_user.yml 从容器中拷贝到rdb文件夹
docker cp canal-adapter:/opt/canal-adapter/conf/rdb/mytest_user.yml ./

Modify the copied application.yml,

Insert picture description here


modify the copied mytest_user.yml

Insert picture description here


Stop the running canal-adapter, delete the container, and re-run canal-adapter

#停止容器
docker stop canal-adapter
#删除容器
docker rm canal-adapter
#重新运行canal-adapter,使用挂载出来的配置文件
docker run --name canal-adapter -p 8081:8081 -v /opt/canal-adapter/conf:/opt/canal-adapter/conf -v /opt/canal-adapter/conf/rdb:/opt/canal-adapter/conf/rdb -d slpcat/canal-adapter:v1.1.5

Check whether canal-adapter is running successfully.

Insert picture description here


At this point, docker installation of canal-adapter is complete

5. Verify data synchronization

Add new data in the source database table mytest1.user, refresh in the target database table mytest2.user, you will see that the data has been automatically synchronized. The data synchronization information can also be viewed in the log of the canal-adapter