Ubuntu20.04 cluster deployment of Azkaban 4.0.0

1 Basic environment installation

1.1 Install git

sudo apt install git

1.2 Install java

sudo apt install openjdk-8-jdk

2 Download and compile

2.1 Download azkaban 4.0.0

2.2 Compile azkaban 4.0.0

./gradlew build installDist -x test

-x test skip the test.

Insert picture description here
Insert picture description here

Figure 2.1 Source code compilation azkaban

3 Generated core files

After azkaban source code is compiled, three core files are generated, as shown in the following table:

Serial numberfiledescription
1azkaban-web-serverClient, configure timing tasks
2azkaban-exec-serverActuator, perform timing tasks, provide computing resources
3azkaban-dbDatabase configuration, storage of timing tasks, actuator information, etc.

Core file download:

3.1 webserver

Client, web page, used to configure timed tasks, after defining tasks, specify the executor to execute.

/home/xindaqi/software/azkaban-4.0.0/azkaban-web-server/build/distributions

3.2 exec-server

Executor, the timed tasks configured on the web side should be executed in exec.

/home/xindaqi/software/azkaban-4.0.0/azkaban-exec-server/build/distributions

3.3 db

Database, storing timing task information, actuator information, etc.

/home/xindaqi/software/azkaban-4.0.0/azkaban-db/build/distributions

4 deployment

Serial numberpathdescription
1/home/xindaqi/software/install/azkaban4/azkaban-db
2/home/xindaqi/software/install/azkaban4/azkaban-exec-server
3/home/xindaqi/software/install/azkaban4/azkaban-web-server

4.1 Unzip unzip

Unzip the compressed file to the above path.

4.2 Create database and data table

-- 创建数据库
create database azkaban;
-- 使用库
use azkaban;
-- 从文件创建表
source /home/xindaqi/software/install/azkaban4/azkaban-db/create-all-sql-0.1.0-SNAPSHOT.sql;
Insert picture description here

Figure 4.1 Create database and table

4.3 Create ssl (can be ignored for local installation)

It is not necessary to use ssl to learn Azkaban locally. If it is for other purposes, ssl needs to be configured.

  • Enter the ssl folder:
cd /home/xindaqi/software/install/azkaban4/ssl
  • Create ssl
keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Insert picture description here

Figure 4.2 Generate ssl

4.4 web configuration

4.4.1 Web page configuration and users

The web page is the data displayed on the client page, as described in the following table:

Serial numberAttributesdescription
1azkaban.namePage name
2azkaban.labelPage label
3default.timezone.idTime zone
4user.manager.xml.fileUser information profile
  • File location
your path/azkaban-web-server/conf/azkaban.properties
  • cmd
vim /home/xindaqi/software/install/azkaban4/azkaban-web-server/conf/azkaban.properties
Insert picture description here

Figure 4.3 Web client page configuration

4.4.2 Database configuration

Web database configuration is to configure the database that the client connects to. The database is azkaban, and the user name and password are the respective MySQL configurations.
The database configuration attributes are as follows:

Serial numberAttributesdescription
1database.typeDatabase type
2mysql.portDatabase port
3mysql.hostDatabase ip
4mysql.databaseName database
5mysql.userDatabase user name
6mysql.passwordDatabase password
  • File location
your path/azkaban-web-server/conf/azkaban.properties
  • cmd
vim /home/xindaqi/software/install/azkaban4/azkaban-web-server/conf/azkaban.properties
Insert picture description here

Figure 4.4 Web client database

4.5 executor configuration

The actuator needs to be configured with two, namely the database and the actuator service port.
The database configuration attributes are as follows:

Serial numberAttributesdescription
1database.typeDatabase type
2mysql.portDatabase port
3mysql.hostDatabase ip
4mysql.databaseName database
5mysql.userDatabase user name
6mysql.passwordDatabase password

The actuator port configuration is described in the following table:

Serial numberAttributesdescription
1executor.portActuator port
  • File location
your path/azkaban-exec-server/conf/azkaban.properties
  • cmd
vim /home/xindaqi/software/install/azkaban4/azkaban-exec-server/conf/azkaban.properties
Insert picture description here

Figure 4.5 Actuator database and actuator port configuration

5 Modify the MySQL driver

The MySQL installed this time is version 8.0. Therefore, the driver for Azkaban to connect to MySQL should be version 8.x. However, the original Azkaban MySQL driver is version 5.x. Therefore, download the 8.x driver again.

https://download.csdn.net/download/Xin_101/19361262

5.2 Modify MySQL driver

  • Replace the MySQL driver of web-server.
    Driver location:
/home/xindaqi/software/install/azkaban4/azkaban-web-server/lib

Replace mysql-connector-java-5.1.28.jar with mysql-connector-java-8.0.23.jar

  • Replace the driver of exex-server.
    Driver location:
/home/xindaqi/software/install/azkaban4/azkaban-exec-server/lib

Replace mysql-connector-java-5.1.28.jar with mysql-connector-java-8.0.23.jar

6 Start

6.1 Start exec

  • path
/home/xindaqi/software/install/azkaban4/azkaban-exec-server
  • cmd
./bin/start-exec.sh

6.2 Activate executor

After starting the exec-server executor service, the executor needs to be activated, that is, the executor status is set to 1, and it is available.

SELECT * FROM azkaban.executors;
UPDATE `azkaban`.`executors` SET `active` = '1', host='192.168.211.129' WHERE (`id` = '1');

注意
host is the ip of the machine where the executor is located, used for communication with the web, and modified to the ip opened to the outside world by the executor.
Active is set to 1, to activate the current executor.

Insert picture description here

Figure 5.1 executor table content

6.3 Start web

  • path
/home/xindaqi/software/install/azkaban4/azkaban-exec-server
  • cmd
./bin/start-web.sh

7 Login

7.1 login

Serial numberAttributesvalue
1usernameazkaban
2passwordazkaban
Insert picture description here

Figure 7.1 Web client login page

7.2 Home

Insert picture description here

Figure 7.2 Web client home page


[1] https://blog.csdn.net/weixin_45737446/article/details/103466605
[2] https://www.cnblogs.com/Transkai/p/10680626.html
[3] https://blog.csdn .net/u013525058/article/details/89203762
[4] https://blog.csdn.net/JavaDestiny/article/details/90091118