Batch workflow task scheduler - Azkaban

Azkaban is a batch workflow task scheduler developed by Linkedin. Azkaban is used to run a set of tasks and processes in a specific order within a workflow. Azkaban is configured with simple key:value pairs and dependencies in the configuration. Azkaban uses job profiles to establish dependencies between tasks and provides an easy-to-use Web user interface to maintain and track your workflow.

component

The characteristics of

Compatible with any version of Hadoop

Easy to use Web user interface

Simple workflow upload

You can easily set relationships between tasks

Scheduling workflow

Modular and pluggable plug-in mechanism

Authentication/Authorization (permission work)

Ability to kill and restart workflows

Email reminders of failures and successes

The installation

Azkaban download address: Azkaban. Making. IO/downloads. H…

Azkaban source: github.com/azkaban/azk…

Azkaban plugin: github.com/azkaban/azk…

Azkaban documentation: github.com/azkaban/azk…

create

mkdir /opt/module/azkaban
Copy the code

The installation files

$ls-la total 355432 - RW-r --r--@ 1 baxiang wheel 11157302 Jul 23 18:15 azkaban-executor-server-2.5.0.tar.gz -rw-r--r--@ 1 baxiang wheel 11989669 Jul 23 18:15 azkaban-sqL-script-2.5.0.tar. gz-rw-r --r--@ 1 baxiang wheel 11989669 Jul 23 18:15 Azkaban - a web server - 2.5.0. Tar. GzCopy the code

Unzip the files

$tar -zxf azkaban-executor-server-2.1.0.tar. gz -c /opt/module/azkaban/ $tar -zxf azkaban-sqL-script-2.1.0.tar. gz -c /opt/module/azkaban/ $tar -zxf azkaban-sqL-script-2.1.0.tar. gz -c /opt/module/azkaban/ $tar -zxf azkaban-web-server-2.5.0.tar.gz -c /opt/module/azkaban/Copy the code

Rename the decompressed file

$ cd/opt/module/azkaban/ $ls-ll total 0 drwxr-xr-x 23 Baxiang wheel 736 Aug 30 00:01 drwxr-xr-x 8 baxiang Wheel 256 Aug 30 00:01 AZkaban -executor-2.5.0 drwxr-xr-x 9 Baxiang Wheel 288 Aug 30 00:04 AZkaban -web-2.5.0 $mv Azkaban-executor-2.5.0 / executor $mv azkaban-web-2.5.0/ serverCopy the code

Start docker mysql

$ docker start mysql
mysql
Copy the code

$ mycli -uroot -hlocalhost
Password:
mysql 5.7.26
mycli 1.19.0
Chat: https://gitter.im/dbcli/mycli
Mail: https://groups.google.com/forum/#! forum/mycli-users
Home: http://mycli.net
Thanks to the contributor - www.mysqlfanboy.com
mysql root@localhost:(none)> create database azkaban;
Query OK, 1 row affected
Time: 0.002s
mysql root@localhost:(none)> use azkaban;
You are now connected to database "azkaban" as user "root"Time: 0.001 sCopy the code

Execute Mysql statement

mysql root@localhost:azkaban> source/ opt/module/azkaban/azkaban - 2.5.0 / create - all - SQL - 2.5.0. SQL; mysql root@localhost:azkaban> show tables; +------------------------+ | Tables_in_azkaban | +------------------------+ | active_executing_flows | | active_sla | | execution_flows | | execution_jobs | | execution_logs | | project_events | | project_files | | project_flows | | project_permissions | | project_properties | | project_versions | | projects | | properties | | schedules | | triggers |  +------------------------+ 15 rowsin setTime: 0.018 sCopy the code

Generate key pairs and certificates

$ cd server/
$ keytool -keystore keystore -alias jetty -genkey -keyalg RSA
Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:
What is the name of your organizational unit?
  [Unknown]:
What is the name of your organization?
  [Unknown]:
What is the name of your City or Locality?
  [Unknown]:
What is the name of your State or Province?
  [Unknown]:
What is the two-letter country code for this unit?
  [Unknown]:
Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?
  [no]:  y

Enter key password for <jetty>
	(RETURN if same as keystore password):
Copy the code

$ ls -ll
total 16
-rw-r--r--  1 baxiang  wheel   266 Apr 22  2014 azkaban-users.xml
-rw-r--r--  1 baxiang  wheel  1022 Apr 22  2014 azkaban.properties
Copy the code

vim azkaban.properties

#Azkaban Personalization Settings
# server UI name, used to display the name above the server
azkaban.name=Test
# description
azkaban.label=My Local Azkaban
# UI color
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
The default web server directory for storing Web files
web.resource.dir=/opt/module/azkaban/server/web/
The default time zone has been changed to Asia/Shanghai and the default time zone is USA
default.timezone.id=Asia/Shanghai

#Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
User permission management default class (absolute path)
user.manager.xml.file=/opt/module/azkaban/server/conf/azkaban-users.xml

#Loader for projects
#global configuration file location (absolute path)
executor.global.properties=/opt/module/azkaban/executor/conf/global.properties
azkaban.project.dir=projects

# database type
database.type=mysql
# port
mysql.port=3306
Database connection IP
mysql.host=localhost
Database instance name
mysql.database=azkaban
Database user name
mysql.user=root
# database password
mysql.password=baxiang
# Maximum number of connections
mysql.numconnections=100

# Velocity dev mode
velocity.dev.mode=false

# Azkaban Jetty server properties.
# Jetty server properties.
# Maximum number of threads
jetty.maxThreads=25
# Jetty SSL port
jetty.ssl.port=8443
# Jetty port
jetty.port=8081
#SSL file name (absolute path)
jetty.keystore=/opt/module/azkaban/server/keystore
SSL file password
jetty.password=000000
#Jetty master password is the same as the keystore file
jetty.keypassword=000000
#SSL file name (absolute path)
jetty.truststore=/opt/module/azkaban/server/keystore
SSL file password
jetty.trustpassword=000000

# Azkaban Executor settings
executor.port=12321

# mail settings
mail.sender=
mail.host=
job.failure.email=
job.success.email=

lockdown.create.projects=false

cache.directory=cache

Copy the code

vim azkaban-users.xml

<azkaban-users>
        <user username="azkaban" password="azkaban" roles="admin" groups="azkaban" />
        <user username="metrics" password="metrics" roles="metrics"/>
        <user username="baxiang" password="baxiang" roles="admin" groups="azkaban" />
        <role name="admin" permissions="ADMIN" />
        <role name="metrics" permissions="METRICS"/>
</azkaban-users>
Copy the code

Perform server configuration modification

$ cd executor/
$ cd conf
$ vim azkaban.properties
Copy the code

Azkaban. properties Modified contents

default.timezone.id=Asia/Shanghai

# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=plugins/jobtypes

#Loader for projects
executor.global.properties=/opt/module/azkaban/executor/conf/global.properties
azkaban.project.dir=projects

database.type=mysql
mysql.port=3306
mysql.host=localhost
mysql.database=azkaban
mysql.user=root
mysql.password=baxiang
mysql.numconnections=100

# Azkaban Executor settings
executor.maxThreads=50
executor.port=12321
executor.flow.threads=30
Copy the code

Start the

 bin baxiang$ ./azkaban-executor-start.sh
 bin baxiang$ ./azkaban-web-start.sh
Copy the code

UI: https://localhost:8443 Note that the value is HTTPS

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Batch workflow task scheduler – Azkaban

component

The characteristics of

The installation

Batch workflow task scheduler – Azkaban

component

The characteristics of

The installation

Related Posts

The shell script batch queries the number of matches for each keyword in a file

3 – Object class

Spring Certification Spring Bean lifecycle tutorial