Centos 7 下安装azkaban 3.23.0多executor

时间:2022-03-21 16:52:37

1 下载源码:https://github.com/azkaban/azkaban

2 通过IDEA编译源码

3 创建数据库

   (1)在10.211.55.6服务器上创建数据库azkaban

           mysql> CREATE DATABASE azkaban;

   (2)授权用户访问

           mysql> CREATE USER 'root'@'%' IDENTIFIED BY '123456';

           mysql> GRANT SELECT,INSERT,UPDATE,DELETE ON azkaban.* to 'root'@'%' WITH GRANT OPTION;

   (3)设置mysql包大小,并重启mysql

           vi /etc/my.cnf

           max_allowed_packet=1024M

    (4)执行sql文件创建数据库表

           sql路径:azkaban-3.23.0/azkaban-db/build/sql/create-all-sql-0.1.0-SNAPSHOT.sql

4 安装azkaban-web-server

   (1)将azkaban-3.23.0/azkaban-web-server/build/distributions/azkaban-web-server-0.1.0-SNAPSHOT.tar.gz解压缩,并拷贝到安装服务器

   (2)解压缩后只有三个目录:bin,lib,web

   (3)手动增加以下几个目录:conf,extlib,plugins,azkaban,logs

   (4)将azkaban-3.23.0/azkaban-solo-server/build/resources/main/conf下的三个文件拷贝到刚创建conf目录中

            将azkaban-3.23.0/azkaban-solo-server/build/resources/main/log4j.properties文件拷贝到刚创建的conf目录中

            将azkaban-3.23.0/azkaban-web-server/src/main/resources/azkaban/下的所有文件拷贝到azkaban目录中,目录结构保持原样不变

   (5)生成证书

           由于azkaban需要https连接,需要生成证书,在工程目录(bin的上级目录)命令执行

           keytool -keystore keystore -alias jetty -genkey-keyalg RSA

           密码设置为:123456


   (6)配置conf目录下azkaban.properties

            #Azkaban Personalization Settings

            azkaban.name=Test                                                                

            azkaban.label=My Local Azkaban

            azkaban.color=#FF3601

            azkaban.default.servlet.path=/index

            web.resource.dir=web/

            #时区

            default.timezone.id=Asia/Shanghai                       

            #Azkaban UserManager class

            user.manager.class=azkaban.user.XmlUserManager

            #平台登录用户名密码

            user.manager.xml.file=conf/azkaban-users.xml  

 

            #Loader for projects

            executor.global.properties=conf/global.properties

            azkaban.project.dir=projects

 

            # mysql

            database.type=mysql                                  

            mysql.port=3306

            mysql.host=10.211.55.6

            mysql.database=azkaban                                                                                                    

            mysql.user=root

            mysql.password=123456

            mysql.numconnections=100


            # Velocity dev mode

            velocity.dev.mode=false

          

            # jetty和证书

            jetty.maxThreads=25

            jetty.ssl.port=8443

            jetty.port=8081

            jetty.keystore=keystore                                                                                                                    

            jetty.password=123456

            jetty.keypassword=123456

            jetty.truststore=keystore

            jetty.trustpassword=123456

 

            # 多执行器配置

           azkaban.use.multiple.executors=true

           azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus

           azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1

5 安装executor

   (1)将azkaban-3.23.0/azkaban-exec-server/build/distributions/azkaban-exec-server-0.1.0-SNAPSHOT.tar.gz解压,并拷贝到安装服务器

   (2)解压后只有两个目录:bin, lib

   (3)手动增加以下目录:extlib, plugins/jobtypes, conf

   (4)将azkaban-3.23.0/azkaban-solo-server/build/resources/main/conf下的三个文件拷贝到刚创建conf目录中

            将azkaban-3.23.0/azkaban-solo-server/build/resources/main/log4j.properties文件拷贝到刚创建的conf目录中

            将azkaban-3.23.0/azkaban-solo-server/src/main/resources/commonprivate.properties文件考本到刚创建的plugins/jobtypes目录中

    (5)配置conf目录下的azkaban.properties文件

             #Azkaban 时区

             default.timezone.id=Asia/Shanghai  

 

             # Azkaban JobTypes Plugins(插件)

             azkaban.jobtype.plugin.dir=plugins/jobtypes

 

             #Loader for projects

             executor.global.properties=conf/global.properties

             #执行器会把数据库中的文件下载到此目录

             azkaban.project.dir=projects

 

             #数据库配置

             database.type=mysql

             mysql.port=3306

             mysql.host=10.211.55.6

             mysql.database=azkaban

             mysql.user=root

             mysql.password=123456

             mysql.numconnections=100

 

             # Azkaban Executor settings

             executor.maxThreads=50

             # 默认端口

             executor.port=12321

             executor.flow.threads=30

    (6)修改commonprivate.properties文件,

            # 默认是true,表示特定用户执行任务

            execute.as.user=false

6 数据库中插入executor节点

    insert into executors(host,port,active) values(“10.211.55.6",12321,1);

    insert into executors(host,port,active) values(“10.211.55.7",12321,1);

    insert into executors(host,port,active) values(“10.211.55.8",12321,1);


7 启动azkaban(先执行executor,再执行web,否则web工程会因为找不到执行器启动失败)

  (1)首先启动executor

            进入azkaban-3.23.0/azkaban-exec-server

            sh bin/start-exec.sh

            查看logs下的日志文件是否有错误,无错误则启动成功


(2)首先启动web

            进入azkaban-3.23.0/azkaban-web-server

            sh bin/start-exec.sh

            查看logs下的日志文件是否有错误,无错误则启动成功


  (3)通过:https://10.211.55.6:8443访问azkaban