oozie:hadoop中的工作流引擎

时间:2021-09-13 10:08:25

一、maven安装

1) 从网站上下载Linux版本的Maven安装包;

apache-maven-3.0.5-bin.tar.gz下载

具体解压目录在 /home/maven/

========================================

2) 解压apache-maven-3.0.5-bin.tar.gz进行安装;

tar -xzvf apache-maven-3.0.5-bin.tar.gz

 

3) 设置maven环境变量;

vim /etc/profile或者vim /etc/environment

 在末尾处增加如下内容:

#maven envirment

export M2_HOME=/home/maven

export M2=$M2_HOME/bin

export MAVEN_OPTS="-Xms256m -Xmx512m"

export PATH=$M2:$PATH

4) 使配置文件生效

source /etc/profile

5)验证maven是否安装成功。

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /home/maven
Java version: 1.6.0_43, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/jdk1.6.0_43/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-24-server", arch: "amd64", family: "unix"
  

二、oozie

1、Build

bin/mkdistro.sh -DskipTests

成功build后,会在目录里生成一个distribution. 
oozie:hadoop中的工作流引擎
hadoop安装时创建的hadoop组和hadoop用户对oozie目录是没有权限,所以最好一开始创建oozie组和oozie用户,同时赋予权限,不然会报错.偷懒用了root用户。 

PS:后来发现,可以单独拷贝/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2下的oozie-3.3.2-distro.tar.gz包到用户自定义目录进行解压使用,除此之外的所有文件可以全部删除。

2、安装

根据官方文档在distibution的目录下(/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2)创建libext,因为装的hadoop版本是1.0.3,所以拷贝$HADOOP_HOME/lib下的所有jar包到libext下,同时拷贝ext-2.2.zip到该目录下。 
还要拷贝$HADOOP_HOME下的所有jar包到libext. 

因为oozie默认并不支持1.0.3版,所以对/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/bin文件夹下的addtowar.sh文件进行修改:

elif [ "${version}" = "0.20.200" ]; then
#List is separated by ":"
hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"
后边增加如下内容:elif [ "${version}" = "1.0.3" ]; then
#List is separated by ":"
hadoopJars="hadoop-core*.jar:jackson-core-asl-*.jar:jackson-mapper-asl-*.jar:commons-configuration-*.jar"

 

 运行oozie-setup.sh创建oozie.war

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-setup.sh -hadoop 1.0.3 /home/hadoop -extjs /home/oozie/ext-2.2.zip

 执行成功后会出现如下界面:

 

拷贝oozie.war到tomcat部署目录下

root@ubuntu:/home/oozie/webapp/src/main/webapp# cp /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie.war ./

修改配置文件:  

<property> 
<name>oozie.service.JPAService.create.db.schema</name>
<value>true</value>
<description>
Creates Oozie DB.
If set to true, it creates the DB schema if it does not exist. If the DB schema exists is a NOP.
If set to false, it does not create the DB schema. If the DB schema does not exist it fails start up.
</description>
</property>

创建Oozie DB

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin  
./ooziedb.sh create -sqlfile oozie.sql -run

结果

DONE
Check DB schema does not exist
DONE
Check OOZIE_SYS table does not exist
DONE
Create SQL schema
DONE
DONE
Create OOZIE_SYS table
DONE
Oozie DB has been created for Oozie version '3.3.1'

运行oozie

cd /usr/local/oozie-3.3.1/distro/target/oozie-3.3.1-distro/oozie-3.3.1/bin
./oozie-start.sh  

开启服务:

  

bin/oozie-run.sh 

运行结果  

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie-run.sh
WARN: Use of this script is deprecated; use 'oozied.sh run' instead

Setting OOZIE_HOME: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2
Setting OOZIE_CONFIG: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf
Sourcing: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf/oozie-env.sh
setting CATALINA_OPTS="$CATALINA_OPTS -Xmx1024m"
Setting OOZIE_CONFIG_FILE: oozie-site.xml
Setting OOZIE_DATA: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data
Setting OOZIE_LOG: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs
Setting OOZIE_LOG4J_FILE: oozie-log4j.properties
Setting OOZIE_LOG4J_RELOAD: 10
Setting OOZIE_HTTP_HOSTNAME: dm4
Setting OOZIE_HTTP_PORT: 11000
Setting OOZIE_ADMIN_PORT: 11001
Setting OOZIE_HTTPS_PORT: 11443
Setting OOZIE_BASE_URL: http://dm4:11000/oozie
Setting CATALINA_BASE: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Setting OOZIE_HTTPS_KEYSTORE_FILE: /root/.keystore
Setting OOZIE_HTTPS_KEYSTORE_PASS: password
Setting CATALINA_OUT: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/catalina.out
Setting CATALINA_PID: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid

Using CATALINA_OPTS: -Xmx1024m -Dderby.stream.error.file=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs/derby.log
Adding to CATALINA_OPTS: -Doozie.home.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2 -Doozie.config.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/conf -Doozie.log.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/logs -Doozie.data.dir=/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/data -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=dm4 -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://dm4:11000/oozie -Doozie.https.keystore.file=/root/.keystore -Doozie.https.keystore.pass=password -Djava.library.path=

Using CATALINA_BASE: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Using CATALINA_HOME: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server
Using CATALINA_TMPDIR: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp
Using JRE_HOME: /usr/lib/jvm/jdk1.6.0_43/jre
Using CLASSPATH: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/bin/bootstrap.jar
Using CATALINA_PID: /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/temp/oozie.pid
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path:
Jun 2, 2013 3:06:46 PM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-11000
Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 338 ms
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Jun 2, 2013 3:06:46 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.36
Jun 2, 2013 3:06:46 PM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor oozie.xml
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-log4j12-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib/slf4j-simple-1.5.8.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
log4j:WARN No appenders could be found for logger (org.apache.jasper.compiler.JspRuntimeContext).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory ROOT
Jun 2, 2013 3:06:50 PM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-11000
Jun 2, 2013 3:06:50 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 3976 ms

 界面:

和例子界面类似,没有内容。

  

 

案例:

 将/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2中的oozie-examples.tar.gz解压,出现examples文件夹

修改

/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples/map-reduce/job.properties文件为如下形式:

nameNode=hdfs://211.80.198.148:9000
jobTracker=211.80.198.148:9001
queueName=default
examplesRoot=examples

oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce
outputDir=map-reduce


 

将文件夹上传到集群

root@dm4:/home/hadoop# bin/hadoop fs -copyFromLocal /home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/examples /user/root

 

运行命令

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run


运行结果:

job: 0000000-130602150647008-oozie-root-W

不上传会出现如下错误:

root@dm4:/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2# bin/oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run
Error: E0504 : E0504: App directory [hdfs://211.80.198.148:9000/user/root/examples/apps/map-reduce] does not exist

 

 

结果

 oozie:hadoop中的工作流引擎

 PS:中间出现的一些错误

 

ERROR: Oozie could not be started

REASON: java.lang.NoClassDefFoundError: org/apache/hadoop/util/ReflectionUtils

 将hadoop的jar包(主目录下和lib目录下)拷贝到如下位置

/home/oozie/distro/target/oozie-3.3.2-distro/oozie-3.3.2/oozie-server/webapps/oozie/WEB-INF/lib

 之后,运行正常。

 

参考:
1、http://angelosun.iteye.com/blog/1835881

2、http://practicalcloudcomputing.com/post/26337621577/installing-and-running-apache-oozie-3-2-x-and-possibly

3、http://wsysisibeibei.blog.163.com/blog/static/16536806120126233155505/

 

 转载请保留:http://www.cnblogs.com/Deron/archive/2013/06/02/3111178.html