Hadoop2.2源代码编译

时间:2021-11-05 08:27:16

1.安装虚拟机Centos环境

安装镜像下载地址:http://vault.centos.org/6.4/isos/x86_64/

Hadoop2.2源代码编译

下载上图标出的两个。安装时选第一个就可以了。

安装过程略。可以参见如下文档安装。

http://pan.baidu.com/s/1dDowIjv

我的环境是:

[root@hadoop1 target]# uname -a
Linux hadoop1 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux


2.安装jdk----jdk-7u67-linux-x64.tar.gz

由于Hadoop是用Java开发的,故编译时要用到jdk.

此次安装的版本是jdk-7u67-linux-x64.tar.gz

jdk安装文件可以到如下网盘地址去下载:

http://pan.baidu.com/s/1jGqUupw


安装方法:

第一步:tar -zxvf  jdk-7u67-linux-x64.tar.gz

第二步:mv jdk1.7.0_67  jdk1.7

第三步:vi /etc/profile

添加如下内容:

export JAVA_HOME=/usr/local/jdk1.7

export PATH=.:$PATH:$JAVA_HOME/bin

保存后:source /etc/profile

第四步:java -version

[root@hadoop1 soft]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)


3.安装maven---apache-maven-3.0.5-bin.tar.gz

由于Hadoop2是用Maven来管理项目的,故要编译源码,需要用Maven

maven下载地址:http://pan.baidu.com/s/1bnlcZeR

当然也可以到官网去下载


安装步骤:

解压并改名

tar -zxvf apache-maven-3.0.5-bin.tar.gz

mv apache-maven-3.0.5 maven

vi /etc/profile

添加如下内容

export MAVEN_HOME=/usr/local/maven

export PATH=.:$PATH:$MAVEN_HOME

source /etc/profile

验证:mvn -version

[root@hadoop1 soft]# mvn -version
Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 21:51:28+0800)
Maven home: /usr/local/maven
Java version: 1.7.0_67, vendor: Oracle Corporation
Java home: /usr/local/jdk1.7/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"


4.安装findbugs -- findbugs-3.0.0.tar.gz

findbugs是用来生成文档的,如果不需要生成编译文档,此步骤可以忽略。

安装文件下载地址:

http://pan.baidu.com/s/1pJr0dR1


安装步骤:

tar -zxvf findbugs-3.0.0.tar.gz

mv findbugs-3.0.0 findbugs


配置环境变量:

vi /etc/profile

export FINDBUGS_HOME=/usr/local/findbugs

export PATH=.:$PATH:$FINDBUGS_HOME/bin

source /etc/profile

验证:findbugs -version

[root@hadoop1 soft]# findbugs  -version
3.0.0


5.安装protoc

由于Hadoop是使用protocol buffer通信的,故要安装protoc.

官网:https://code.google.com/p/protobuf/downloads/list

我的下载地址:http://pan.baidu.com/s/1zshyA

为了安装protoc,需要安装以下几个工具:

前提:需要centos虚拟机可以连网。

    yum install gcc  
yum intall gcc-c++
yum install make
安装步骤:

tar -zxvf protobuf-2.5.0.tar.gz 

cd protobuf-2.5.0

./configure --prefix=/usr/local/protoc

make 

make install

以上执行步骤只要不出错就可以了

编译文件位于:/usr/local/protoc

配置环境变量:

export PROTOC_HOME=/usr/local/protoc

export PATH=.:$PATH:$PROTOC_HOME

source /etc/profile

验证:

[root@hadoop1 soft]# protoc --version
libprotoc 2.5.0


6.安装其它依赖包

    yum install cmake  
yum install openssl-devel
yum install ncurses-devel
7.编译源码

从官网下载Hadoop2.2源码

http://apache.fayea.com/apache-mirror/hadoop/common/hadoop-2.2.0/

或者从下面的网盘下载:

http://pan.baidu.com/s/1dD6DC2l


安装步骤:

tar -zxvf hadoop-2.2.0-src.tar.gz

cd hadoop-2.2.0-src

修复bug一枚

hadoop-2.2.0-src目录下的如下文件:

/usr/local/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth中的文件pom.xml

在第55行加入如下配置:

     <dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
cd  hadoop-2.2.0-src

mvn package -DskipTests -Pdist,native,docs

如果没有安装生成文档的findbugs,则去掉后边的docs


由于maven需要上网去下载需要的jar包,帮该命令会执行时间较长。

如果最后控制台输出如下信息:

[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop Main ................................ SUCCESS [2.625s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1.623s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [3.604s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.365s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [4.368s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [3.922s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [13.938s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.717s]
[INFO] Apache Hadoop Common .............................. SUCCESS [5:56.983s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [12.786s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.097s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [10:03.524s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [39.430s]
[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [13.182s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [6.215s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.179s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.392s]
[INFO] hadoop-yarn-api ................................... SUCCESS [57.293s]
[INFO] hadoop-yarn-common ................................ SUCCESS [38.550s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.608s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [15.493s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [17.368s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [3.668s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [14.096s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.432s]
[INFO] hadoop-yarn-client ................................ SUCCESS [5.831s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.140s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [3.396s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.148s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [30.974s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [4.917s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.177s]
[INFO] hadoop-yarn-project ............................... SUCCESS [5.514s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [24.587s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.728s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [12.216s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [6.015s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [4.927s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [1.892s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [7.238s]
[INFO] hadoop-mapreduce .................................. SUCCESS [2.417s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [6.014s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [9.235s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [2.558s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [7.183s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [5.011s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [3.364s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [3.617s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [3.970s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [3.288s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.060s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [41.536s]
[INFO] Apache Hadoop Client .............................. SUCCESS [6.460s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.586s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:42.823s
[INFO] Finished at: Fri Aug 08 07:08:01 CST 2014
[INFO] Final Memory: 71M/239M
[INFO] ------------------------------------------------------------------------

其中关键字:BUILD SUCCESS

则表示编译完成。


编译后的文件位于:target目录中。

Hadoop2.2源代码编译

上图标出的就是编译后的Hadoop2.2.