Mac搭建Hadoop源码阅读环境

时间:2022-05-11 17:27:45

1、本次Hadoop源码阅读环境使用的阅读工具是idea,Hadoop版本是2.7.3。需要安装的工具包括idea、jdk、maven、protobuf等

2、jdk,使用的版本是1.8版,在jdk官网下载jdk-8u111-macosx-x64.dmg,点击安装,一路next。

3、idea安装,略

4、maven,使用的版本是3.3.9,下载apache-maven-3.3.9-bin.tar,解压:

tar -zxvf  apache-maven-3.3.9-bin.tar

进入 Maven 安装根目录 cd apache-maven-3.3.9

进入 Maven配置文件目录 cd conf

编辑settings.xml文件 vi  settings.xml

设置Maven本地库的路径:这个路径要创建好。

 <localRepository>/opt/PRG/apache-maven-3.3.9/repository</localRepository>

同时在加入阿里云的Maven库,参考:maven使用阿里云仓库

<mirror>
<id>nexus-aliyun</id>
<mirrorOf>*</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>

设置Maven环境变量:

vi ~/.bash_profile

export MAVEN_HOME=/opt/PRG/apache-maven-3.3.9
export PATH=${PATH}:${MAVEN_HOME}/bin:
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
export PATH

更新环境变量 source ~/.bash_profile

mvn -version 能看到相应的版本,则Maven安装成功。

5、安装protobuf,下载protobuf-2.5.0.tar.gz,下载地址: http://pan.baidu.com/s/1pJlZubT (网友提供)

解压:tar -zxvf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

依次执行,参考:http://blog.sina.com.cn/s/blog_7d1531ed0101fmc5.html

 1 ./configure
2 make
3 make check
4 make install

如果报没有权限的错误,su root 切换到root执行。

最后输入  protoc --version命令,如显示libprotoc 2.5.0则安装成功

6、编译Hadoop源码

在Hadoop官网下载hadoop-2.7.3-src.tar.gz源码

解压:tar -zxvf  hadoop-2.7.3-src.tar.gz

cd hadoop-2.7.3-src

执行:

mvn package -Pdist -DskipTests -Dtar

最后是漫长的编译过程。。。

下载依赖jar和相关文件。。。

最后输出:[INFO] Reactor Summary:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 5.429 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 2.148 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.061 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.297 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.200 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.627 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.751 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 4.644 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 6.750 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 3.126 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:10 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 5.191 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [01:12 min]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.037 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:38 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 34.932 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [ 13.482 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 3.823 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.039 s]
[INFO] hadoop-yarn ........................................ SUCCESS [ 0.041 s]
[INFO] hadoop-yarn-api .................................... SUCCESS [ 28.981 s]
[INFO] hadoop-yarn-common ................................. SUCCESS [ 22.892 s]
[INFO] hadoop-yarn-server ................................. SUCCESS [ 0.028 s]
[INFO] hadoop-yarn-server-common .......................... SUCCESS [ 7.887 s]
[INFO] hadoop-yarn-server-nodemanager ..................... SUCCESS [ 9.785 s]
[INFO] hadoop-yarn-server-web-proxy ....................... SUCCESS [ 2.782 s]
[INFO] hadoop-yarn-server-applicationhistoryservice ....... SUCCESS [ 5.300 s]
[INFO] hadoop-yarn-server-resourcemanager ................. SUCCESS [ 14.558 s]
[INFO] hadoop-yarn-server-tests ........................... SUCCESS [ 4.048 s]
[INFO] hadoop-yarn-client ................................. SUCCESS [ 4.785 s]
[INFO] hadoop-yarn-server-sharedcachemanager .............. SUCCESS [ 3.260 s]
[INFO] hadoop-yarn-applications ........................... SUCCESS [ 0.024 s]
[INFO] hadoop-yarn-applications-distributedshell .......... SUCCESS [ 2.129 s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SUCCESS [ 1.734 s]
[INFO] hadoop-yarn-site ................................... SUCCESS [ 0.026 s]
[INFO] hadoop-yarn-registry ............................... SUCCESS [ 4.279 s]
[INFO] hadoop-yarn-project ................................ SUCCESS [ 3.812 s]
[INFO] hadoop-mapreduce-client ............................ SUCCESS [ 0.124 s]
[INFO] hadoop-mapreduce-client-core ....................... SUCCESS [ 14.922 s]
[INFO] hadoop-mapreduce-client-common ..................... SUCCESS [ 11.181 s]
[INFO] hadoop-mapreduce-client-shuffle .................... SUCCESS [ 2.904 s]
[INFO] hadoop-mapreduce-client-app ........................ SUCCESS [ 7.072 s]
[INFO] hadoop-mapreduce-client-hs ......................... SUCCESS [ 4.399 s]
[INFO] hadoop-mapreduce-client-jobclient .................. SUCCESS [ 4.986 s]
[INFO] hadoop-mapreduce-client-hs-plugins ................. SUCCESS [ 1.767 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 4.552 s]
[INFO] hadoop-mapreduce ................................... SUCCESS [ 2.718 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 5.164 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 8.356 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 2.022 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 4.104 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 3.251 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.897 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 1.855 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 2.406 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 0.023 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.771 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 17.460 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 4.841 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 6.743 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.029 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 4.379 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 6.696 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.022 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 29.114 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:08 min
[INFO] Finished at: 2017-01-02T11:41:11+08:00
[INFO] Final Memory: 229M/959M
[INFO] ------------------------------------------------------------------------

没有报错,则说明编译成功。

7、把编译好的Hadoop源码导入Idea

Mac搭建Hadoop源码阅读环境

源码路径

找到编译好的Hadoop,选择即可,next。

选择Maven工程

Mac搭建Hadoop源码阅读环境

之后一路next即可,之后就可以开始阅读源码了。