Hadoop 3.2.1 win10 64位系统 vs2015 编译
1 环境配置
1.1 JDK下载安装
1.1.1 下载
JDK 1.8 （jdk1.8.0_102）
下载地址 ： http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html（按照电脑版本下载安装）
1.1.2
安装
解压到指定文件夹
（1） 安装JDK
（2）新建系统变量JAVA_HOME=D:\Program
Files\Java\jdk1.8.0_102
（3）编辑系统变量Path，新增%JAVA_HOME%\bin和%JAVA_HOME%\jre\bin
1.2
Marven下载和安装
1.2.1
下载
https://blog.csdn.net/changge458/article/details/53576178
从该网站 http://maven.apache.org/download.cgi 下载
1.2.2
安装
解压到文件夹D:\marven\apache-maven-3.6.3
添加系统环境变量：MARVEN_HOME
D:\marven\apache-maven-3.6.3\
在系统环境变量path中加入
测试mvn是否安装成功，打开cmd，以管理员身份运行，否则容易报错：'mvn' 不是内部或外部命令，也不是可运行的程序
查看是否配置成功可在黑窗口中输入 mvn –v 出现如下图所示说明配置成功
打开D:\marven\apache-maven-3.6.3\conf\settings.xml
加入阿里云镜像，找到mirrors标签，添加如下阿里云镜像地址，marven下载库从这里下载更快速。
<mirror>
<id>nexus-aliyun</id>
<mirrorOf>central</mirrorOf>
<name>Nexus aliyun</name>
<url>http://maven.aliyun.com/nexus/content/groups/public</url>
</mirror>
1.3
编译安装protobuff
Protobuff是用于序列化数据的，hadoop编译需要依赖这个库。
1.3.1
下载
ProtocolBuffer
2.5.0 （两个文件protobuf-2.5.0.zip protoc-2.5.0-win32.zip
）
下载地址 ：
https://github.com/google/protobuf/releases/tag/v2.5.0
注：除了下载protobuf源码外，还需要下载相应版本的编译过的用于Windows平台的protoc命令（protoc-2.5.0-win32.zip）,该命令用于将.proto文件转化为Java或C++源文件。
1.3.2
安装
①
解压ProtocolBuffer到指定目录
②
解压protoc-2.5.0-win32.zip，将protoc.exe复制到C:\WorkSpace\protobuf-2.5.0\src目录下
③
安装ProtocolBuffer，打开CMD命令行
cd
C:\WorkSpace\protobuf-2.5.0\java
mvn
test
mvn
install
protoc
–version 这个命令失败
将protoc.exe所在路径C:\WorkSpace\protobuf-2.5.0\src，添加到系统变量Path
1.4
Git下载和安装
Git是在window的cmd黑窗口中，可以使用linux的命令。不安装会出现错误
Failed
to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on
project hadoop-project-dist: Command execution failed.: Cannot run program
"bash" (in directory
"D:\hadoop\hadoop-3.2.1-src\hadoop-project-dist\target"):
CreateProcess error=2, 系统找不到指定的文件。 -> [Help 1]
1.4.1
下载
https://git-for-windows.github.io/
1.4.2
安装
（1）
（2）、
（3）、
（4）、
（5）、
1.5
CMake 安装
1.5.1
下载
Windows
win64-x64 ZIP cmake-3.16.0-win64-x64.zip
1.5.2
安装
解压到指定文件夹即可，path环境变量添加bin路径。
D:\hadoop\cmake-3.16.0-win64-x64\bin
1.6
Zlib下载安装
1.6.1
下载
http://jaist.dl.sourceforge.net/project/libpng/zlib/1.2.8/
、
1.6.2
安装
解压到指定文件夹，添加zlib系统变量
2
编译hadoop
2.1
升级项目Visual Studio的版本
编译hadoop时，自动编译会去根据项目文件找到编译器visual studio去编译，hadoop3.2.1默认是VS2010，现在要用VS2015去编译，需要将项目文件winutils.sln和native.sln升级为vs2015。由于Visual
Studio版本问题，官方默认使用Visual Studio 2010 Professional，但本文采用Visual
Studio 2015，因此对于生成失败的项目，需要用Visual Studio 2015重新打开，会弹出升级的对话框，升级项目至Visual Studio 2015版本即可：
(1）
Window工具编译
D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\winutils
（2）hadoop.dll编译
D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\native
这个项目编译输出的就是hadoop.dll文件，编辑编译生成按钮，会报很多头文件错误，是因为投啊文件件包含了..\..\..\target\native\javah，这个文件夹内的头文件，编译hadoop时，编译命令自动先复制头文件到此文件夹，然后才能编译。
打开cmd窗口，进入路径D:\hadoop\hadoop-3.2.1-src\，执行下面的命令：
mvn package -Pdist,native-win-DskipTests -Dtar
2.2
Apache
Hadoop Common编译错误
2.2.1
convert-ms-winutils错误描述
错误描述
[INFO]
Apache Hadoop Auth Examples ........................ SUCCESS [ 12.916 s]
[INFO]
Apache Hadoop Common ............................... FAILURE [02:06 min]
[INFO]
Apache Hadoop NFS .................................. SKIPPED
[INFO]
Apache Hadoop KMS .................................. SKIPPED
[INFO]
------------------------------------------------------------------------
[ERROR]
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(convert-ms-winutils) on project hadoop-common: Command execution failed.:
Process exited with an error: 1 (Exit value: 1) -> [Help 1]
[ERROR]
[ERROR]
To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR]
Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR]
For more information about the errors and possible solutions, please read the
following articles:
[ERROR]
[Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR]
After correcting the problems, you can resume the build with the command
[ERROR] mvn <args> -rf :hadoop-common
解决办法：
（1） 确认2.1中两个项目都升级成功，sln文件都是显示Visual
studio 14
（2） VS 2015 x64 native Tools 工具执行命令mvn
package -Pdist,native-win -DskipTests -Dtar -e -X
2.2.2
Javah错误
错误描述
[ERROR]
Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah
(default) on project hadoop-common: Error running javah command: Error
executing command line. Exit code:1 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.codehaus.mojo:native-maven-plugin:1.0-alpha-8:javah
(default) on project hadoop-common: Error running javah command
解决办法
因为D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common路径下的pom.xml文件中，<javahPath>${env.JAVA_HOME}/bin/javah</javahPath>采用env.JAVA_HOME
无法识别，所以讲pom.xml文件中换成javah的绝对路径：D:\Java\jdk1.8.0_181\bin\,一共有两个地方。
然后命令加入参数执行–rf :hadoop-common 表示从hadoop-common开始编译：
mvn
package -Pdist,native-win-DskipTests –Dtar –rf :hadoop-common
2.3
HDFS Native Client编译错误
2.3.1
错误描述
[INFO]
Apache Hadoop HDFS Native Client ................... FAILURE [02:26 min]
[ERROR]
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client: An Ant BuildException has occured:
exec returned: 1
[ERROR]
around Ant part ...<exec failonerror="true"
dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
executable="cmake">... @ 5:140 in
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR]
-> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client: An Ant BuildException has occured:
exec returned: 1
around
Ant part ...<exec failonerror="true"
dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
executable="cmake">... @ 5:140 in
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
2.3.2
解决方法
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\pom.xml文件打开，修改如下部分的true为false；
2.4
hadoop-hdfs-native-client :RelWithDebInfo编译错误
2.4.1
错误描述
Finished
at: 2019-12-01T18:20:30+08:00
[INFO]
------------------------------------------------------------------------
[ERROR]
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client: An Ant BuildException has occured:
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo
does not exist.
[ERROR]
around Ant part ...<copy
todir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/bin">...
@ 13:101 in D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
[ERROR]
-> [Help 1]
2.4.2
解决方法
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\bin\RelWithDebInfo
does not exist.错误是这个目录不存在，则创建这个目录
2.5
执行maven-plugin:1.3.1:exec
(pre-dist)失败
2.5.1
错误描述
Failed
to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (pre-dist) on
project hadoop-hdfs-native-client: Command execution failed.: Process exited
with an error: 1 (Exit value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(pre-dist) on project hadoop-hdfs-native-client: Command execution failed.
2.5.2
解决办法
下载安装Cygwin
下载：http://www.cygwin.com/setup-x86_64.exe ,安装
配置D:\cygwin64\bin到PATH
然后打开cygwin64的terminal窗口，执行命令mvn package -Pdist,native-win
-DskipTests -Dtar -e -X -rf :hadoop-hdfs-native-client
2.6
Msbuild编译错误
2.6.1
错误描述
Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run
(make) on project hadoop-hdfs-native-client:
An Ant BuildException has occured: Execute failed:
java.io.IOException: Cannot run program "msbuild" (in directory
"D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native"):
CreateProcess error=2, ϵͳ▒Ҳ▒▒▒ָ▒▒▒▒▒ļ▒▒▒
[ERROR] around Ant part ...<exec failonerror="false"
dir="D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native"
executable="msbuild">... @ 9:143 in
D:\hadoop\hadoop-3.2.1-src\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml
2.6.2
解决办法
下载安装vscode https://code.visualstudio.com/docs/?dv=win
将路径C:\Program Files
(x86)\MSBuild\14.0\Bin添加到path环境变量。
2.7
maven-surefire-plugin编译错误
2.7.1
错误描述
Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test
(default-test)
2.7.2
解决方法
在D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\pom.xml找到org.apache.maven.plugins，在configuration中添加<testFailureIgnore>true</testFailureIgnore>屏蔽测试错误。
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<testFailureIgnore>true</testFailureIgnore>
<forkCount>${testsThreadCount}</forkCount>
<reuseForks>false</reuseForks>
<argLine>${maven-surefire-plugin.argLine}
-DminiClusterDedicatedDirs=true</argLine>
<systemPropertyVariables>
<testsThreadCount>${testsThreadCount}</testsThreadCount>
<test.build.data>${test.build.data}/${surefire.forkNumber}</test.build.data>
<test.build.dir>${test.build.dir}/${surefire.forkNumber}</test.build.dir>
<hadoop.tmp.dir>${hadoop.tmp.dir}/${surefire.forkNumber}</hadoop.tmp.dir>
<!-- Due to a Maven quirk,
setting this to just -->
<!-- surefire.forkNumber
won't do the parameter substitution. -->
<!-- Putting a prefix in
front of it like "fork-" makes it -->
<!-- work. -->
<test.unique.fork.id>fork-${surefire.forkNumber}</test.unique.fork.id>
</systemPropertyVariables>
</configuration>
</plugin>
2.8
Apache Hadoop Distribution编译错误
2.8.1
错误描述
[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (dist) on project hadoop-dist:
Command execution failed.: Process exited with an error: 1 (Exit value: 1)
-> [Help 1]
2.8.2
解决办法
执行命令mvn package -Pdist,native-win
-DskipTests -Dtar -e -X -rf :hadoop-dist，加上-e和-X参数显示详细的错误。会发现是目录结构问题，找不到如下路径。解决方法就是创建缺少的路径，并将生成的jar包成果物复制到路径下
[DEBUG] Executing command line: [bash,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target]
cp: cannot stat
'/d/hadoop/hadoop-3.2.1-src/hadoop-common-project/hadoop-kms/target/hadoop-kms-3.2.1/*': No such file or
directory
2.9
hadoop-dist: exec (toolshooks)编译错误
2.9.1
错误描述
Executing command line: [bash,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../hadoop-tools]
找不到文件 - *.tools-builtin.txt
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker:
line 137:
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new:
No such file or directory
mv: cannot stat
'D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new':
No such file or directory
Rewriting
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop
Distribution 3.2.1:
[INFO]
[INFO] Apache Hadoop Distribution
......................... FAILURE [ 25.920 s]
[INFO] Apache Hadoop Client Modules
....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage
........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project
................ SKIPPED
[INFO]
------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 33.530 s
[INFO] Finished at:
2019-12-08T11:56:12+08:00
[INFO]
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (toolshooks) on project
hadoop-dist: Command execution failed.: Process exited with an error: 1 (Exit
value: 1) -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException:
Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(toolshooks) on project hadoop-dist: Command execution failed.
2.9.2
解决方法
（1）错误原因和2.8相似，缺少了文件夹和文件，按照错误提示创建文件夹，找到缺少的文件放入指定文件夹。
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../hadoop-tools]
找不到文件 - *.tools-builtin.txt
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-tools-hooks-maker:
line 137:
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new:
No such file or directory
mv: cannot stat
'D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh.new':
No such file or directory
Rewriting
D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target/hadoop-3.2.1/etc/hadoop/hadoop-env.sh
（2）根据描述是添加了/etc/hadoop文件夹，并且从D:\hadoop\hadoop-3.2.1-src\hadoop-common-project\hadoop-common\src\main\conf
\ 中追到了hadoop-env.sh复制过去，但是发现在执行编译时，总是会自动删除这个文件夹和文件。在执行下面的命令时，删除的：
[DEBUG] Executing command line: [bash,
D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching,
3.2.1, D:\hadoop\hadoop-3.2.1-src\hadoop-dist\target]
Current directory /d/hadoop/hadoop-3.2.1-src/hadoop-dist/target
$ rm -rf hadoop-3.2.1
$ mkdir hadoop-3.2.1
$ cd hadoop-3.2.1
（3）打开D:\hadoop\hadoop-3.2.1-src\hadoop-dist/../dev-support/bin/dist-layout-stitching文件，发现dist-layout-stitching中有一个语句run rm
-rf "hadoop-${VERSION}"会删除hadoop-3.2.1文件夹，然后再新建，新建之后再复制，所以怎么加etc/hadoop文件夹都会被删除
（4）在文件末尾处添加如下代码，用代码去创建文件夹，并复制hadoop-env.sh文件。
run mkdir "etc"
run cd "etc"
run mkdir "hadoop"
run cd "hadoop"
run copy
"${ROOT}\hadoop-common-project\hadoop-common\src\main\conf\hadoop-env.sh"
（5）执行命令>mvn package -Pdist,native-win -DskipTests -Dtar -e -X -rf
:hadoop-dist
3
大工告成
[DEBUG]
(f) siteDirectory =
D:\hadoop\hadoop-3.2.1-src\hadoop-cloud-storage-project\src\site
[DEBUG]
(f) skip = false
[DEBUG] -- end configuration --
[INFO] No site descriptor found: nothing to
attach.
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop
Distribution 3.2.1:
[INFO]
[INFO] Apache Hadoop Distribution
......................... SUCCESS [ 39.961 s]
[INFO] Apache Hadoop Client Modules
....................... SUCCESS [ 5.721
s]
[INFO] Apache Hadoop Cloud Storage
........................ SUCCESS [ 10:36
h]
[INFO] Apache Hadoop Cloud Storage Project
................ SUCCESS [ 1.471 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 10:37 h
[INFO] Finished at:
2019-12-09T10:05:07+08:00
[INFO]
------------------------------------------------------------------------
自己开发了一个股票智能分析软件，功能很强大，需要的点击下面的链接获取：
https://www.cnblogs.com/bclshuai/p/11380657.html
参考文献
https://blog.csdn.net/qq_37475168/article/details/90746823