Ubuntu/UbuntuKylin平台下源码编译Ambari 2.4
1、Ubuntu环境配置
1.1 网络配置
kylin@Ubuntu:~$ sudo vim /etc/network/interfaces
\# interfaces(5) file used by ifup(8) and ifdown(8)
auto ens32
iface ens32 inet static
address 192.168.1.166
gateway 192.168.1.1
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
重启系统
kylin@Ubuntu:~$ sudo /etc/init.d/networking restart
[ ok ] Restarting networking (via systemctl): networking.service.1.2 Ubuntu安装SSH
kylin@Ubuntu:~$ dpkg -l | grep ssh
ii libssh-4:amd64 0.7.3-1 amd64 tiny C SSH library (OpenSSL flavor)
ii libssh-gcrypt-4:amd64 0.7.3-1 amd64 tiny C SSH library (gcrypt flavor)
ii openssh-client 1:7.3p1-1 amd64 secure shell (SSH) client, for secure access to remote machines
ii openssh-server 1:7.3p1-1 amd64 secure shell (SSH) server, for secure access from remote machines
ii openssh-sftp-server 1:7.3p1-1 amd64 secure shell (SSH) sftp server module, for SFTP access from remote machines
ii ssh 1:7.3p1-1 all secure shell client and server (metapackage)
ii ssh-import-id 5.6-0ubuntu1 all securely retrieve an SSH public key and install it locally
1.3 如何让ubuntukylin开机进入纯命令模式
找了搜索了很多方法都是针对ubuntu，ubuntukylin没有用！！
kylin@Ubuntu:~
Failed to use bus name org.freedesktop.DisplayManager, do you have appropriate permissions?
1.4 JDK
kylin@Ubuntu:~$ dpkg -l | grep jdk
kylin@Ubuntu:~$ dpkg -l | grep java
ii gir1.2-javascriptcoregtk-4.0:amd64 2.14.0-1 amd64 JavaScript engine library from WebKitGTK+ - GObject introspection data
ii libjavascriptcoregtk-1.0-0:amd64 2.4.11-3 amd64 JavaScript engine library from WebKitGTK+
ii libjavascriptcoregtk-4.0-18:amd64 2.14.0-1 amd64 JavaScript engine library from WebKitGTK+
kylin@Ubuntu:~$
kylin@Ubuntu:~$ java -version
程序 'java' 已包含在下列软件包中：
* default-jre
* gcj-5-jre-headless
* openjdk-8-jre-headless
* gcj-4.8-jre-headless
* gcj-4.9-jre-headless
* gcj-6-jre-headless
* openjdk-9-jre-headless
请尝试：sudo apt install <选定的软件包>
kylin@Ubuntu:~$ sudo vi .profile
#set for java
export JAVA_HOME=/home/kylin/jdk1.7.0_67
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export _JAVA_OPTIONS="-Xmx2048m -XX:MaxPermSize=512m -Djava.awt.headless=true"
export PATH=$PATH:$JAVA_HOME/bin
kylin@Ubuntu:~$ source .profile
kylin@Ubuntu:~$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)1.5 Maven
kylin@Ubuntu:~$ vi .profile
#set for maven
export MAVEN_HOME=/home/kylin/apache-maven-3.0.5
export PATH=$PATH:$MAVEN_HOME/bin
kylin@Ubuntu:~$ source .profile maven有2个位置可以选择配置M2_HOME/conf/setting.xml和~/.m2/setting.xml
前者是全局性的，后者是针对用户的。
查询资料是说用户配置优先级高于全局配置
kylin@Ubuntu:~/apache-maven-3.0.5/conf$ cp settings.xml ~/.m2/1.6 Node.js
kylin@Ubuntu:~$ sudo tar -zxvf node-v0.10.44-linux-x64.tar.gz -C /opt
kylin@Ubuntu:~$ vi .profile
#set for nodejs
export NODE_HOME=/home/kylin/node-v0.10.44-linux-x64
export PATH=$PATH:$NODE_HOME/bin
kylin@Ubuntu:~$ source .profile 配置淘宝源
kylin@Ubuntu:~$ npm config set registry https://registry.npm.taobao.org
kylin@Ubuntu:~$ npm info underscor 安装brunch
kylin@Ubuntu:~$ npm install -g brunch@1.7.20
npm WARN deprecated minimatch@1.0.0: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN optional dep failed, continuing fsevents@0.3.8
npm WARN deprecated minimatch@0.2.14: Please update to minimatch 3.0.2 or higher to avoid a RegExp DoS issue
npm WARN deprecated graceful-fs@2.0.3: graceful-fs v3.0.0 and before will fail on node releases >= v7.0. Please update to graceful-fs@^4.0.0 as soon as possible. Use 'npm ls graceful-fs' to find it in the tree.
/home/kylin/node-v0.10.44-linux-x64/bin/brunch -> /home/kylin/node-v0.10.44-linux-x64/lib/node_modules/brunch/bin/brunch
> brunch@1.7.20 postinstall /home/kylin/node-v0.10.44-linux-x64/lib/node_modules/brunch
> node setup.js postinstall
brunch@1.7.20 /home/kylin/node-v0.10.44-linux-x64/lib/node_modules/brunch
├── debug@0.7.4
├── async-waterfall@0.1.5
├── commonjs-require-definition@0.1.2
├── async-each@0.1.6
├── commander@2.0.0
├── ncp@0.4.2
├── mkdirp@0.3.5
├── init-skeleton@0.2.4 (rimraf@2.2.8)
├── read-components@0.6.1 (component-builder@0.10.1)
├── source-map@0.1.43 (amdefine@1.0.1)
├── coffee-script@1.8.0
├── anymatch@1.0.0 (minimatch@1.0.0)
├── chokidar@0.12.6 (readdirp@1.3.0)
├── anysort@1.0.1 (anymatch@1.3.0)
├── pushserve@0.1.6 (connect-slashes@0.0.11, express@3.3.8)
└── loggy@0.2.2 (growl@1.8.1, ansi-color@0.2.1, date-utils@1.2.21)1.7 安装rpmbuild
ambari编译时依赖rpmbuild打包成rpm安装包，因而需要rpm套件，很幸运，ubuntu可以安装这一套件：
kylin@Ubuntu:~$ sudo apt-get install -y rpm yum1.8 Python
Ubuntu默认是Python 2.7，直接设置setuptools-0.6c11-py2.7.egg
kylin@Ubuntu:~$ sudo sh setuptools-0.6c11-py2.7.egg
Processing setuptools-0.6c11-py2.7.egg
Copying setuptools-0.6c11-py2.7.egg to /usr/local/lib/python2.7/dist-packages
Adding setuptools 0.6c11 to easy-install.pth file
Installing easy_install script to /usr/local/bin
Installing easy_install-2.7 script to /usr/local/bin
Installed /usr/local/lib/python2.7/dist-packages/setuptools-0.6c11-py2.7.egg
Processing dependencies for setuptools==0.6c11
Finished processing dependencies for setuptools==0.6c111.9 git
kylin@Ubuntu:~$ sudo apt-get install -y git2 源码编译Ambari
2.1 下载源码
apache-ambari-2.4.2-src.tar.gz
解压缩
kylin@Ubuntu:~$ tar -zxvf apache-ambari-2.4.2-src.tar.gz
2.2 安装node-gyp
进入源码根目录安装node-gyp
kylin@Ubuntu:~$ cd apache-ambari-2.4.2-src/
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ ls
ambari-admin ambari-client ambari-funtest ambari-metrics ambari-server ambari-views contrib docs LICENSE.txt NOTICE.txt pom.xml
ambari-agent ambari-common ambari-logsearch ambari-project ambari-shell ambari-web dev-support KEYS logo.png pass.txt version
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ npm install node-gyp2.3 准备编译
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ mvn versions:set -DnewVersion=2.4.2.0.0
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ pushd ambari-metrics
~/apache-ambari-2.4.2-src/ambari-metrics ~/apache-ambari-2.4.2-src
kylin@Ubuntu:~/apache-ambari-2.4.2-src/ambari-metrics$ cd ..
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ popd
~/apache-ambari-2.4.2-src2.4 开始编译
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ mvn -B clean install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Preplaceurl3、遇到问题：
问题1、Too many files with unapproved license
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project ambari: Too many files with unapproved license: 780 See RAT report in: /home/kylin/apache-ambari-2.4.2-src/target/rat.txt -> [Help 1]
解决办法：加上-Drat.skip=true
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ mvn -B install package jdeb:jdeb -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true -Preplaceurl问题2、#include
问题3、UnknownHostException Unknown host
问题3.1 Unknown host grafanarel.s3.amazonaws.com
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-grafana: An Ant BuildException has occured: java.net.UnknownHostException: grafanarel.s3.amazonaws.com
[ERROR] around Ant part …… @ 5:217 in /home/kylin/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana/target/antrun/build-Download Ambari Grafana.xml: Unknown host grafanarel.s3.amazonaws.com
问题3.2 Unknown host private-repo-1.hortonworks.com
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (default) on project ambari-metrics-timelineservice: An Ant BuildException has occured: java.net.UnknownHostException: private-repo-1.hortonworks.com
[ERROR] around Ant part …… @ 5:265 in /home/kylin/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-timelineservice/target/antrun/build-Download HBase.xml: Unknown host private-repo-1.hortonworks.com
问题3.3 Unknown host www.w3.org
[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /home/kylin/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: java.net.UnknownHostException: www.w3.org: Unknown host www.w3.org -> [Help 1]
解决办法：重新执行
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true问题4、findbugsXml.html
[ERROR] Failed to execute goal org.codehaus.mojo:xml-maven-plugin:1.0:transform (default) on project ambari-server: Failed to transform input file /home/kylin/apache-ambari-2.4.2-src/ambari-server/target/findbugs/findbugsXml.html: The entity name must immediately follow the ‘&’ in the entity reference. -> [Help 1]
解决办法1：
重新执行
kylin@Ubuntu:~/apache-ambari-2.4.2-src$ mvn -B install package jdeb:jdeb -DnewVersion=2.4.2.0.0 -DskipTests -Dpython.ver="python >= 2.6" -Drat.skip=true解决办法2:
如果网络情况不好，这一步就很难过去。那只好修改pom.xml文件，将findbugs内容注释掉：
kylin@Ubuntu:~/apache-ambari-2.4.2-src/ambari-server$ vim pom.xml
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>3.0.3</version>
<configuration>
<failOnError>false</failOnError>
<threshold>Low</threshold>
<findbugsXmlOutputDirectory>${project.basedir}/target/findbugs</findbugsXmlOutputDirectory>
</configuration>
<executions>
<!--
<execution>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
-->
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>xml-maven-plugin</artifactId>
<version>1.0</version>
<executions>
<!--
<execution>
<phase>verify</phase>
<goals>
<goal>transform</goal>
</goals>
</execution>
-->
</executions>
问题5、 is not a valid ‘control’ directory
参考：https://issues.apache.org/jira/browse/AMBARI-12977
问题5.1 ambari-metrics-grafana
Caused by: org.vafer.jdeb.PackagingException: “/home/kylin/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana/src/main/package/deb/control” is not a valid ‘control’ directory)
kylin@Ubuntu:~/apache-ambari-2.4.2-src/ambari-metrics/ambari-metrics-grafana$ vim pom.xml
在最后添加一个添加，如下代码
<plugin>
<groupId>org.vafer</groupId>
<artifactId>jdeb</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<!--Stub execution on direct plugin call - workaround for ambari deb build process-->
<id>stub-execution</id>
<phase>none</phase>
<goals>
<goal>jdeb</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>true</skip>
<attach>false</attach>
<submodules>false</submodules>
<controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
</configuration>
</plugin>
</plugins>
</build>问题5.2 ambari-funtest
[ERROR] Failed to create debian package /home/kylin/apache-ambari-2.4.2-src/ambari-funtest/target/ambari-funtest_2.4.2.0.0_all.deb
org.vafer.jdeb.PackagingException: “/home/kylin/apache-ambari-2.4.2-src/ambari-funtest/src/main/package/deb/control” is not a valid ‘control’ directory)
解决办法：在上面添加一个元素
kylin@Ubuntu:~/apache-ambari-2.4.2-src/ambari-funtest$ vi pom.xml
<plugin>
<groupId>org.vafer</groupId>
<artifactId>jdeb</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<!--Stub execution on direct plugin call - workaround for ambari deb build process-->
<id>stub-execution</id>
<phase>none</phase>
<goals>
<goal>jdeb</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>true</skip>
<attach>false</attach>
<submodules>false</submodules>
<controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
</configuration>
</plugin>
</plugins>
<resources>
<resource>问题5.3 ambari-logsearch
[ERROR] Failed to execute goal org.vafer:jdeb:1.0.1:jdeb (default-cli) on project ambari-logsearch: Failed to create debian package /home/kylin/apache-ambari-2.4.2-src/ambari-logsearch/target/ambari-logsearch_2.4.2.0.0_all.deb: “/home/kylin/apache-ambari-2.4.2-src/ambari-logsearch/src/main/package/deb/control” is not a valid ‘control’ directory) -> [Help 1]
解决办法：在上面添加一个元素
kylin@Ubuntu:~/apache-ambari-2.4.2-src/ambari-logsearch$ vim pom.xml
<plugin>
<groupId>org.vafer</groupId>
<artifactId>jdeb</artifactId>
<version>1.0.1</version>
<executions>
<execution>
<!--Stub execution on direct plugin call - workaround for ambari deb build process-->
<id>stub-execution</id>
<phase>none</phase>
<goals>
<goal>jdeb</goal>
</goals>
</execution>
</executions>
<configuration>
<skip>true</skip>
<attach>false</attach>
<submodules>false</submodules>
<controlDir>${project.basedir}/../src/main/package/deb/control</controlDir>
</configuration>
</plugin>
</plugins>
</build>4、编译完成
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Ambari Main ....................................... SUCCESS [1.620s]
[INFO] Apache Ambari Project POM ......................... SUCCESS [0.078s]
[INFO] Ambari Web ........................................ SUCCESS [59.983s]
[INFO] Ambari Views ...................................... SUCCESS [0.906s]
[INFO] Ambari Admin View ................................. SUCCESS [19.604s]
[INFO] ambari-metrics .................................... SUCCESS [0.434s]
[INFO] Ambari Metrics Common ............................. SUCCESS [0.421s]
[INFO] Ambari Metrics Hadoop Sink ........................ SUCCESS [2.242s]
[INFO] Ambari Metrics Flume Sink ......................... SUCCESS [1.049s]
[INFO] Ambari Metrics Kafka Sink ......................... SUCCESS [0.697s]
[INFO] Ambari Metrics Storm Sink ......................... SUCCESS [2.175s]
[INFO] Ambari Metrics Storm Sink (Legacy) ................ SUCCESS [1.617s]
[INFO] Ambari Metrics Collector .......................... SUCCESS [13.404s]
[INFO] Ambari Metrics Monitor ............................ SUCCESS [1.155s]
[INFO] Ambari Metrics Grafana ............................ SUCCESS [15.134s]
[INFO] Ambari Metrics Assembly ........................... SUCCESS [1:38.729s]
[INFO] Ambari Server ..................................... SUCCESS [2:03.242s]
[INFO] Ambari Functional Tests ........................... SUCCESS [0.445s]
[INFO] Ambari Agent ...................................... SUCCESS [19.449s]
[INFO] Ambari Client ..................................... SUCCESS [0.021s]
[INFO] Ambari Python Client .............................. SUCCESS [0.741s]
[INFO] Ambari Groovy Client .............................. SUCCESS [3.822s]
[INFO] Ambari Shell ...................................... SUCCESS [0.056s]
[INFO] Ambari Python Shell ............................... SUCCESS [0.744s]
[INFO] Ambari Groovy Shell ............................... SUCCESS [1.118s]
[INFO] ambari-logsearch .................................. SUCCESS [0.014s]
[INFO] Ambari Logsearch Appender ......................... SUCCESS [0.287s]
[INFO] Ambari Logsearch Solr Client ...................... SUCCESS [1.501s]
[INFO] Ambari Logsearch Portal ........................... SUCCESS [3:58.085s]
[INFO] Ambari Logsearch Log Feeder ....................... SUCCESS [7:39.905s]
[INFO] Ambari Logsearch Assembly ......................... SUCCESS [0.155s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:49.903s
[INFO] Finished at: Wed Mar 01 15:43:16 CST 2017
[INFO] Final Memory: 231M/1188M
[INFO] ------------------------------------------------------------------------备注：
Ambari Server will create following packages
- RPM will be created under
- AMBARI_DIR/ambari-server/target/rpm/ambari-server/RPMS/noarch.
- DEB will be created under AMBARI_DIR/ambari-server/target/
Ambari Agent will create following packages
- RPM will be created under
- AMBARI_DIR/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64.
- DEB will be created under AMBARI_DIR/ambari-agent/target
Ubuntu 14
1)Ambari Repositories
Base URL http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.4.2.0
Repo File http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.4.2.0/ambari.list
Tarball md5 http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.4.2.0/ambari-2.4.2.0-ubuntu14.tar.gz
2)HDP 2.5 Repositories
Repo File http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.5.3.0/hdp.list
Tarball md5 http://public-repo-1.hortonworks.com/HDP/ubuntu14/2.x/updates/2.5.3.0/HDP-2.5.3.0-ubuntu14-deb.tar.gz
Tarball md5 http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/ubuntu14/HDP-UTILS-1.1.0.21-ubuntu14.tar.gz