本课主题
- 集群搭建
- 设置 Web 服务器
- 启动 ClouderManager
- 登入 Cloudera Manager
引言
这部份是一个补充资料，记录如何安装 Cloudera 服务器
集群搭建
- 查看你主机的 IP 地址并且设置静态 IP，这里显示的是 192.168.80.145
ifconfig -a
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDQvMTAwNTc5NC0yMDE3MDQyNTEwMDEyOTM1MC0xODYwMjIxMjA2LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
用图型界面来设置静态 IP![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDQvMTAwNTc5NC0yMDE3MDQyNTEwMTMzODk0NC0yNjQ2MTk5MjEucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 修改 network 文件来修改 HOSTNAME
vim /etc/sysconfig/network
hostname elephant
service network restart![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDQvMTAwNTc5NC0yMDE3MDQyNTEwMTY1OTg4MS03OTE2ODIzMDAucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 修改 /etc/hosts 文件
192.168.80.145 elephant
192.168.80.146 monkey
设置 Web 服务器
- 首先是关掉防火墙
service iptables stop
chkconfig iptables off
vim /etc/selinux/config
setenforce 0 - 查看 httpd 服务和启动状况，如果发现没有安装 httpd 服务，请先安装 sudo yum -y install httpd
[enmoedu@elephant softwares]$ rpm -qa | grep httpd
httpd-2.2.15-59.el6.centos.x86_64
httpd-tools-2.2.15-59.el6.centos.x86_64rpm -qa | grep httpd
[root@elephant softwares]# service httpd status
httpd is stopped
[root@elephant softwares]# service httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.80.145 for ServerName
[ OK ]
[root@elephant softwares]# service httpd status
httpd (pid 3002) is running...service httpd status
- 查看 WEB 服务器的状况: http://elephant
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDQvMTAwNTc5NC0yMDE3MDQyNTEyMTEwNzA4NC0xODE0OTU5MTUyLnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 安装好之后把 cloudera-cdh5 和 cloudera-manager 移动到 /var/www/html/ 文件夹里。
http://elephant/cloudera-manager5.9
http://elephant/cloudera-cdh5.9![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDQvMTAwNTc5NC0yMDE3MDQyNTEyMTE0OTYzMS02NDI3MTQ3NjAucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
此时你可以创建安装包啦，进入 cloudera-cdh5 和 cloudera-manager 创建 repo
createrepo .
[root@elephant html]# cd cloudera-cdh5.9/
[root@elephant cloudera-cdh5.9]# ls
repodata RPMS
[root@elephant cloudera-cdh5.9]# createrepo .
Spawning worker 0 with 117 pkgs
Workers Finished
Gathering worker results Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs complete
[root@elephant cloudera-cdh5.9]# cd ..
[root@elephant html]# ls
cloudera-cdh5.9 cloudera-manager5.9
[root@elephant html]# cd cloudera-manager5.9/
[root@elephant cloudera-manager5.9]# ls
repodata RPMS
[root@elephant cloudera-manager5.9]# createrepo .
Spawning worker 0 with 6 pkgs
Workers Finished
Gathering worker results Saving Primary metadata
Saving file lists metadata
Saving other metadata
Generating sqlite DBs
Sqlite DBs completecreaterepo
- 完成以上的步骤，服务器已经搭建好了，剩下来是客户端的配置。创建好 repo 之后可以到 /etc/yum.repos.d/ 文件夹创建 repo文件，新建 cloudera-cdh5 的 yum 源。
[Cloudera-cdh5]
name=Cloudera-cdh5
baseurl=http://elephant/Cloudera-cdh5
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6Cloudera-cdh5.repo
[cloudera-manager]
name=cloudera-manager
baseurl=http://elephant/cloudera-manager
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6cloudera-manager.repo
mv Cloudera-cdh5/ cloudera-manager/ /var/www/html/
service httpd status
cd /var/www/html/ cd Cloudera-cdh5/
createrepo . cd cloudera-manager/
createrepo . cd /etc/yum.repos.d/ cat << EOF > /etc/yum.repos.d/Cloudera-cdh5.repo
[Cloudera-cdh5]
name=Cloudera-cdh5
baseurl=http://elephant/Cloudera-cdh5
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
EOF cat << EOF > /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
name=cloudera-manager
baseurl=http://elephant/cloudera-manager
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
EOFcompleted-code
清理一下 yum
[root@elephant yum.repos.d]# yum clean all
Loaded plugins: fastestmirror, refresh-packagekit
Cleaning repos: base cloudera-cdh5.9 cloudera-manager extras updates
Cleaning up Everything
Cleaning up list of fastest mirrorsyum clean all
[root@elephant yum.repos.d]# yum install spark-core
Loaded plugins: fastestmirror, refresh-packagekit
Setting up Install Process
Determining fastest mirrors
* base: centos.01link.hk
* extras: centos.01link.hk
* updates: centos.01link.hk
base | 3.7 kB 00:00
base/primary_db | 4.7 MB 00:02
cloudera-cdh5.9 | 2.9 kB 00:00
cloudera-cdh5.9/primary_db | 64 kB 00:00
cloudera-manager | 2.9 kB 00:00
cloudera-manager/primary_db | 8.8 kB 00:00
extras | 3.4 kB 00:00
extras/primary_db | 37 kB 00:00
updates | 3.4 kB 00:00
updates/primary_db | 803 kB 00:00
Resolving Dependencies
--> Running transaction check
---> Package spark-core.noarch 0:1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6 will be installed
base/filelists_db | 6.4 MB 00:02
cloudera-cdh5.9/filelists_db | 413 kB 00:00
cloudera-manager/filelists_db | 94 kB 00:00
extras/filelists_db | 38 kB 00:00
updates/filelists_db | 644 kB 00:00
--> Processing Dependency: bigtop-utils >= 0.7 for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: parquet for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hive for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hadoop-client for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: flume-ng for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: avro-libs for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: /lib/lsb/init-functions for package: spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch
--> Running transaction check
---> Package avro-libs.noarch 0:1.7.6+cdh5.9.0+115-1.cdh5.9.0.p0.30.el6 will be installed
---> Package bigtop-utils.noarch 0:0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6 will be installed
---> Package flume-ng.noarch 0:1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: kite >= 0.10.0+cdh5.1.0 for package: flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hive-jdbc >= 1.1.0+cdh5.4.0 for package: flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hadoop-yarn for package: flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hadoop-mapreduce for package: flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hadoop-hdfs for package: flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch
--> Processing Dependency: hadoop for package: flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch
---> Package hadoop-client.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: hadoop-0.20-mapreduce = 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 for package: hadoop-client-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64
--> Processing Dependency: hadoop-0.20-mapreduce = 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 for package: hadoop-client-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64
--> Processing Dependency: zookeeper for package: hadoop-client-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64
---> Package hive.noarch 0:1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: sentry for package: hive-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch
---> Package parquet.noarch 0:1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: parquet-format >= 2.1.0 for package: parquet-1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6.noarch
---> Package redhat-lsb-core.x86_64 0:4.0-7.el6.centos will be installed
--> Processing Dependency: perl-Test-Simple for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: perl-Test-Harness for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: perl-ExtUtils-MakeMaker for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: perl-CGI for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/bin/time for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/bin/pax for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/bin/man for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/bin/bc for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/bin/batch for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /usr/bin/at for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /bin/mailx for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Processing Dependency: /bin/ed for package: redhat-lsb-core-4.0-7.el6.centos.x86_64
--> Running transaction check
---> Package at.x86_64 0:3.1.10-49.el6 will be installed
---> Package bc.x86_64 0:1.06.95-1.el6 will be installed
---> Package ed.x86_64 0:1.1-3.3.el6 will be installed
---> Package hadoop.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: nc for package: hadoop-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64
---> Package hadoop-0.20-mapreduce.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 will be installed
---> Package hadoop-hdfs.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: bigtop-jsvc for package: hadoop-hdfs-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64
---> Package hadoop-mapreduce.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 will be installed
---> Package hadoop-yarn.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 will be installed
---> Package hive-jdbc.noarch 0:1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6 will be installed
---> Package kite.noarch 0:1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: solr >= 4.4.0+cdh5.1.0 for package: kite-1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6.noarch
---> Package mailx.x86_64 0:12.4-8.el6_6 will be installed
---> Package man.x86_64 0:1.6f-39.el6 will be installed
---> Package parquet-format.noarch 0:2.1.0+cdh5.9.0+14-1.cdh5.9.0.p0.30.el6 will be installed
---> Package pax.x86_64 0:3.4-10.1.el6 will be installed
---> Package perl-CGI.x86_64 0:3.51-144.el6 will be installed
--> Processing Dependency: perl = 4:5.10.1-144.el6 for package: perl-CGI-3.51-144.el6.x86_64
---> Package perl-ExtUtils-MakeMaker.x86_64 0:6.55-144.el6 will be installed
--> Processing Dependency: perl-devel for package: perl-ExtUtils-MakeMaker-6.55-144.el6.x86_64
---> Package perl-Test-Harness.x86_64 0:3.17-144.el6 will be installed
---> Package perl-Test-Simple.x86_64 0:0.92-144.el6 will be installed
---> Package sentry.noarch 0:1.5.1+cdh5.9.0+261-1.cdh5.9.0.p0.30.el6 will be installed
---> Package time.x86_64 0:1.7-38.el6 will be installed
---> Package zookeeper.x86_64 0:3.4.5+cdh5.9.0+98-1.cdh5.9.0.p0.30.el6 will be installed
--> Running transaction check
---> Package bigtop-jsvc.x86_64 0:0.6.0+cdh5.9.0+855-1.cdh5.9.0.p0.30.el6 will be installed
---> Package nc.x86_64 0:1.84-24.el6 will be installed
---> Package perl.x86_64 4:5.10.1-141.el6_7.1 will be updated
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: 1:perl-Pod-Escapes-1.04-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: perl-IO-Compress-Base-2.021-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: 1:perl-Compress-Raw-Zlib-2.021-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: perl-IO-Compress-Zlib-2.021-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: 1:perl-Module-Pluggable-3.90-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: perl-Compress-Zlib-2.021-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: 3:perl-version-0.77-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: 4:perl-libs-5.10.1-141.el6_7.1.x86_64
--> Processing Dependency: perl = 4:5.10.1-141.el6_7.1 for package: 1:perl-Pod-Simple-3.13-141.el6_7.1.x86_64
---> Package perl.x86_64 4:5.10.1-144.el6 will be an update
---> Package perl-devel.x86_64 4:5.10.1-144.el6 will be installed
--> Processing Dependency: perl(ExtUtils::ParseXS) for package: 4:perl-devel-5.10.1-144.el6.x86_64
--> Processing Dependency: gdbm-devel for package: 4:perl-devel-5.10.1-144.el6.x86_64
--> Processing Dependency: db4-devel for package: 4:perl-devel-5.10.1-144.el6.x86_64
---> Package solr.noarch 0:4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6 will be installed
--> Processing Dependency: bigtop-tomcat for package: solr-4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6.noarch
--> Running transaction check
---> Package bigtop-tomcat.noarch 0:0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6 will be installed
---> Package db4-devel.x86_64 0:4.7.25-22.el6 will be installed
--> Processing Dependency: db4-cxx = 4.7.25-22.el6 for package: db4-devel-4.7.25-22.el6.x86_64
--> Processing Dependency: db4 = 4.7.25-22.el6 for package: db4-devel-4.7.25-22.el6.x86_64
--> Processing Dependency: libdb_cxx-4.7.so()(64bit) for package: db4-devel-4.7.25-22.el6.x86_64
---> Package gdbm-devel.x86_64 0:1.8.0-39.el6 will be installed
---> Package perl-Compress-Raw-Zlib.x86_64 1:2.021-141.el6_7.1 will be updated
---> Package perl-Compress-Raw-Zlib.x86_64 1:2.021-144.el6 will be an update
---> Package perl-Compress-Zlib.x86_64 0:2.021-141.el6_7.1 will be updated
---> Package perl-Compress-Zlib.x86_64 0:2.021-144.el6 will be an update
---> Package perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-144.el6 will be installed
---> Package perl-IO-Compress-Base.x86_64 0:2.021-141.el6_7.1 will be updated
---> Package perl-IO-Compress-Base.x86_64 0:2.021-144.el6 will be an update
---> Package perl-IO-Compress-Zlib.x86_64 0:2.021-141.el6_7.1 will be updated
---> Package perl-IO-Compress-Zlib.x86_64 0:2.021-144.el6 will be an update
---> Package perl-Module-Pluggable.x86_64 1:3.90-141.el6_7.1 will be updated
---> Package perl-Module-Pluggable.x86_64 1:3.90-144.el6 will be an update
---> Package perl-Pod-Escapes.x86_64 1:1.04-141.el6_7.1 will be updated
---> Package perl-Pod-Escapes.x86_64 1:1.04-144.el6 will be an update
---> Package perl-Pod-Simple.x86_64 1:3.13-141.el6_7.1 will be updated
---> Package perl-Pod-Simple.x86_64 1:3.13-144.el6 will be an update
---> Package perl-libs.x86_64 4:5.10.1-141.el6_7.1 will be updated
---> Package perl-libs.x86_64 4:5.10.1-144.el6 will be an update
---> Package perl-version.x86_64 3:0.77-141.el6_7.1 will be updated
---> Package perl-version.x86_64 3:0.77-144.el6 will be an update
--> Running transaction check
---> Package db4.x86_64 0:4.7.25-20.el6_7 will be updated
--> Processing Dependency: db4 = 4.7.25-20.el6_7 for package: db4-utils-4.7.25-20.el6_7.x86_64
---> Package db4.x86_64 0:4.7.25-22.el6 will be an update
---> Package db4-cxx.x86_64 0:4.7.25-22.el6 will be installed
--> Running transaction check
---> Package db4-utils.x86_64 0:4.7.25-20.el6_7 will be updated
---> Package db4-utils.x86_64 0:4.7.25-22.el6 will be an update
--> Finished Dependency Resolution Dependencies Resolved ==============================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================
Installing:
spark-core noarch 1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 112 M
Installing for dependencies:
at x86_64 3.1.10-49.el6 base 61 k
avro-libs noarch 1.7.6+cdh5.9.0+115-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 45 M
bc x86_64 1.06.95-1.el6 base 110 k
bigtop-jsvc x86_64 0.6.0+cdh5.9.0+855-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 27 k
bigtop-tomcat noarch 0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 7.5 M
bigtop-utils noarch 0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 11 k
db4-cxx x86_64 4.7.25-22.el6 base 589 k
db4-devel x86_64 4.7.25-22.el6 base 6.6 M
ed x86_64 1.1-3.3.el6 base 72 k
flume-ng noarch 1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 52 M
gdbm-devel x86_64 1.8.0-39.el6 base 26 k
hadoop x86_64 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 27 M
hadoop-0.20-mapreduce x86_64 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 36 M
hadoop-client x86_64 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 34 k
hadoop-hdfs x86_64 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 24 M
hadoop-mapreduce x86_64 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 32 M
hadoop-yarn x86_64 2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 15 M
hive noarch 1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 42 M
hive-jdbc noarch 1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 46 M
kite noarch 1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 114 M
mailx x86_64 12.4-8.el6_6 base 235 k
man x86_64 1.6f-39.el6 base 265 k
nc x86_64 1.84-24.el6 base 57 k
parquet noarch 1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 26 M
parquet-format noarch 2.1.0+cdh5.9.0+14-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 467 k
pax x86_64 3.4-10.1.el6 base 69 k
perl-CGI x86_64 3.51-144.el6 base 210 k
perl-ExtUtils-MakeMaker x86_64 6.55-144.el6 base 294 k
perl-ExtUtils-ParseXS x86_64 1:2.2003.0-144.el6 base 46 k
perl-Test-Harness x86_64 3.17-144.el6 base 232 k
perl-Test-Simple x86_64 0.92-144.el6 base 113 k
perl-devel x86_64 4:5.10.1-144.el6 base 424 k
redhat-lsb-core x86_64 4.0-7.el6.centos base 25 k
sentry noarch 1.5.1+cdh5.9.0+261-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 56 M
solr noarch 4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 76 M
time x86_64 1.7-38.el6 base 26 k
zookeeper x86_64 3.4.5+cdh5.9.0+98-1.cdh5.9.0.p0.30.el6 cloudera-cdh5.9 4.4 M
Updating for dependencies:
db4 x86_64 4.7.25-22.el6 base 564 k
db4-utils x86_64 4.7.25-22.el6 base 131 k
perl x86_64 4:5.10.1-144.el6 base 10 M
perl-Compress-Raw-Zlib x86_64 1:2.021-144.el6 base 70 k
perl-Compress-Zlib x86_64 2.021-144.el6 base 46 k
perl-IO-Compress-Base x86_64 2.021-144.el6 base 70 k
perl-IO-Compress-Zlib x86_64 2.021-144.el6 base 136 k
perl-Module-Pluggable x86_64 1:3.90-144.el6 base 41 k
perl-Pod-Escapes x86_64 1:1.04-144.el6 base 33 k
perl-Pod-Simple x86_64 1:3.13-144.el6 base 213 k
perl-libs x86_64 4:5.10.1-144.el6 base 579 k
perl-version x86_64 3:0.77-144.el6 base 52 k Transaction Summary
==============================================================================================================================================================
Install 38 Package(s)
Upgrade 12 Package(s) Total download size: 736 M
Is this ok [y/N]:y
Downloading Packages:
(1/50): at-3.1.10-49.el6.x86_64.rpm | 61 kB 00:00
(2/50): avro-libs-1.7.6+cdh5.9.0+115-1.cdh5.9.0.p0.30.el6.noarch.rpm | 45 MB 00:00
(3/50): bc-1.06.95-1.el6.x86_64.rpm | 110 kB 00:00
(4/50): bigtop-jsvc-0.6.0+cdh5.9.0+855-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 27 kB 00:00
(5/50): bigtop-tomcat-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6.noarch.rpm | 7.5 MB 00:00
(6/50): bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6.noarch.rpm | 11 kB 00:00
(7/50): db4-4.7.25-22.el6.x86_64.rpm | 564 kB 00:00
(8/50): db4-cxx-4.7.25-22.el6.x86_64.rpm | 589 kB 00:00
(9/50): db4-devel-4.7.25-22.el6.x86_64.rpm | 6.6 MB 00:02
(10/50): db4-utils-4.7.25-22.el6.x86_64.rpm | 131 kB 00:00
(11/50): ed-1.1-3.3.el6.x86_64.rpm | 72 kB 00:00
(12/50): flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch.rpm | 52 MB 00:00
(13/50): gdbm-devel-1.8.0-39.el6.x86_64.rpm | 26 kB 00:00
(14/50): hadoop-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 27 MB 00:00
(15/50): hadoop-0.20-mapreduce-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 36 MB 00:00
(16/50): hadoop-client-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 34 kB 00:00
(17/50): hadoop-hdfs-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 24 MB 00:00
(18/50): hadoop-mapreduce-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 32 MB 00:00
(19/50): hadoop-yarn-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 15 MB 00:00
(20/50): hive-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch.rpm | 42 MB 00:00
(21/50): hive-jdbc-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch.rpm | 46 MB 00:00
(22/50): kite-1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6.noarch.rpm | 114 MB 00:01
(23/50): mailx-12.4-8.el6_6.x86_64.rpm | 235 kB 00:00
(24/50): man-1.6f-39.el6.x86_64.rpm | 265 kB 00:00
(25/50): nc-1.84-24.el6.x86_64.rpm | 57 kB 00:00
(26/50): parquet-1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6.noarch.rpm | 26 MB 00:00
(27/50): parquet-format-2.1.0+cdh5.9.0+14-1.cdh5.9.0.p0.30.el6.noarch.rpm | 467 kB 00:00
(28/50): pax-3.4-10.1.el6.x86_64.rpm | 69 kB 00:00
(29/50): perl-5.10.1-144.el6.x86_64.rpm | 10 MB 00:05
(30/50): perl-CGI-3.51-144.el6.x86_64.rpm | 210 kB 00:00
(31/50): perl-Compress-Raw-Zlib-2.021-144.el6.x86_64.rpm | 70 kB 00:00
(32/50): perl-Compress-Zlib-2.021-144.el6.x86_64.rpm | 46 kB 00:00
(33/50): perl-ExtUtils-MakeMaker-6.55-144.el6.x86_64.rpm | 294 kB 00:00
(34/50): perl-ExtUtils-ParseXS-2.2003.0-144.el6.x86_64.rpm | 46 kB 00:00
(35/50): perl-IO-Compress-Base-2.021-144.el6.x86_64.rpm | 70 kB 00:00
(36/50): perl-IO-Compress-Zlib-2.021-144.el6.x86_64.rpm | 136 kB 00:00
(37/50): perl-Module-Pluggable-3.90-144.el6.x86_64.rpm | 41 kB 00:00
(38/50): perl-Pod-Escapes-1.04-144.el6.x86_64.rpm | 33 kB 00:00
(39/50): perl-Pod-Simple-3.13-144.el6.x86_64.rpm | 213 kB 00:00
(40/50): perl-Test-Harness-3.17-144.el6.x86_64.rpm | 232 kB 00:00
(41/50): perl-Test-Simple-0.92-144.el6.x86_64.rpm | 113 kB 00:00
(42/50): perl-devel-5.10.1-144.el6.x86_64.rpm | 424 kB 00:00
(43/50): perl-libs-5.10.1-144.el6.x86_64.rpm | 579 kB 00:00
(44/50): perl-version-0.77-144.el6.x86_64.rpm | 52 kB 00:00
(45/50): redhat-lsb-core-4.0-7.el6.centos.x86_64.rpm | 25 kB 00:00
(46/50): sentry-1.5.1+cdh5.9.0+261-1.cdh5.9.0.p0.30.el6.noarch.rpm | 56 MB 00:00
(47/50): solr-4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6.noarch.rpm | 76 MB 00:00
(48/50): spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch.rpm | 112 MB 00:01
(49/50): time-1.7-38.el6.x86_64.rpm | 26 kB 00:00
(50/50): zookeeper-3.4.5+cdh5.9.0+98-1.cdh5.9.0.p0.30.el6.x86_64.rpm | 4.4 MB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 35 MB/s | 736 MB 00:21
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : avro-libs-1.7.6+cdh5.9.0+115-1.cdh5.9.0.p0.30.el6.noarch 1/62
Installing : bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6.noarch 2/62
Installing : zookeeper-3.4.5+cdh5.9.0+98-1.cdh5.9.0.p0.30.el6.x86_64 3/62
Updating : db4-4.7.25-22.el6.x86_64 4/62
Updating : 1:perl-Pod-Escapes-1.04-144.el6.x86_64 5/62
Updating : 4:perl-libs-5.10.1-144.el6.x86_64 6/62
Updating : 1:perl-Pod-Simple-3.13-144.el6.x86_64 7/62
Updating : 1:perl-Module-Pluggable-3.90-144.el6.x86_64 8/62
Updating : 3:perl-version-0.77-144.el6.x86_64 9/62
Updating : 4:perl-5.10.1-144.el6.x86_64 10/62
Updating : 1:perl-Compress-Raw-Zlib-2.021-144.el6.x86_64 11/62
Updating : perl-IO-Compress-Base-2.021-144.el6.x86_64 12/62
Updating : perl-IO-Compress-Zlib-2.021-144.el6.x86_64 13/62
Installing : perl-CGI-3.51-144.el6.x86_64 14/62
Installing : bigtop-jsvc-0.6.0+cdh5.9.0+855-1.cdh5.9.0.p0.30.el6.x86_64 15/62
Installing : bigtop-tomcat-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6.noarch 16/62
Installing : nc-1.84-24.el6.x86_64 17/62
Installing : ed-1.1-3.3.el6.x86_64 18/62
Installing : mailx-12.4-8.el6_6.x86_64 19/62
Installing : man-1.6f-39.el6.x86_64 20/62
Installing : db4-cxx-4.7.25-22.el6.x86_64 21/62
Installing : db4-devel-4.7.25-22.el6.x86_64 22/62
Installing : gdbm-devel-1.8.0-39.el6.x86_64 23/62
Installing : perl-ExtUtils-MakeMaker-6.55-144.el6.x86_64 24/62
Installing : perl-Test-Harness-3.17-144.el6.x86_64 25/62
Installing : 1:perl-ExtUtils-ParseXS-2.2003.0-144.el6.x86_64 26/62
Installing : 4:perl-devel-5.10.1-144.el6.x86_64 27/62
Installing : perl-Test-Simple-0.92-144.el6.x86_64 28/62
Installing : time-1.7-38.el6.x86_64 29/62
Installing : bc-1.06.95-1.el6.x86_64 30/62
Installing : pax-3.4-10.1.el6.x86_64 31/62
Installing : at-3.1.10-49.el6.x86_64 32/62
Installing : redhat-lsb-core-4.0-7.el6.centos.x86_64 33/62
Installing : parquet-format-2.1.0+cdh5.9.0+14-1.cdh5.9.0.p0.30.el6.noarch 34/62
Installing : parquet-1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6.noarch 35/62
Installing : hadoop-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 36/62
Installing : hadoop-hdfs-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 37/62
Installing : hadoop-yarn-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 38/62
Installing : hadoop-mapreduce-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 39/62
Installing : hadoop-0.20-mapreduce-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 40/62
Installing : hadoop-client-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 41/62
Installing : hive-jdbc-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch 42/62
Installing : hive-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch 43/62
Installing : sentry-1.5.1+cdh5.9.0+261-1.cdh5.9.0.p0.30.el6.noarch 44/62
Installing : solr-4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6.noarch 45/62
The following warning applies to any collections configured to
use Non-SolrCloud mode. Any such collection configuration will
need to be upgraded, see Upgrading Cloudera Search for details.
Installing : kite-1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6.noarch 46/62
Installing : flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch 47/62
Installing : spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch 48/62
Updating : perl-Compress-Zlib-2.021-144.el6.x86_64 49/62
Updating : db4-utils-4.7.25-22.el6.x86_64 50/62
Cleanup : perl-Compress-Zlib-2.021-141.el6_7.1.x86_64 51/62
Cleanup : perl-IO-Compress-Zlib-2.021-141.el6_7.1.x86_64 52/62
Cleanup : perl-IO-Compress-Base-2.021-141.el6_7.1.x86_64 53/62
Cleanup : 1:perl-Compress-Raw-Zlib-2.021-141.el6_7.1.x86_64 54/62
Cleanup : 1:perl-Module-Pluggable-3.90-141.el6_7.1.x86_64 55/62
Cleanup : 1:perl-Pod-Escapes-1.04-141.el6_7.1.x86_64 56/62
Cleanup : 1:perl-Pod-Simple-3.13-141.el6_7.1.x86_64 57/62
Cleanup : 3:perl-version-0.77-141.el6_7.1.x86_64 58/62
Cleanup : 4:perl-libs-5.10.1-141.el6_7.1.x86_64 59/62
Cleanup : 4:perl-5.10.1-141.el6_7.1.x86_64 60/62
Cleanup : db4-utils-4.7.25-20.el6_7.x86_64 61/62
Cleanup : db4-4.7.25-20.el6_7.x86_64 62/62
Verifying : solr-4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6.noarch 1/62
Verifying : spark-core-1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6.noarch 2/62
Verifying : zookeeper-3.4.5+cdh5.9.0+98-1.cdh5.9.0.p0.30.el6.x86_64 3/62
Verifying : perl-Compress-Zlib-2.021-144.el6.x86_64 4/62
Verifying : 4:perl-5.10.1-144.el6.x86_64 5/62
Verifying : at-3.1.10-49.el6.x86_64 6/62
Verifying : perl-ExtUtils-MakeMaker-6.55-144.el6.x86_64 7/62
Verifying : redhat-lsb-core-4.0-7.el6.centos.x86_64 8/62
Verifying : pax-3.4-10.1.el6.x86_64 9/62
Verifying : perl-Test-Harness-3.17-144.el6.x86_64 10/62
Verifying : db4-utils-4.7.25-22.el6.x86_64 11/62
Verifying : 1:perl-ExtUtils-ParseXS-2.2003.0-144.el6.x86_64 12/62
Verifying : flume-ng-1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6.noarch 13/62
Verifying : hadoop-0.20-mapreduce-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 14/62
Verifying : 4:perl-libs-5.10.1-144.el6.x86_64 15/62
Verifying : perl-CGI-3.51-144.el6.x86_64 16/62
Verifying : bc-1.06.95-1.el6.x86_64 17/62
Verifying : bigtop-jsvc-0.6.0+cdh5.9.0+855-1.cdh5.9.0.p0.30.el6.x86_64 18/62
Verifying : parquet-1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6.noarch 19/62
Verifying : hadoop-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 20/62
Verifying : hive-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch 21/62
Verifying : hadoop-mapreduce-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 22/62
Verifying : bigtop-utils-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6.noarch 23/62
Verifying : 4:perl-devel-5.10.1-144.el6.x86_64 24/62
Verifying : 1:perl-Pod-Simple-3.13-144.el6.x86_64 25/62
Verifying : 1:perl-Pod-Escapes-1.04-144.el6.x86_64 26/62
Verifying : bigtop-tomcat-0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6.noarch 27/62
Verifying : parquet-format-2.1.0+cdh5.9.0+14-1.cdh5.9.0.p0.30.el6.noarch 28/62
Verifying : time-1.7-38.el6.x86_64 29/62
Verifying : avro-libs-1.7.6+cdh5.9.0+115-1.cdh5.9.0.p0.30.el6.noarch 30/62
Verifying : 1:perl-Compress-Raw-Zlib-2.021-144.el6.x86_64 31/62
Verifying : sentry-1.5.1+cdh5.9.0+261-1.cdh5.9.0.p0.30.el6.noarch 32/62
Verifying : gdbm-devel-1.8.0-39.el6.x86_64 33/62
Verifying : db4-cxx-4.7.25-22.el6.x86_64 34/62
Verifying : man-1.6f-39.el6.x86_64 35/62
Verifying : db4-4.7.25-22.el6.x86_64 36/62
Verifying : db4-devel-4.7.25-22.el6.x86_64 37/62
Verifying : 1:perl-Module-Pluggable-3.90-144.el6.x86_64 38/62
Verifying : mailx-12.4-8.el6_6.x86_64 39/62
Verifying : 3:perl-version-0.77-144.el6.x86_64 40/62
Verifying : ed-1.1-3.3.el6.x86_64 41/62
Verifying : hive-jdbc-1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6.noarch 42/62
Verifying : hadoop-hdfs-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 43/62
Verifying : perl-IO-Compress-Zlib-2.021-144.el6.x86_64 44/62
Verifying : perl-Test-Simple-0.92-144.el6.x86_64 45/62
Verifying : kite-1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6.noarch 46/62
Verifying : hadoop-client-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 47/62
Verifying : hadoop-yarn-2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6.x86_64 48/62
Verifying : perl-IO-Compress-Base-2.021-144.el6.x86_64 49/62
Verifying : nc-1.84-24.el6.x86_64 50/62
Verifying : db4-utils-4.7.25-20.el6_7.x86_64 51/62
Verifying : perl-IO-Compress-Zlib-2.021-141.el6_7.1.x86_64 52/62
Verifying : 4:perl-5.10.1-141.el6_7.1.x86_64 53/62
Verifying : 1:perl-Compress-Raw-Zlib-2.021-141.el6_7.1.x86_64 54/62
Verifying : perl-IO-Compress-Base-2.021-141.el6_7.1.x86_64 55/62
Verifying : 1:perl-Pod-Simple-3.13-141.el6_7.1.x86_64 56/62
Verifying : 1:perl-Pod-Escapes-1.04-141.el6_7.1.x86_64 57/62
Verifying : db4-4.7.25-20.el6_7.x86_64 58/62
Verifying : 3:perl-version-0.77-141.el6_7.1.x86_64 59/62
Verifying : 4:perl-libs-5.10.1-141.el6_7.1.x86_64 60/62
Verifying : 1:perl-Module-Pluggable-3.90-141.el6_7.1.x86_64 61/62
Verifying : perl-Compress-Zlib-2.021-141.el6_7.1.x86_64 62/62 Installed:
spark-core.noarch 0:1.6.0+cdh5.9.0+229-1.cdh5.9.0.p0.30.el6 Dependency Installed:
at.x86_64 0:3.1.10-49.el6 avro-libs.noarch 0:1.7.6+cdh5.9.0+115-1.cdh5.9.0.p0.30.el6
bc.x86_64 0:1.06.95-1.el6 bigtop-jsvc.x86_64 0:0.6.0+cdh5.9.0+855-1.cdh5.9.0.p0.30.el6
bigtop-tomcat.noarch 0:0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6 bigtop-utils.noarch 0:0.7.0+cdh5.9.0+0-1.cdh5.9.0.p0.30.el6
db4-cxx.x86_64 0:4.7.25-22.el6 db4-devel.x86_64 0:4.7.25-22.el6
ed.x86_64 0:1.1-3.3.el6 flume-ng.noarch 0:1.6.0+cdh5.9.0+74-1.cdh5.9.0.p0.30.el6
gdbm-devel.x86_64 0:1.8.0-39.el6 hadoop.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6
hadoop-0.20-mapreduce.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 hadoop-client.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6
hadoop-hdfs.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 hadoop-mapreduce.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6
hadoop-yarn.x86_64 0:2.6.0+cdh5.9.0+1799-1.cdh5.9.0.p0.30.el6 hive.noarch 0:1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6
hive-jdbc.noarch 0:1.1.0+cdh5.9.0+752-1.cdh5.9.0.p0.30.el6 kite.noarch 0:1.0.0+cdh5.9.0+138-1.cdh5.9.0.p0.30.el6
mailx.x86_64 0:12.4-8.el6_6 man.x86_64 0:1.6f-39.el6
nc.x86_64 0:1.84-24.el6 parquet.noarch 0:1.5.0+cdh5.9.0+176-1.cdh5.9.0.p0.30.el6
parquet-format.noarch 0:2.1.0+cdh5.9.0+14-1.cdh5.9.0.p0.30.el6 pax.x86_64 0:3.4-10.1.el6
perl-CGI.x86_64 0:3.51-144.el6 perl-ExtUtils-MakeMaker.x86_64 0:6.55-144.el6
perl-ExtUtils-ParseXS.x86_64 1:2.2003.0-144.el6 perl-Test-Harness.x86_64 0:3.17-144.el6
perl-Test-Simple.x86_64 0:0.92-144.el6 perl-devel.x86_64 4:5.10.1-144.el6
redhat-lsb-core.x86_64 0:4.0-7.el6.centos sentry.noarch 0:1.5.1+cdh5.9.0+261-1.cdh5.9.0.p0.30.el6
solr.noarch 0:4.10.3+cdh5.9.0+458-1.cdh5.9.0.p0.30.el6 time.x86_64 0:1.7-38.el6
zookeeper.x86_64 0:3.4.5+cdh5.9.0+98-1.cdh5.9.0.p0.30.el6 Dependency Updated:
db4.x86_64 0:4.7.25-22.el6 db4-utils.x86_64 0:4.7.25-22.el6 perl.x86_64 4:5.10.1-144.el6
perl-Compress-Raw-Zlib.x86_64 1:2.021-144.el6 perl-Compress-Zlib.x86_64 0:2.021-144.el6 perl-IO-Compress-Base.x86_64 0:2.021-144.el6
perl-IO-Compress-Zlib.x86_64 0:2.021-144.el6 perl-Module-Pluggable.x86_64 1:3.90-144.el6 perl-Pod-Escapes.x86_64 1:1.04-144.el6
perl-Pod-Simple.x86_64 1:3.13-144.el6 perl-libs.x86_64 4:5.10.1-144.el6 perl-version.x86_64 3:0.77-144.el6 Complete!
[root@elephant yum.repos.d]#yum install spark-core
- 安裝 cloudera-manager
[root@elephant yum.repos.d]# yum install cloudera-manager-server
Loaded plugins: fastestmirror, refresh-packagekit
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: centos.01link.hk
* extras: centos.01link.hk
* updates: centos.01link.hk
Resolving Dependencies
--> Running transaction check
---> Package cloudera-manager-server.x86_64 0:5.9.0-1.cm590.p0.249.el6 will be installed
--> Processing Dependency: cloudera-manager-daemons = 5.9.0 for package: cloudera-manager-server-5.9.0-1.cm590.p0.249.el6.x86_64
--> Running transaction check
---> Package cloudera-manager-daemons.x86_64 0:5.9.0-1.cm590.p0.249.el6 will be installed
--> Finished Dependency Resolution Dependencies Resolved ==============================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================
Installing:
cloudera-manager-server x86_64 5.9.0-1.cm590.p0.249.el6 cloudera-manager 8.3 k
Installing for dependencies:
cloudera-manager-daemons x86_64 5.9.0-1.cm590.p0.249.el6 cloudera-manager 528 M Transaction Summary
==============================================================================================================================================================
Install 2 Package(s) Total download size: 528 M
Installed size: 679 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): cloudera-manager-daemons-5.9.0-1.cm590.p0.249.el6.x86_64.rpm | 528 MB 00:15
(2/2): cloudera-manager-server-5.9.0-1.cm590.p0.249.el6.x86_64.rpm | 8.3 kB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 34 MB/s | 528 MB 00:15
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : cloudera-manager-daemons-5.9.0-1.cm590.p0.249.el6.x86_64 1/2
Installing : cloudera-manager-server-5.9.0-1.cm590.p0.249.el6.x86_64 2/2
Verifying : cloudera-manager-server-5.9.0-1.cm590.p0.249.el6.x86_64 1/2
Verifying : cloudera-manager-daemons-5.9.0-1.cm590.p0.249.el6.x86_64 2/2 Installed:
cloudera-manager-server.x86_64 0:5.9.0-1.cm590.p0.249.el6 Dependency Installed:
cloudera-manager-daemons.x86_64 0:5.9.0-1.cm590.p0.249.el6 Complete!
[root@elephant yum.repos.d]#yum install cloudera-manager-server
- 在 monkey 主机下安装 cloudera-agent
yum install cloudera-manager-agent
cloudera-manager-agent
- 在 elephant 主机下安装 cloudera-server 和 cloudera-manager-server-db
yum install cloudera-manager-server
yum install cloudera-manager-server-db-cloudera-manager-server
- 完成后可以查看安装了什么程序
[root@elephant enmoedu]# rpm -qa | grep cloudera
cloudera-manager-server-5.9.-.cm590.p0..el6.x86_64
cloudera-manager-daemons-5.9.-.cm590.p0..el6.x86_64
cloudera-manager-server-db--5.9.-.cm590.p0..el6.x86_64
cloudera-manager-agent-5.9.-.cm590.p0..el6.x86_64 [root@monkey ~]# rpm -qa | grep cloudera
cloudera-manager-daemons-5.9.-.cm590.p0..el6.x86_64
cloudera-manager-agent-5.9.-.cm590.p0..el6.x86_64rpm -qa | grep cloudera
[root@elephant yum.repos.d]# chkconfig | grep cloudera
cloudera-scm-agent :off :off :off :on :on :on :off
cloudera-scm-server :off :off :off :on :on :on :off
cloudera-scm-server-db :off :off :off :on :on :on :offchkconfig | grep cloudera
-
安装 Java 并设置环境变量
[root@elephant x86_64]# cd /var/www/html/cloudera-manager5./RPMS/x86_64
[root@elephant x86_64]# yum install oracle-j2sdk1.
Failed to set locale, defaulting to C
Loaded plugins: fastestmirror, refresh-packagekit
Setting up Install Process
Examining oracle-j2sdk1.-1.7.+update67-.x86_64.rpm: oracle-j2sdk1.-1.7.+update67-.x86_64
Marking oracle-j2sdk1.-1.7.+update67-.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
* base: ftp.cuhk.edu.hk
* extras: ftp.cuhk.edu.hk
* updates: ftp.cuhk.edu.hk
Resolving Dependencies
--> Running transaction check
---> Package oracle-j2sdk1..x86_64 :1.7.+update67- will be installed
--> Finished Dependency Resolution Dependencies Resolved ==============================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================
Installing:
oracle-j2sdk1. x86_64 1.7.+update67- /oracle-j2sdk1.-1.7.+update67-.x86_64 M Transaction Summary
==============================================================================================================================================================
Install Package(s) Total size: M
Installed size: M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : oracle-j2sdk1.-1.7.+update67-.x86_64 /
Verifying : oracle-j2sdk1.-1.7.+update67-.x86_64 / Installed:
oracle-j2sdk1..x86_64 :1.7.+update67- Complete!
[root@elephant x86_64]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) -Bit Server VM (build 24.65-b04, mixed mode)yum install oracle-j2sdk1.7
echo -e 'export JAVA_HOME=/usr/java/default' >> /etc/profile
echo -e 'export PATH=$JAVA_HOME/bin:$PATH'>> /etc/profile
alternatives --install /usr/bin/java java /usr/java/jdk1..0_67-cloudera/bin/java
alternatives --set java /usr/java/jdk1..0_67-cloudera/bin/java
alternatives --install /usr/bin/javac javac /usr/java/jdk1..0_67-cloudera/bin/javac
alternatives --set javac /usr/java/jdk1..0_67-cloudera/bin/javac
source /etc/profile安装 java 并设置环境变量
启动 ClouderManager
启动 Cloudera DB 并登录 Clouder Manager 去安装所需要的应用软件
[root@elephant cloudera-manager5.]# service cloudera-scm-server-db status
pg_ctl: no server running
[root@elephant cloudera-manager5.]# service cloudera-scm-server-db start
The files belonging to this database system will be owned by user "cloudera-scm".
This user must also own the server process. The database cluster will be initialized with locale en_US.UTF8.
The default text search configuration will be set to "english". fixing permissions on existing directory /var/lib/cloudera-scm-server-db/data ... ok
creating subdirectories ... ok
selecting default max_connections ...
selecting default shared_buffers ... 32MB
creating configuration files ... ok
creating template1 database in /var/lib/cloudera-scm-server-db/data/base/ ... ok
initializing pg_authid ... ok
setting password ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok Success. You can now start the database server using: postgres -D /var/lib/cloudera-scm-server-db/data
or
pg_ctl -D /var/lib/cloudera-scm-server-db/data -l logfile start Adding configs
Creating SCM configuration file: /etc/cloudera-scm-server/db.properties
waiting for server to start.... done
server started
CREATE ROLE
CREATE DATABASE
Created db properties file /etc/cloudera-scm-server/db.properties
Creating DB amon for role ACTIVITYMONITOR
CREATE ROLE
CREATE DATABASE
Created DB for role ACTIVITYMONITOR
Creating DB rman for role REPORTSMANAGER
CREATE ROLE
CREATE DATABASE
Created DB for role REPORTSMANAGER
Creating DB nav for role NAVIGATOR
CREATE ROLE
CREATE DATABASE
Created DB for role NAVIGATOR
Creating DB navms for role NAVIGATORMETASERVER
CREATE ROLE
CREATE DATABASE
Created DB for role NAVIGATORMETASERVER
Enabled remote connections
waiting for server to shut down.... done
server stopped
DB initialization done.
waiting for server to start.... done
server started
[root@elephant cloudera-manager5.]#
service cloudera-scm-server-db start
[root@elephant cloudera-manager5.]# service cloudera-scm-server start
Starting cloudera-scm-server: [OK]
service cloudera-scm-server start
查看Cloudera DB 的状态
[root@elephant yum.repos.d]# service cloudera-scm-server-db status
pg_ctl: server is running (PID: )
/usr/bin/postgres "-D" "/var/lib/cloudera-scm-server-db/data"
查看Cloudera DB 的状态
登入 Cloudera Manager
- 登入 ClouderaManager 然后输入用户名和密码：admin/ admin
http://elephant:7180
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0MjgwNDA3MC0zNzkxNjY2NTYucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 选择版本
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0MzEwNjcyNi0yMDY4MDg1MDA2LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0MzE0NjU4Ni0xNzI3MDUwNjEucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 选择安装的主机名称
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0MzMwMDA4Ni0yMzc2NjU0MDQucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0MzQ1OTcyNi0xMzAxMzg1MDcucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 选择 yum 的安装源
http://elephant/cloudera-cdh5.9
http://elephant/cloudera-manager5.9![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDQvMTAwNTc5NC0yMDE3MDQyNTIyNTA0MTE0Ny0yNjc1NTk0MS5wbmc%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 安装中
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0NDIwOTI1Ny0yMTA5MTY2OC5wbmc%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0NTY1OTI3My0yMDUyNTM2MjM3LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 检查系统
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE0NTg0NzI3My0yMDI3MzEwNzYzLnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE1MDEwNzUzOS0xNjQ4NzI3NjU0LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE1MDEzMjQ3Ni0xMjI1MTYwODA2LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 检查 DB
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE1MDIxNjU3MC04MTkwODY4MDYucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 安装并启动所有服务
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNTAwMTgxNTc1Ny0xODE3MTA1MzM5LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 然后登入 Cloudera Manager
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE2MjgxNjI1Ny0zNzA3MzM4NTEucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE2Mjg0MzY0OC0xMTU4NTQ3MzE1LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
问题
- cloudera-scm-server 启动不了，请检查 clouder 的日志 /var/log/cloudera-scm-server/cloudera-scm-server.out
[root@elephant cloudera-manager5.]# service cloudera-scm-server start
Starting cloudera-scm-server: [FAILED]
[root@elephant log]# cd /var/log/cloudera-scm-server/
[root@elephant cloudera-scm-server]# ls
cloudera-scm-server.out db.log
[root@elephant cloudera-scm-server]# cat cloudera-scm-server.out
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Oracle JDK from the Oracle Java web site |
| > http://www.oracle.com/technetwork/java/javase/index.html < |
| |
| Cloudera Manager requires Java 1.6 or later. |
| NOTE: This script will find Oracle Java whether you install using |
| the binary or the RPM based installer. |
+======================================================================+/var/log/cloudera-scm-server/cloudera-scm-server.out
- check the log file: tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
[root@elephant enmoedu]# tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
-- ::, INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.mgmt.properties
-- ::, INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.mgmt.properties
-- ::, INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.mgmt.properties
-- ::, INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.mgmt.properties
-- ::, INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.properties
-- ::, INFO main:com.cloudera.server.cmf.components.EmbeddedDbManager: Reading embedded database password from /var/lib/cloudera-scm-server-db/data/generated_password.txt
-- ::, INFO main:com.cloudera.enterprise.CommonMain: Reading database properties from /etc/cloudera-scm-server/db.properties
-- ::, INFO main:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check will use thread pool of size
-- ::, INFO main:com.cloudera.cmf.cdhclient.util.CDHUrlClassLoader: Detected that this program is running in a JAVA 1.7.0_67 JVM. CDH5 jars will be loaded from:lib/cdh5
-- ::, INFO main:com.cloudera.cmf.cdhclient.CdhExecutorFactory: Adding options to bad options list:tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
- 可能会遇到 Host Service is not running 信息，此时你需要检查你有没有启动 Cloudera Manager Service
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE2MTY0MTM4Mi0xNjY4NTg3ODIyLnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE1MzU0MjUyMy0yMjEyMjg5MDQucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 启动 Host Monitor 和 Service Monitor
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE2MTkwMjk3Ni0xNDM5NTk2MjcucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- 启动 Cluster
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE1NDcxNDc1Ny0xMzI2NjY5MjY1LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
- Request to the Service Monitor failed. This may cause slow page responses. View the status of the Service Monitor.
[root@elephant cloudera-scm-server]# cd /var/log/cloudera-scm-firehose/
[root@elephant cloudera-scm-firehose]# ls
mgmt-cmf-mgmt-HOSTMONITOR-elephant.log.out mgmt-cmf-mgmt-SERVICEMONITOR-elephant.log.out stacks/var/log/cloudera-scm-firehose/
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNDE1NTU1NzgzNi02OTc3OTQ1ODMucG5n.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
https://community.cloudera.com/t5/Cloudera-Manager-Installation/Request-to-the-Host-Monitor-failed-This-may-cause-slow-page/td-p/24126 - stopping the server cloudera-scm-server
[root@elephant enmoedu]# service cloudera-scm-server stop
Stopping cloudera-scm-server: [ OK ]
[root@elephant enmoedu]# service cloudera-scm-server-db stop
waiting for server to shut down.... done
server stoppedservice cloudera-scm-server stop
-
遇上这个错误请检查是不是两个主机都启动了 httpd 服务
[root@monkey ~]# service httpd status
httpd is stopped
[root@monkey ~]# service httpd start
Starting httpd: httpd: Could not reliably determine the server's fully qualified domain name, using 172.16.201.156 for ServerName
[ OK ]
[root@monkey ~]# service httpd status
httpd (pid ) is running...service httpd status
![[补充资料] 手动搭建 Cloudera 集群](https://image.shishitao.com:8440/aHR0cHM6Ly9pbWFnZXMyMDE1LmNuYmxvZ3MuY29tL2Jsb2cvMTAwNTc5NC8yMDE3MDUvMTAwNTc5NC0yMDE3MDUwNTAwNDY1OTEzMi0xNTk0NDA0NDA1LnBuZw%3D%3D.png?w=700&webp=1 "[补充资料] 手动搭建 Cloudera 集群")
# cloudera master server
alias elephant='ssh -i "~/.ssh/key.pem" centos@xxx.xxx.xxx'
alias monkey='ssh -i "~/.ssh/key.pem" centos@xxx.xxx.xxx'
alias tiger='ssh -i "~/.ssh/key.pem" centos@xxx.xxx.xxx' scp -i ~/.ssh/key.pem training_materials.zip centos@xxx.xxx.xxx:/home/centos #AWS centos os 6
sudo yum update -y
sudo yum install vim -y
sudo yum install wget -y
sudo yum install gcc -y
sudo yum install unzip -y
echo "done" cat << EOF > ~/.vimrc
set number
syntax enable
set ts=4
set expandtab
set cursorline
set shiftwidth=4
EOF cat << EOF >> ~/.bashrc
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera/
export CLASSPATH=/usr/java/jdk1.7.0_67-cloudera/jre/
export PYTHON_HOME=/usr/src/Python-3.4.5 alias python3=python3.4
alias pip3=pip3.4
PATH=/usr/local/bin:$JAVA_HOME/bin:$PYTHON_HOME/bin:$PATH:$HOME/bin
EOF cat << EOF > /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 xxx.xxx.xxx elephant
xxx.xxx.xxx monkey
xxx.xxx.xxx tiger
EOF # update hostname
cat << EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=elephant
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTS=elephant
EOF hostname elephant # update hostname
cat << EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=monkey
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTS=monkey
EOF hostname monkey # update hostname
cat << EOF > /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=tiger
NETWORKING_IPV6=no
NOZEROCONF=yes
HOSTS=tiger
EOF hostname tiger useradd enmoedu
passwd enmoedu cat << EOF > /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these two values:
# targeted - Targeted processes are protected,
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
EOF service iptables stop
chkconfig iptables off sudo setenforce 0
sestatus # check the selinux
#update hosts file
vim /etc/ssh/sshd_config # update a few confguration to yes
RSAAuthentication yes
PubkeyAuthentication yes
PermitRootLogin yes
PasswordAuthentication yes # restart the sshd service
sudo systemctl restart sshd
sudo service sshd restart
sudo systemctl stop iptables.service ssh-keygen -f $HOME/.ssh/id_rsa -t rsa -N ''
# generate ssh key
su enmoedu # switch to enmoedu to generate the ssh key
ssh-keygen -f $HOME/.ssh/id_rsa -t rsa -N ''
exit # switch back to root
cat /home/enmoedu/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
cat /home/enmoedu/.ssh/id_rsa.pub >> /home/enmoedu/.ssh/authorized_keys
cat /root/.ssh/id_rsa.pub >> /home/enmoedu/.ssh/authorized_keys
chown enmoedu:enmoedu /home/enmoedu/.ssh/authorized_keys cat << EOF >> /etc/sudoers.d/90-cloud-init-users
root ALL=(ALL:ALL) ALL
%admin ALL=(ALL) ALL
%sudo ALL=(ALL:ALL) ALL
enmoedu ALL=(ALL)NOPASSWD:ALL
EOF cat << EOF >> /etc/sudoers
root ALL=(ALL:ALL) ALL
%admin ALL=(ALL) ALL
%sudo ALL=(ALL:ALL) ALL
enmoedu ALL=(ALL)NOPASSWD:ALL
EOF # edit the vm swswappiness
cat << EOF > /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
vm.swappiness=10
EOF cat << EOF > /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot. touch /var/lock/subsys/local
"echo never > /sys/kernel/mm/transparent_hugepage/defrag"
"echo never > /sys/kernel/mm/transparent_hugepage/enabled"
EOF cd ~/
wget http://archive.cloudera.com/cm5/installer/latest/cloudera-manager-installer.bin
chmod +x cloudera-manager-installer.bin
sudo ./cloudera-manager-installer.bin # create a enmoedu folder in hadoop
sudo -u hdfs hadoop fs -mkdir /user/enmoedu
sudo -u hdfs hadoop fs -chmod +w /user/enmoedu
sudo -u hdfs hadoop fs -chown enmoedu:enmoedu /user/enmoedu # download the python source code
cd /usr/src
wget https://www.python.org/ftp/python/3.4.5/Python-3.4.5.tgz
tar xzf Python-3.4.5.tgz
cd Python-3.4.5
./configure
make altinstall
rm Python-3.4.5.tgz
python3 -V pip3 install numpy
pip3 install matplotlib
pip3 install kazoo reboot -h now # upload the cloudera training material to Server sudo -s
sudo mkdir /home/enmoedu/src
sudo chown enmoedu:enmoedu /home/enmoedu/src
sudo mv training_materials.zip /home/enmoedu/src su enmoedu
unzip training_materials.zip
mv training_materials/ ../
sudo chown enmoedu:enmoedu training_materials.zip # Set install Mysql (switch to monkey server)
rpm -Uvh http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm
yum install mysql-community-server
/usr/bin/systemctl enable mysqld
reboot -h now # download MySQL JDBC connector
cd ~
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.43.tar.gz
tar zxvf mysql-connector-java-5.1.43.tar.gz
mkdir /usr/share/java/
cd /usr/share/java/
cp ~/mysql-connector-java-5.1.43/mysql-connector-java-5.1.43-bin.jar ./
mv mysql-connector-java-5.1.43-bin.jar mysql-connector-java.jar # Set Up Hive
mysql -u root
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
GRANT ALL ON *.* TO 'hive'@'%';
FLUSH PRIVILEGES;
EXIT; mysql -h monkey -u hive -p # Connect to Beeline
beeline -u jdbc:hive2://monkey:10000
create database testdb;
show databases;
use testdb;
show tables;
create table testdb.test_tb1(id INT, name STRING);
drop table testdb.test_tb1;
insert into testdb.test_tb1 values (1, "Alex");
AWS CentOS Notes