在mac上安装hadoop伪分布式

时间：2022-09-21 16:13:56

换了macbook pro之后，要重新安装hadoop，但是mac上的jdk跟windows上的不同，导致折腾了挺久的，现在分享出来，希望对大家有用。

一：下载jdk

选择最新版本下载，地址：http://www.oracle.com/technetwork/java/javase/downloads/index.html

安装完成之后，打开终端，输入java -version ,出现类似如下说明安装成功。

java version "1.8.0"

Java(TM) SE Runtime Environment (build 1.8.0-b132)

Java HotSpot(TM) 64-Bit Server VM (build 25.0-b70, mixed mode)

二：配置hadoop

下载hadoop，自己可到官网下载稳定版本，不用下载最新的，因为最新的可能不稳定。

配置hadoop 里面conf文件夹四个文件（hadoop-env.sh,core-site.xml,mapred-site.xml,hdfs-site.xml）

下载完hadoop之后，把它解压到你想存放的文件夹，然后进入hadoop的conf目录

1.配置hadopp-env.sh

打开该文件之后找到

＃export JAVA_HOME=

＃export HADOOP_HEAPSIZE=2000

＃export HADOOP_OPTS=-server

修改为：

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_79.jdk/Contents/Home

export HADOOP_HEAPSIZE=2000

#export HADOOP_OPTS=-server

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

即去掉前面的解释符＃

特别注意：

JAVA_HOME=的配置是类似上面的目录，百度上有些文章写到是在终端输入whereis java出现的目录，这是不对的。

因为mac的jdk安装在根目录的Library文件夹下面。

2.配置core-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

   <name>hadoop.tmp.dir</name>

   <value>/Users/username/Documents/apache/hadoop-2.6.0/hadoop_tmp</value>

   <description>A base for other temporary directories.</description>

 </property>

<property>

   <name>fs.default.name</name>

   <value>hdfs://localhost:9000</value>

</property>

<property>

    <name>mapred.job.tracker</name>

    <value>hdfs://localhost:9001</value>

</property>

<property>

    <name>dfs.replication</name>

    <value>1</value>

</property>

</configuration>

3.配置mapred-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>mapred.job.tracker</name>

<value>localhost:9001</value>

</property>

</configuration>

4.配置hdfs-site.xml

<?xml version="1.0"?>

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>

<name>dfs.replication</name>

<value>1</value>

</property>

</configuration>

三：配置ssh，无密码登录

1.mac 上已经ssh了，在终端输入ssh-keygen -t rsa命令，碰到需要输入密码的直接按enter健即可。出现如下成功

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /Users/jia/.ssh/id_rsa.

Your public key has been saved in /Users/jia/.ssh/id_rsa.pub.

The key fingerprint is:

d4:85:aa:83:ae:db:50:48:0c:5b:dd:80:bb:fa:26:a7 jia@JIAS-MacBook-Pro.local

The key's randomart image is:

+--[ RSA 2048]----+

|. .o.o     ..    |

| =. . .  ...     |

|. o.    ...      |

| ...   ..        |

|  .... .S        |

|  ... o          |

| ...   .         |

|o oo.            |

|E*+o.            |

+-----------------+

在终端输入cd .ssh 进入.ssh目录，输入命令。

cp id_rsa.pub authorized_keys

即可。

2.在mac上输入

ssh localhost

如果出现

ssh: connect to host localhost port : Connection refused

表示当前用户没有权限。这个可能是系统为安全考虑，默认设置的。

更改设置如下：进入system preference --> sharing --> 勾选remote login，并设置allow access for all users。再次输入“ssh localhost"，再输入密码并确认之后，可以看到ssh成功

如果出现

No such file or directory...

1.启动sshd服务：
sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

2.查看服务的启动，筛选：
sudo launchctl list | grep ssh

四：配置环境变量

suodo vim /etc/profile

添加：

export HADOOP_HOME=/Users/username/Documents/apache/hadoop-2.6.0

export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

五：启动hadoop

1.进入hadoop文件夹，用如下命令格式化：

bin/hadoop namenode -format

出现如下，说明成功

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   host = JIAS-MacBook-Pro.local/192.168.1.3

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 0.20.2

STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010

************************************************************/

Re-format filesystem in /tmp/hadoop-jia/dfs/name ? (Y or N) Y

14/07/14 13:55:17 INFO namenode.FSNamesystem: fsOwner=jia,staff,everyone,localaccounts,_appserverusr,admin,_appserveradm,_lpadmin,com.apple.sharepoint.group.1,_appstore,_lpoperator,_developer,com.apple.access_screensharing,com.apple.access_ssh

14/07/14 13:55:17 INFO namenode.FSNamesystem: supergroup=supergroup

14/07/14 13:55:17 INFO namenode.FSNamesystem: isPermissionEnabled=true

14/07/14 13:55:17 INFO common.Storage: Image file of size 93 saved in 0 seconds.

14/07/14 13:55:17 INFO common.Storage: Storage directory /tmp/hadoop-jia/dfs/name has been successfully formatted.

14/07/14 13:55:17 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at JIAS-MacBook-Pro.local/192.168.1.3

************************************************************/

2.启动hadoop守护进程

bin/start-all.sh

3.停止hadoop守护进程

bin/stop-all.sh

六：错误分析

1.之前运行时总出现如下错误

cat: Documents/hadoop-0.20./conf/slaves: No such file or directory

cat: Documents/hadoop-0.20./conf/masters: No such file or directory

其实这是在配置mahout的环境时加上了下面这句环境变量导致的错误

export HADOOP_CONF_DIR=Documents/hadoop-0.20./conf

之前我电脑的环境变量为(在终端输入命令:open /etc/  , 找的profile文件，打开，配置好环境变量在覆盖之前的）

export HADOOP_HOME=Documents/hadoop-0.20.

export MAHOUT_HOME=Documents/hadoop-0.20./mahout-distribution-0.9

export MAVEN_HOME=Documents/apache-maven-3.2.

export PATH=$PATH:$HADOOP_HOME/bin:$MAHOUT_HOME/bin:$MAVEN_HOME/bin

export HADOOP_CONF_DIR=Documents/hadoop-0.20./conf

export MAHOUT_CONF_DIR=Documents/hadoop-0.20./mahout-distribution-0.9/conf

export classpath=$classpath:$MAHOUT_HOME/lib:$HADOOP_CONF_DIR:$MAHOUT_CONF_DIR

把环境配置改为如下就可以正常运行了

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0.jdk/Contents/Home
export HADOOP_HOME=Documents/hadoop-0.20.0
export MAHOUT_HOME=Documents/hadoop-0.20.2/mahout-distribution-0.9
export MAVEN_HOME=Documents/apache-maven-3.2.2

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$MAVEN_HOME/bin:$MAHOUT_HOME/bin

export MAHOUT_CONF_DIR=Documents/hadoop-0.20.2/mahout-distribution-0.9/conf

export classpath=$classpath:$JAVA_HOME/lib:$MAHOUT_HOME/lib:$MAHOUT_CONF_DIR

红色部分做好加进去

既去掉

export HADOOP_CONF_DIR=Documents/hadoop-0.20./conf

注意：

修改/etc/profile上的环境配置文件之后，用如下命令使其生效

source /etc/profile

2.如果需要自己编译hadoop源码的话，编译过程可能会出现org.apache.maven.plugin.MojoExecutionException的错误

 Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.6.2:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)

    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)

    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)

    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)

    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)

    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)

    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)

    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)

    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)

    at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

    at java.lang.reflect.Method.invoke(Method.java:606)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)

    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)

    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)

    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

Caused by: org.apache.maven.plugin.MojoExecutionException: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version

    at org.apache.hadoop.maven.plugin.protoc.ProtocMojo.execute(ProtocMojo.java:105)

    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)

    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)

    ... 20 more

Caused by: org.apache.maven.plugin.MojoExecutionException: 'protoc --version' did not return a version

    at org.apache.hadoop.maven.plugin.protoc.ProtocMojo.execute(ProtocMojo.java:68)

    ... 22 more

下载protobuf-2.4.1.jar

http://vdisk.weibo.com/s/tYlk6JrNUYTY

安装protobuf

tar xzf protobuf-2.4.1.tar.gz

cd protobuf-2.4.1

./configure

make

sudo make install

sudo ldconfig

在重新编译

 3.namenode没启动

解决方法：先可视化之后再启动

bin/hadoop namenode -format

相关文章