cygwin下安装hadoop0.20

时间:2021-07-22 01:26:13

安装时选择软件包:

editor-两个VIM、base-SED、Net-OPENSSH、OPENSSL、libs-libintl3和libintl8

环境变量:

path=%JAVA_HOME%\bin;%JAVA_HOME%\jre6\bin;D:\Program Files\MySQL\MySQL Server 5.5\bin;D:\Program Files\Python2.;D:\Program Files\MongoDB\bin;D:\Program Files\TortoiseSVN\bin;%MAVEN_HOME%\bin;D:\cygwin64\bin;D:\cygwin64\usr\bin

CYGWIN=ntsec tty

然后开始安装,安装完开始启动sshd服务:

进入终端执行:

tree@treePC ~
$ ssh-host-config *** Query: Overwrite existing /etc/ssh_config file? (yes/no) yes
*** Info: Creating default /etc/ssh_config file
*** Query: Overwrite existing /etc/sshd_config file? (yes/no) yes
*** Info: Creating default /etc/sshd_config file
*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.
*** Query: Should privilege separation be used? (yes/no) no
*** Info: Updating /etc/sshd_config file *** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: [] ntsec
*** Info: On Windows Server , Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires. You need to have or to create a privileged
*** Info: account. This script will help you do so. *** Info: You appear to be running Windows XP 64bit, Windows Server,
*** Info: or later. On these systems, it's not possible to use the LocalSystem
*** Info: account for services that can change the user id without an
*** Info: explicit password (such as passwordless logins [e.g. public key
*** Info: authentication] via sshd). *** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless a similar account
*** Info: already exists). This account is then used to run these special
*** Info: servers. *** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself. *** Info: No privileged account could be found. *** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.
*** Query: Do you want to use a different name? (yes/no) no
*** Query: Create new privileged user account 'cyg_server'? (yes/no) yes
*** Info: Please enter a password for new user cyg_server. Please be sure
*** Info: that this password matches the password rules given on your system.
*** Info: Entering no password will exit the configuration.
*** Query: Please enter the password:
*** Query: Reenter: *** Info: User 'cyg_server' has been created with password 'lee**jan'.
*** Info: If you change the password, please remember also to change the
*** Info: password for the installed services which use (or will soon use)
*** Info: the 'cyg_server' account. *** Info: Also keep in mind that the user 'cyg_server' needs read permissions
*** Info: on all users' relevant files for the services running as 'cyg_server'.
*** Info: In particular, for the sshd server all users' .ssh/authorized_keys
*** Info: files must have appropriate permissions to allow public key
*** Info: authentication. (Re-)running ssh-user-config for each user will set
*** Info: these permissions correctly. [Similar restrictions apply, for
*** Info: instance, for .rhosts files if the rshd server is running, etc]. *** Info: The sshd service has been installed under the 'cyg_server'
*** Info: account. To start the service now, call `net start sshd' or
*** Info: `cygrunsrv -S sshd'. Otherwise, it will start automatically
*** Info: after the next reboot. *** Info: Host configuration finished. Have fun!

OK,启动SSH:

cygwin下安装hadoop0.20

配置无密码SSH:

在终端执行

tree@treePC ~/.ssh
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tree/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tree/.ssh/id_rsa.
Your public key has been saved in /home/tree/.ssh/id_rsa.pub.
The key fingerprint is:
a0:4c:cd::e4::6f:c4::::eb:d6::c9: tree@treePC
The key's randomart image is:
+--[ RSA ]----+
| .+= |
| .B + |
| +.O o . |
| o.+ = E |
| .o. So . |
| o ... + |
| . . . |
| |
| |
+-----------------+

将公钥加入公钥授权文件中:

tree@treePC ~/.ssh
$ pwd
/home/tree/.ssh tree@treePC ~/.ssh
$ ls
id_rsa id_rsa.pub tree@treePC ~/.ssh
$ cp id_rsa.pub authorized_keys tree@treePC ~/.ssh
$ ls
authorized_keys id_rsa id_rsa.pub

弄完之后重启cygWin,ssh localhost试试能不能连接。

如果报错,重启sshd服务。然后重启cygWin试试。

创建连接,将C盘的jdk链接进来:

tree@treePC ~
$ ln -s "/cygdrive/c/Program Files/Java/jdk1.6.0_43" "/usr/local/jdk"

配置文件:

hadoop-env.sh文件加上JDK

# The java implementation to use.  Required.
export JAVA_HOME=/usr/local/jdk

core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop-0.20.2/temp</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

hdfs-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>dfs.name.dir</name>
<value>/hadoop-0.20.2/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/hadoop-0.20.2/hdfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Put site-specific property overrides in this file. --> <configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/hadoop-0.20.2/mapred</value>
</property>
</configuration>

配置完之后,格式化namenode:

tree@treePC /hadoop-0.20./bin
$ ./hadoop namenode -format
// :: INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = treePC/192.168.0.157
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
// :: INFO namenode.FSNamesystem: fsOwner=treepc\tree,None,root,Administrators,Users,HomeUsers,ora_dba
// :: INFO namenode.FSNamesystem: supergroup=supergroup
// :: INFO namenode.FSNamesystem: isPermissionEnabled=true
// :: INFO common.Storage: Image file of size saved in seconds.
// :: INFO common.Storage: Storage directory \hadoop-0.20.\hdfs\name has been successfully formatted.
// :: INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at treePC/192.168.0.157
************************************************************/

启动:

tree@treePC /hadoop-0.20./bin
$ ./start-all.sh
starting namenode, logging to /hadoop-0.20./bin/../logs/hadoop-tree-namenode-treePC.out
localhost: starting datanode, logging to /hadoop-0.20./bin/../logs/hadoop-tree-datanode-treePC.out
localhost: starting secondarynamenode, logging to /hadoop-0.20./bin/../logs/hadoop-tree-secondarynamenode-treePC.out
starting jobtracker, logging to /hadoop-0.20./bin/../logs/hadoop-tree-jobtracker-treePC.out
localhost: starting tasktracker, logging to /hadoop-0.20./bin/../logs/hadoop-tree-tasktracker-treePC.out

运行一个圆周率DEMO:

tree@treePC /hadoop-0.20./bin
$ ./hadoop jar ../hadoop-0.20.-examples.jar pi
Number of Maps =
Samples per Map =
Wrote input for Map #
Wrote input for Map #
Starting Job
// :: INFO mapred.FileInputFormat: Total input paths to process :
// :: INFO mapred.JobClient: Running job: job_201402281919_0001
// :: INFO mapred.JobClient: map % reduce %
// :: INFO mapred.JobClient: map % reduce %
// :: INFO mapred.JobClient: map % reduce %
// :: INFO mapred.JobClient: Job complete: job_201402281919_0001
// :: INFO mapred.JobClient: Counters:
// :: INFO mapred.JobClient: Job Counters
// :: INFO mapred.JobClient: Launched reduce tasks=
// :: INFO mapred.JobClient: Launched map tasks=
// :: INFO mapred.JobClient: Data-local map tasks=
// :: INFO mapred.JobClient: FileSystemCounters
// :: INFO mapred.JobClient: FILE_BYTES_READ=
// :: INFO mapred.JobClient: HDFS_BYTES_READ=
// :: INFO mapred.JobClient: FILE_BYTES_WRITTEN=
// :: INFO mapred.JobClient: HDFS_BYTES_WRITTEN=
// :: INFO mapred.JobClient: Map-Reduce Framework
// :: INFO mapred.JobClient: Reduce input groups=
// :: INFO mapred.JobClient: Combine output records=
// :: INFO mapred.JobClient: Map input records=
// :: INFO mapred.JobClient: Reduce shuffle bytes=
// :: INFO mapred.JobClient: Reduce output records=
// :: INFO mapred.JobClient: Spilled Records=
// :: INFO mapred.JobClient: Map output bytes=
// :: INFO mapred.JobClient: Map input bytes=
// :: INFO mapred.JobClient: Combine input records=
// :: INFO mapred.JobClient: Map output records=
// :: INFO mapred.JobClient: Reduce input records=
Job Finished in 24.927 seconds
Estimated value of Pi is 3.80000000000000000000

上浏览器看看:

mapreduce:

http://localhost:50030

cygwin下安装hadoop0.20

HDFS:

http://localhost:50070

cygwin下安装hadoop0.20

OK,运行成功就证明安装成功了。