文件名称：Hadoop入门实战手册
文件大小：881KB
文件格式：PDF
更新时间：2017-09-11 08:43:41
Hadoop 入门 实战 手册
目录 1 ...........................................................................................................................4 概述 1.1 ..................................................................................................4 什么是Hadoop？ 1.2 .......................................................................................4 为什么要选择Hadoop？ 1.2.1 ........................................................................................................4 系统特点 1.2.2 ........................................................................................................5 使用场景 2 ...........................................................................................................................5 术语 3 ....................................................................................................6 Hadoop的单机部署 3.1 .....................................................................................................................6 目的 3.2 ..............................................................................................................6 先决条件 3.2.1 ........................................................................................................6 支持平台 3.2.2 ........................................................................................................6 所需软件 3.2.3 ........................................................................................................6 安装软件 3.3 .....................................................................................................................7 下载 3.4 ................................................................................7 运行Hadoop集群的准备工作 3.5 ............................................................................................7 单机模式的操作方法 3.6 .....................................................................................8 伪分布式模式的操作方法 3.6.1 ................................................................................................................8 配置 3.6.2 ................................................................................................9 免密码ssh设置 3.6.3 ................................................................................................................9 执行 4 .......................................................................................11 Hadoop集群搭建过程手记 4.1 .................................................................................................12 免密码SSH设置 4.2 ................................................................................................12 Hadoop软件安装 4.3 ..................................................................................................13 Master(85)配置 4.4 .........................................................................................14 Slave(60,245上)配置 4.5 ..................................................................................15 初始化和启动hadoop集群 4.5.1 ............................................................................................15 初始化文件系统 4.5.2 .................................................................................................15 启动Hadoop 4.5.3 .................................................................................................17 停止Hadoop 4.6 ...................................................................................................................17 测试 4.7 .................................................................................................19 管理界面与命令 4.7.1 ........................................................................................19 hdfs运行状态界面 4.7.2 .........................................................................20 Map-reduce的运行状态界面 4.7.3 ........................................................................................20 直接的命令行查看 4.7.1 ............................................................................................21 运行的进程查看 5 ..................................................................................................................22 架构分析 5.1 .................................................................................................................22 HDFS 5.1.1 ..................................................................................23 HDFS的三个重要角色 5.1.2 .............................................................................................24 HDFS设计特点 5.2 .......................................................................................................25 MapReduce www.linuxidc.com Linux公社(LinuxIDC.com) 是包括Ubuntu,Fedora,SUSE技术，最新IT资讯等Linux专业类网站。 5.2.1 ......................................................................................................25 算法介绍 5.2.2 ........................................................................27 Hadoop框架下的mapreduce 5.3 .....................................................................................................28 综合架构分析 6 .........................................................................................................37 Hadoop的应用 7 ..................................................................................................................38 系统维护 7.1 ............................................................................................38 Hadoop的系统监控 7.2 Hadoop中的命令（Command）总结.....................................错误！未定义书签。 7.3 ..............................................................38 NameNode与JobTracker单点故障说明 7.4 ............................................................................................................39 经验总结 7.5 .......................................39 如何在一个hadoop集群新增或删除一些机器而不重启 7.5.1 ......................................................................................................39 新增节点 7.5.2 ......................................................................................................40 删除节点 7.6 ..............................................................................................42 其它日常问题说明 7.6.1 ..............42 datanode启动失败，各slave节点的namespaceIDs与masters不同 7.6.2 .............................................................43 taskTracker和jobTracker 启动失败 7.6.3 ...43 Shuffle Error: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out 7.6.4 ...............................................................................44 Too many fetch-failures 7.6.5 ..................................44 能够启动datanode，但无法访问，也无法结束的错误 7.6.6 ................................................44 java.io.IOException: Could not obtain block: 7.6.7 ..........................................44 java.lang.OutOfMemoryError: Java heap space 7.6.8 ........................................................45 解决hadoop OutOfMemoryError问题： 7.6.9 .......................................................................45 Hadoop java.io.IOException: 7.7 .......................................................................................45 防火墙的端口开放要求 7.7.1 ....................................................................45 与HDFS有关的地址及端口属性 7.7.2 .........................................................46 与MapReduce 有关的地址及端口属性 8 .........................................................................................................................47 附录 8.1 .......................................................................................................47 hadoop历史 8.2 ...................................................................................................49 Hadoop大记事 8.3 .................................................................................49 Hadoop的几个主要子项目 8.4 ..............................................................................................50 官方集群搭建参考 8.4.1 ......................................................................................................50 配置文件 8.4.2 ...............................................................................................50