搭建Hadoop2.6+Hbase0.98.6+Nutch2.3环境

**

编译Nutch并抓取网页

**
1、修改ivy/ivy.xml

<dependency org="org.apache.gora" name="gora-core" rev="0.6" conf="*->default"/>

<!---->//取消该注释
<dependency org="org.apache.gora" name="gora-hbase" rev="0.6" conf="*->default" />
<dependency org="org.apache.gora" name="gora-compiler-cli" rev="0.6" conf="*->default"/>
<dependency org="org.apache.gora" name="gora-compiler" rev="0.6" conf="*->default"/>
   将hadoop1.2相关的去掉，然后添加：

<dependency org="org.apache.hadoop" name="hadoop-client" rev="2.6.0" conf="*->default"/>

2、修改ivysetting.xml
编译时部分jar包不能下载，需要修改如下配置。

<property name="repository.apache.org" value="http://maven.restlet.org/" override="false"/>

3、修改nutch-site.xml

<configuration>
<property>  
<name>storage.data.store.class</name>  
<value>org.apache.gora.hbase.store.HBaseStore</value>  
<description>Default class for storing data</description>  
</property>  
<property>  
<name>http.agent.name</name>  
<value>My Nutch Spider</value>  
</property> 

<property>
  <name>io.serializations</name>
  <value>org.apache.hadoop.io.serializer.WritableSerialization</value>
  <description>A list of serialization classes that can be used for
  obtaining serializers and deserializers.</description>
</property>
</configuration>

4、修改gora.properties

gora.datastore.default=org.apache.gora.hbase.store.HBaseStore

5、编译

ant runtime

6、添加缺失jar包
拷贝commons-compress-1.9.jar，vorbis-java-core-0.6.jar，nekohtml-1.9.13.jar到/runtime/local/lib/下

7、创建seek文件
在/runtime/local/下创建urls文件夹，并创建seek.txt，内容为：
http://nutch.apache.org/

8、上传urls文件夹到HDFS

hadoop fs -copyFromLocal urls .

9、抓取网页

 bin/crawl seed crawl 5

10、查看HBase数据库

scan 'crawl_webpage'

常见问题

1、**org.apache.avro.io.BinaryDecoder.ensureBounds(BinaryDecoder.java:473)问题
nutch-site.xml新增配置**

<property>
  <name>io.serializations</name>
  <value>org.apache.hadoop.io.serializer.WritableSerialization</value>
  <description>A list of serialization classes that can be used for
  obtaining serializers and deserializers.</description>
</property>