IDEA远程连接Hadoop

时间：2021-07-29 10:53:37

IDEA远程连接Hadoop

Win

1.Hadoop配置

下载并配置到本地环境

HADOOP_HOME D:\Tools\hadoop-2.7.7\hadoop-2.7.7

HADOOP_PREFIX   D:\Tools\hadoop-2.7.7\hadoop-2.7.7

PATH += %HADOOP_HOME%\bin

2.Maven pom.xml

<dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-common</artifactId>

    <version>2.7.7</version>

</dependency>

<dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-hdfs</artifactId>

    <version>2.7.7</version>

</dependency>

<dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-mapreduce-client-core</artifactId>

    <version>2.7.7</version>

</dependency>

<dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>

    <version>2.7.7</version>

</dependency>

<dependency>

    <groupId>org.apache.hadoop</groupId>

    <artifactId>hadoop-mapreduce-client-common</artifactId>

    <version>2.7.7</version>

</dependency>

3.core-site.xml

hadoop集群里的core-site.xml文件拿下来，放到resource目录下（记在etc/hosts里的名字记得换成ip地址）

4.替换

windows下开发的可执行文件肥肠重要，就是\(HADOOP_HOME\)/bin下的windows可执行文件或者dll，下载下来的覆盖掉bin下的目录下即可

(hadoop.dll winutils.exe winutils.pdb......balabala)(版本版本！！！)

5.码代码

String filePath = "hdfs://172.18.145.167//user//zwj//wordCountTest.txt";

Configuration conf = new Configuration();

FileSystem fs = FileSystem.get(URI.create(filePath),conf);

InputStream in = null;

try

{

    in = fs.open(new Path(filePath));

    IOUtils.copyBytes(in,System.out,4096,false);

}finally {

    IOUtils.closeStream(in);

}

完全ojbk

问题

1.org.apache.hadoop.hdfs.BlckMissingException

子节点datanode崩了，去log里找问题

2.Exception in thread "main" java.lang.UnsatisfiedLinkError:...nativeIO$Wi.....

win下可执行文件版本不对

3.hadoop Permission denied : USER=ZWJ,access=WRITE......

windows系统的用户名和ubuntu上的用户名不一致，炸

hadoop fs -chmod 777 filePath（filePath下文件所有者、群组用户、其他用户均可读可写可执行OTZ）

相关文章