如何解决WARN : Unable to load native-hadoop library for your platform...

时间：2024-11-16 07:45:56

1、在启动和操作Hadoop过程中，总是会提示warning信息：

WARN : Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
如图所示

搜索的结果大部分说是操作系统与hadoop的位数不一致造成的，然而事实并非如此。

由于上述warning并不能提供更多有效的信息，需要增加debug信息：

export HADOOP_ROOT_LOGGER=DEBUG,console

再次执行./，这次debug到一些有用的信息：

16/07/05 15:33:54 DEBUG : setsid exited with exit code 0
16/07/05 15:33:54 DEBUG : field . $ with annotation @.(value=[Rate of successful kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
16/07/05 15:33:54 DEBUG : field . $ with annotation @.(value=[Rate of failed kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
16/07/05 15:33:54 DEBUG : field . $ with annotation @.(value=[GetGroups], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops)
16/07/05 15:33:54 DEBUG : UgiMetrics, User and group related metrics
16/07/05 15:33:55 DEBUG :  Creating new Groups object
16/07/05 15:33:55 DEBUG : Trying to load the custom-built native-hadoop library...
<span style="color:#FF0000;">16/07/05 15:33:55 DEBUG : Failed to load native-hadoop with error: : no hadoop in 
16/07/05 15:33:55 DEBUG : =/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib
16/07/05 15:33:55 WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable</span>
16/07/05 15:33:55 DEBUG : Falling back to shell based
16/07/05 15:33:55 DEBUG : Group mapping impl=
16/07/05 15:33:55 DEBUG : Group mapping impl=; cacheTimeout=300000; warningDeltaMs=5000
16/07/05 15:33:55 DEBUG : hadoop login
16/07/05 15:33:55 DEBUG : hadoop login commit
16/07/05 15:33:55 DEBUG : using local user:UnixPrincipal: hadoop
16/07/05 15:33:55 DEBUG : Using user: "UnixPrincipal: hadoop" with name hadoop
16/07/05 15:33:55 DEBUG : User entry: "hadoop"
16/07/05 15:33:55 DEBUG : UGI loginUser:hadoop (auth:SIMPLE)
16/07/05 15:33:55 DEBUG : PrivilegedAction as:hadoop (auth:SIMPLE) from:(:314)

我们发现，是存放在/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib/native下的
在hadoop包中搜索发现，被定义为$JAVA_LIBRARY_PATH

$ grep -R '' *
bin/yarn:  YARN_OPTS="$YARN_OPTS -=$JAVA_LIBRARY_PATH"
bin/:    set YARN_OPTS=%YARN_OPTS% -=%JAVA_LIBRARY_PATH%
etc/hadoop/:  YARN_OPTS="$YARN_OPTS -=$JAVA_LIBRARY_PATH"
etc/hadoop/:  set YARN_OPTS=%YARN_OPTS% -=%JAVA_LIBRARY_PATH%
libexec/:# setup '' for native-hadoop code if necessary
libexec/:  HADOOP_OPTS="$HADOOP_OPTS -=$JAVA_LIBRARY_PATH"
libexec/:@rem setup '' for native hadoop code if necessary
libexec/:  set HADOOP_OPTS=%HADOOP_OPTS% -=%JAVA_LIBRARY_PATH%
share/doc/hadoop/common/:    HADOOP-8756. Fix SEGV when libsnappy is in  but
share/doc/hadoop/common/:    HADOOP-1660. Add the cwd of the map/reduce task to the 
share/doc/hadoop/common/: 50. HADOOP-1493.  Permit specification of "" system
share/doc/hadoop/common/:10. HADOOP-873.      Pass  correctly to child processes.
share/doc/hadoop/common/:42. HADOOP-838.  Fix tasktracker to pass  to
share/doc/hadoop/mapreduce/:    MAPREDUCE-4458. Warn if  is used for AM or Task
share/doc/hadoop/mapreduce/:    MAPREDUCE-4072. User set  seems to overwrite default
share/doc/hadoop/mapreduce/:    MAPREDUCE-3259. Added  of NodeManager to

但是此时发现环境变量 J A V A L I B R A R Y P A T H 并未定义，那么问题到这里终于算是找到了，解决方法也很简单了， v i m   / . b a s h r c ，添加 JAVA_LIBRARY_PATH并未定义，那么问题到这里终于算是找到了，解决方法也很简单了，vim ~/.bashrc，添加 JAVAL​IBRARYP​ATH并未定义，那么问题到这里终于算是找到了，解决方法也很简单了，vim /.bashrc，添加JAVA_LIBRARY_PATH环境变量

export JAVA_LIBRARY_PATH=‘/usr/local/sinasrv2/hadoop/hadoop-2.6.4/lib/native’
如图所示

这里还要特别说明的是，一定要注意自己的Hadoop以及JDK版本型号，配置环境变量的时候不能有丝毫差错！
还有一点就是有些人在/etc/profile文件中配置变量，那就要在该文件中进行修改，本人是在~/.bashrc文件中进行修改的。
source ~/.bashrc 使之生效。
再执行，则不会出现上述warning信息
如图所示
希望对大家有所帮助，此文是本人在借鉴前人的经验上总结出来的，我们热爱开源，于是将此文放到网上，谢谢！
大数据相关视频请访问B站个人空间

相关文章