Kafka：ZK+Kafka+Spark Streaming集群环境搭建（七）针对hadoop2.9.0启动DataManager失败问题

时间：2023-03-08 17:17:56
Kafka：ZK+Kafka+Spark Streaming集群环境搭建（七）针对hadoop2.9.0启动DataManager失败问题

DataManager启动失败

启动过程中发现一个问题：slave1,slave2,slave3都是只启动了DataNode，而DataManager并没有启动：

[spark@slave1 hadoop-2.9.]$ jps

 Jps

 DataNode

[spark@slave2 hadoop-2.9.]$ jps

 Jps

 DataNode

[spark@slave3 hadoop-2.9.]$ jps

 Jps

 DataNode

这里一个错误原因可以从： slaves各自的nodemanager日志查看。

查看slave1虚拟机的/opt/hadoop-2.9.0/logs/yarn-spark-nodemanager-slave1.log，错误信息为：

-- ::, ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Unexpected error starting NodeStatusUpdater

org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager, Registration of NodeManager failed, Message from ResourceManager: NodeManager from  slave1 doesn't sa

tisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager.

        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:)

        at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:)

        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:)

        at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:)

        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:)

        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:)

解决问题：

错误原因是hadoop dfs启动内存默认为1024G，而我的配置yarn-site.xml中分配内存资源为512，导致内存不足。

[spark@master hadoop-2.9.]$ grep "HEAP" /opt/hadoop-2.9./etc/hadoop/yarn-env.sh

JAVA_HEAP_MAX=-Xmx1000m

# For setting YARN specific HEAP sizes please use this

# YARN_HEAPSIZE=

if [ "$YARN_HEAPSIZE" != "" ]; then

  JAVA_HEAP_MAX="-Xmx""$YARN_HEAPSIZE""m"

# If not specified, the default value will be picked from either YARN_HEAPMAX

# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_RESOURCEMANAGER_HEAPSIZE=

# If not specified, the default value will be picked from either YARN_HEAPMAX

# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_TIMELINESERVER_HEAPSIZE=

# If not specified, the default value will be picked from either YARN_HEAPMAX

# or JAVA_HEAP_MAX with YARN_HEAPMAX as the preferred option of the two.

#export YARN_NODEMANAGER_HEAPSIZE=

解决方案，修改yarn-site.xml中的配置项：

    <property>

        <name>yarn.nodemanager.resource.memory-mb</name>

        <value></value>

    </property>

备注：之前“yarn.nodemanager.resource.memory-mb”配置的是512，修改为2048或者更大，集群环境中肯能要上2G~8G左右。

重新启动hadoop，在slaves节点上验证是否slave启动成功：

[spark@slave1 hadoop-2.9.]$ jps

 DataNode

 NodeManager

 Jps

[spark@slave1 hadoop-2.9.]$