hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

时间:2021-06-27 09:14:17

 

   

  问题详情

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

 

 

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

 

 

 

 

 

 

 

 解决办法

[hadoop@master ~]$ hadoop job -kill job_1493782088693_0001
DEPRECATED: Use of
this script to execute mapred command is deprecated.
Instead use the mapred command
for it.

17/05/03 11:48:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/03 11:48:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/03 11:48:29 INFO impl.YarnClientImpl: Killed application application_1493782088693_0001
Killed job job_1493782088693_0001






[hadoop@master
~]$ hadoop job -kill job_1493782088693_0002
DEPRECATED: Use of
this script to execute mapred command is deprecated.
Instead use the mapred command
for it.

17/05/03 11:48:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/03 11:48:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/03 11:48:29 INFO impl.YarnClientImpl: Killed application application_1493782088693_0001
Killed job job_1493782088693_0002






[hadoop@master
~]$ hadoop job -kill job_1493782088693_0003
DEPRECATED: Use of
this script to execute mapred command is deprecated.
Instead use the mapred command
for it.

17/05/03 11:48:25 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/03 11:48:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/03 11:48:29 INFO impl.YarnClientImpl: Killed application application_1493782088693_0001
Killed job job_1493782088693_0003

 

 

  有时候上述这样kill做下来,并不管用,得再来

[hadoop@master ~]$ yarn application -kill application_1493782088693_0001
17/05/03 14:45:23 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/03 14:45:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Application application_1493782088693_0001 has already finished
[hadoop@master
~]$ yarn application -kill application_1493782088693_0002
17/05/03 14:45:59 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/03 14:45:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Application application_1493782088693_0002 has already finished
[hadoop@master
~]$ yarn application -kill application_1493782088693_0003
17/05/03 14:46:47 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/05/03 14:46:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Application application_1493782088693_0003 has already finished
[hadoop@master
~]$

 

 

 

 

 

 

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

[hadoop@master ~]$ hadoop job -list
DEPRECATED: Use of
this script to execute mapred command is deprecated.
Instead use the mapred command
for it.

17/05/03 14:39:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/05/03 14:39:24 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
Total jobs:
0
JobId State StartTime UserName Queue Priority UsedContainers RsvdContainers UsedMem RsvdMem NeededMem AM info
[hadoop@master
~]$

 

 

 

 

 

  同时,大家要注意,是否是进程的消失?

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

  也会是你的slave1 还是 slave2的进程自动消失了。注意 ,这是个很隐蔽的问题。

   重新停止集群,再重新启动集群。

  如果还是出现这个问题的话,则

MapReduce任务运行到running job卡住

 

 

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

 

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

 

 

  关于这里参数配置的解释,见

Hadoop YARN配置参数剖析(2)—权限与日志聚集相关参数

 

 

 注意,对master、slave1和slave2都要操作,然后,再[hadoop@master hadoop-2.6.0]$ sbin/stop-all.sh

  再[hadoop@master hadoop-2.6.0]$ sbin/start-all.sh即可。

 

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

 

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

hadoop job -kill 与 yarn application -kii(作业卡了或作业重复提交或MapReduce任务运行到running job卡住)

 

   成功!