Hadoop运维:作业的map过多被杀死

时间:2021-06-18 21:53:21

这几天因为数据源更改的原因,每天一个任务运行的作业数是8w多个,每当运行到4w多的时候,整个任务就被kill掉了,追踪日志得到原因。resourceManager的配置yarn.app.mapreduce.am.resource.mb设置为1GB,造成内存溢出,而导致作业被杀死。将参数修改为3GB。

附作业错误log:

Job Overview
Job Name: insert overwrite tabl...og_raw_locate_newtmp(Stage-1)
User Name: root
Queue: root.base
State: KILLED
Uberized: false
Submitted: Tue Nov 11 10:20:33 CST 2014
Started: Tue Nov 11 11:45:19 CST 2014
Finished: Tue Nov 11 12:53:05 CST 2014
Elapsed: 1hrs, 7mins, 46sec
Diagnostics:  
Average Map Time 34sec