如下创建主类程序
public class JobMain extends Configured implements Tool {
@Override
public int run(String[] args) throws Exception {
//1:创建job对象
Job job = Job.getInstance(super.getConf(), "mapreduce_sort");
...
}
结果单机上跑没有问题,但是放在集群上就会出问题
报如下错误
20/12/19 19:03:26 INFO : Connecting to ResourceManager at node01/192.168.177.101:8032
20/12/19 19:03:26 WARN : No job jar file set. User classes may not be found. See Job or Job#setJar(String).
20/12/19 19:03:26 INFO : Total input paths to process : 1
20/12/19 19:03:30 INFO : number of splits:1
20/12/19 19:03:30 INFO : Submitting tokens for job: job_1608426890333_0007
20/12/19 19:03:30 INFO : Job jar is not present. Not adding any jar to the list of resources.
20/12/19 19:03:30 INFO : Submitted application application_1608426890333_0007
20/12/19 19:03:30 INFO : The url to track the job: http://node01:8088/proxy/application_1608426890333_0007/
20/12/19 19:03:30 INFO : Running job: job_1608426890333_0007
20/12/19 19:04:10 INFO : Job job_1608426890333_0007 running in uber mode : true
20/12/19 19:04:10 INFO : map 100% reduce 0%
20/12/19 19:04:11 INFO : map 100% reduce 100%
20/12/19 19:04:12 INFO : Job job_1608426890333_0007 failed with state FAILED due to: Task failed task_1608426890333_0007_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
20/12/19 19:04:12 INFO : Counters: 17
Job Counters
Failed map tasks=1
Failed reduce tasks=1
Launched map tasks=1
Launched reduce tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=1065
Total time spent by all reduces in occupied slots (ms)=155
TOTAL_LAUNCHED_UBERTASKS=2
NUM_UBER_SUBMAPS=1
NUM_UBER_SUBREDUCES=1
NUM_FAILED_UBERTASKS=2
Total time spent by all map tasks (ms)=1065
Total time spent by all reduce tasks (ms)=155
Total vcore-milliseconds taken by all map tasks=1065
Total vcore-milliseconds taken by all reduce tasks=155
Total megabyte-milliseconds taken by all map tasks=1090560
Total megabyte-milliseconds taken by all reduce tasks=158720
查阅了一下资料,原因是找不到job任务运行的资源,如map类、reduce类等。
public class JobMain extends Configured implements Tool {
@Override
public int run(String[] args) throws Exception {
//1:创建job对象
Job job = Job.getInstance(super.getConf(), "mapreduce_sort");
//只要是Initmain类所在包下的任一个类名都可以,默认为当前 job所在类名
job.setJarByClass(JobMain.class);
...
再回到集群上运行就好了。