hadoop执行统计问题,文件大小1G,三台机器都是1G内存

时间:2020-12-10 17:21:11
之前小文件测试的时候是没问题的,但是大文件之后就出这问题:
15/11/08 03:47:50 INFO mapred.MapTask: Starting flush of map output
15/11/08 03:47:50 INFO mapred.MapTask: Spilling map output
15/11/08 03:47:50 INFO mapred.MapTask: bufstart = 0; bufend = 146030; bufvoid = 104857600
15/11/08 03:47:50 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26180040(104720160); length = 34357/6553600
15/11/08 03:47:50 INFO mapred.MapTask: Finished spill 0
15/11/08 03:47:50 INFO mapred.LocalJobRunner: map task executor complete.
15/11/08 03:47:50 WARN mapred.LocalJobRunner: job_local2078971688_0001
java.lang.Exception: java.lang.NullPointerException
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.io.Text.encode(Text.java:450)
        at org.apache.hadoop.io.Text.set(Text.java:198)
        at com.myhadoop.Log.statics.statictics$StaticIPMapper.map(statictics.java:33)
        at com.myhadoop.Log.statics.statictics$StaticIPMapper.map(statictics.java:1)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
15/11/08 03:47:50 INFO mapred.LocalJobRunner: map > sort
15/11/08 03:47:51 INFO mapreduce.Job:  map 50% reduce 0%
15/11/08 03:47:51 INFO mapreduce.Job: Job job_local2078971688_0001 failed with state FAILED due to: NA
15/11/08 03:47:51 INFO mapreduce.Job: Counters: 26
        File System Counters
                FILE: Number of bytes read=55771
                FILE: Number of bytes written=1117678
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=11182080
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=40
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=5
        Map-Reduce Framework
                Map input records=23630
                Map output records=23626
                Map output bytes=401642
                Map output materialized bytes=1677
                Input split bytes=480
                Combine input records=23626
                Combine output records=87
                Spilled Records=87
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=1331
                CPU time spent (ms)=0
                Physical memory (bytes) snapshot=0
                Virtual memory (bytes) snapshot=0
                Total committed heap usage (bytes)=513314816
        File Input Format Counters
                Bytes Read=2199552

1 个解决方案

#1


已经找到问题

#1


已经找到问题