hadoop公平调度其Fair Scheduler运行错误

时间：2021-11-09 14:37:56

      下午将hadoop的调度器更改为Fair Scheduler，没想到晚上Fair Scheduler就抛出了异常，异常错误信息如下：

23:51:36,437 ERROR org.mortbay.log: /scheduler
java.lang.NullPointerException
at org.apache.hadoop.mapred.FairSchedulerServlet$1.compare(FairSchedulerServlet.java:172)
at org.apache.hadoop.mapred.FairSchedulerServlet$1.compare(FairSchedulerServlet.java:166)
at java.util.Arrays.mergeSort(Arrays.java:1270)
at java.util.Arrays.mergeSort(Arrays.java:1281)
at java.util.Arrays.sort(Arrays.java:1210)
at java.util.Collections.sort(Collections.java:159)
at org.apache.hadoop.mapred.FairSchedulerServlet.showPools(FairSchedulerServlet.java:166)
at org.apache.hadoop.mapred.FairSchedulerServlet.doGet(FairSchedulerServlet.java:142)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
       当时正在运行的任务抛出的错误信息如下： 

java.io.IOException: Call to test/1.2.3.4:9001 failed on local exception: java.io.EOFException
        at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065)
        at org.apache.hadoop.ipc.Client.call(Client.java:1033)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224)
        at org.apache.hadoop.mapred.$Proxy9.getJobStatus(Unknown Source)
        at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1011)
        at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:1023)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.progress(ExecDriver.java:351)
        at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:672)
        at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:123)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:130)
        at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1063)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:900)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:748)
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:164)
        at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:241)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:456)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:375)
        at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767)
        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712)
Ended Job = job_201111161516_0696 with exception 'java.io.IOException(Call to test/1.2.3.4:9001 failed on local exception: java.io.EOFException)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
       不过唯一值得欣慰的是，jobtracker并没有停止工作，而是切换到了FIFO调度器（这句话可能有错误，在排查今天的jobtracker日志时，也发现的同样的错误，但是Fair Scheduler仍然在运行），Fair Scheduler运行错误只影响了当时正在运行的任务。

       根据jobtracker的错误信息google了一下，但是并没有找到相关信息，不知道有没有使用hadoop的同学遇上了同样的问题，求解决办法？

       (18日添加)

       今天早*问Fair Scheduler的调度页面时（http://1.2.3.4:50030/scheduler），页面报错：

HTTP ERROR 500

Problem accessing /scheduler. Reason:

    INTERNAL_SERVER_ERROR

Caused by:

java.lang.NullPointerException
at org.apache.hadoop.mapred.FairSchedulerServlet$1.compare(FairSchedulerServlet.java:172)
at org.apache.hadoop.mapred.FairSchedulerServlet$1.compare(FairSchedulerServlet.java:166)
at java.util.Arrays.mergeSort(Arrays.java:1270)
at java.util.Arrays.sort(Arrays.java:1210)
at java.util.Collections.sort(Collections.java:159)
at org.apache.hadoop.mapred.FairSchedulerServlet.showPools(FairSchedulerServlet.java:166)
at org.apache.hadoop.mapred.FairSchedulerServlet.doGet(FairSchedulerServlet.java:142)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:816)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Powered by Jetty://
      页面报错的同时，jobtracker的日志中也打印出了第一段错误代码，看了之前的分析是错误的。





标签：

相关文章