UHP博客文章地址：http://yuntai.1kapp.com/?p=1060
原创文章，转载请注明出处：http://blog.csdn.net/wind5shy/article/details/8563355
物理环境
主节点 |
10.28.171.33 |
处理节点1 |
10.28.171.40 |
处理节点2 |
10.28.171.41 |
处理节点3 |
10.28.169.42 |
处理节点4 |
10.28.171.43 |
查询SQL语句
select jobinfo.dt,user,max(taskinfo.finish_time-taskinfo.start_time),max(jobinfo.finish_time-jobinfo.submit_time)
fromtaskinfo join jobinfo on jobinfo.jobid=taskinfo.jobid
where jobinfo.job_status='SUCCESS' and taskinfo.task_status='SUCCESS'
group by jobinfo.dt,user
连接到处理节点3上进行处理。
处理节点3日志
解释SQL语句，生成查询计划树
Explain query: selectjobinfo.dt,user,max(taskinfo.finish_time-taskinfo.start_time),max(jobinfo.finish_time-jobinfo.submit_time)from taskinfo join jobinfo on jobinfo.jobid=taskinfo.jobid where jobinfo.job_status='SUCCESS' andtaskinfo.task_status='SUCCESS' group by jobinfo.dt,user
协调器PlanFragment，查询计划树的根
Plan Fragment 0
UNPARTITIONED
AGGREGATE
从各backend结果中再次处理MAX与GROUP BY
OUTPUT: MAX(<slot 12>), MAX(<slot 13>)
GROUP BY: <slot 10>, <slot 11>
TUPLE IDS: 2
EXCHANGE (5)
TUPLE IDS: 2
Plan Fragment 1
Plan Fragment 1
RANDOM
STREAM DATA SINK
EXCHANGE ID: 5
UNPARTITIONED
AGGREGATE
处理MAX与GROUP BY
OUTPUT: MAX(taskinfo.finish_time - taskinfo.start_time),MAX(jobinfo.finish_time - jobinfo.submit_time)
GROUP BY: jobinfo.dt, user
TUPLE IDS: 2
处理join
HASH JOIN
JOIN OP: INNER JOIN
HASH PREDICATES:
taskinfo.jobid = jobinfo.jobid
TUPLE IDS: 0 1
扫描taskinfo表数据并用taskinfo.task_status= 'SUCCESS'进行筛选
SCAN HDFS table=default.taskinfo (0)
PREDICATES: taskinfo.task_status = 'SUCCESS'
TUPLE IDS: 0
EXCHANGE (4)
TUPLE IDS: 1
Plan Fragment 2
Plan Fragment 2
RANDOM
STREAM DATA SINK
EXCHANGE ID: 4
UNPARTITIONED
扫描jobinfo表数据并用jobinfo.job_status= 'SUCCESS'进行筛选
SCAN HDFS table=default.jobinfo (1) compact
PREDICATES: jobinfo.job_status = 'SUCCESS'
TUPLE IDS: 1
查询配置
I0122 20:10:04.561157 1031 impala-server.cc:1516] TClientRequest.queryOptions:TQueryOptions {
01:abort_on_error (bool) = false,
02:max_errors (i32) = 0,
03:disable_codegen (bool) = false,
04:batch_size (i32) = 0,
05:return_as_ascii (bool) = true,
06:num_nodes (i32) = 0,
07:max_scan_range_length (i64) = 0,
08:num_scanner_threads (i32) = 0,
09:max_io_buffers (i32) = 0,
10:allow_unsupported_formats (bool) = false,
11:partition_agg (bool) = false,
}
开始执行查询
I0122 20:10:04.561363 1031 impala-server.cc:863] query(): query=selectjobinfo.dt,user,max(taskinfo.finish_time-taskinfo.start_time),max(jobinfo.finish_time-jobinfo.submit_time)from taskinfo join jobinfo on jobinfo.jobid=taskinfo.jobid where jobinfo.job_status='SUCCESS' andtaskinfo.task_status='SUCCESS' group by jobinfo.dt,user
协调器根据查询计划树给各个Backend节点分配任务
impala-server.cc:1119
I0122 20:10:05.271327 1031 coordinator.cc:219] Exec()query_id=67692cad49234453:8cf581505134c8ca
coordinator.cc:1161
Plan Fragment 1任务分配
I0122 20:10:05.271467 1031 simple-scheduler.cc:171] SimpleSchedulerassignment (data->backend): (10.28.171.43:50010 -> 10.28.171.43:22000), (10.28.171.41:50010 ->10.28.171.41:22000), (10.28.171.42:50010 -> 10.28.171.42:22000),(10.28.171.40:50010 -> 10.28.171.40:22000)
Plan Fragment 1 分配到任务的4个Backend均为本地处理
I0122 20:10:05.271487 1031 simple-scheduler.cc:174] SimpleSchedulerlocality percentage 100% (4 out of 4)
Plan Fragment 2任务分配
I0122 20:10:05.271610 1031 simple-scheduler.cc:171] SimpleSchedulerassignment (data->backend): (10.28.171.43:50010 -> 10.28.171.43:22000), (10.28.171.41:50010 ->10.28.171.41:22000), (10.28.171.42:50010 -> 10.28.171.42:22000),(10.28.171.40:50010 -> 10.28.171.40:22000)
Plan Fragment 2 分配到任务的4个Backend均为本地处理
I0122 20:10:05.271627 1031 simple-scheduler.cc:174] SimpleSchedulerlocality percentage 100% (4 out of 4)
协调器开始在各个Backend执行查询计划
coordinator.cc:255
I0122 20:10:05.271853 1031 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cb
I0122 20:10:05.277370 1031 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8cb
I0122 20:10:05.335903 1031 coordinator.cc:308] starting 8 backendsfor query 67692cad49234453:8cf581505134c8ca
本节点开始处理PlanFragment 1 Backend 2的任务
coordinator.cc:698
I0122 20:10:05.336571 1119 impala-server.cc:1655(1784)]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8cecoord=10.28.171.42:22000 backend#=2
I0122 20:10:05.336617 1119 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8ca instance_id=67692cad49234453:8cf581505134c8ce
I0122 20:10:05.339373 1119 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8ce
I0122 20:10:05.462807 11899plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8ce
本节点开始处理PlanFragment 2 Backend 6的任务
coordinator.cc:698
I0122 20:10:05.468166 1119 impala-server.cc:1655]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8d2coord=10.28.171.42:22000 backend#=6
I0122 20:10:05.468377 1119 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8d2
I0122 20:10:05.475342 1119 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8d2
I0122 20:10:05.486824 11899 aggregation-node.cc:166]AggregationNode(node_id=3) using llvm codegend functions.
I0122 20:10:05.490782 11899hash-join-node.cc:157] HashJoinNode(node_id=2) using llvm codegend function forbuilding hash table.
I0122 20:10:05.499639 11913plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8d2
I0122 20:10:05.499886 11916plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8cb
I0122 20:10:05.499991 11899hash-join-node.cc:170] HashJoinNode(node_id=2) using llvm codegend function forprobing hash table.
协调器接受各个后端的处理进度
I0122 20:10:05.500530 1560 coordinator.cc:904] Backend 4 completed,7 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:10:05.500568 1560 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000
I0122 20:10:05.525993 11916aggregation-node.cc:166] AggregationNode(node_id=6) using llvm codegendfunctions.
I0122 20:10:10.237057 1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 2% Complete (1 out of 36)
I0122 20:10:10.237105 1127 coordinator.cc:904] Backend 5 completed,6 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:10:10.237129 1127 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000
I0122 20:10:10.518767 1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 5% Complete (2 out of 36)
I0122 20:10:11.710980 1129 coordinator.cc:904] Backend 7 completed,5 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:10:11.711033 1129 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000
I0122 20:10:12.263823 1119 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 11% Complete (4 out of 36)
I0122 20:10:12.263871 1119 coordinator.cc:904] Backend 6 completed,4 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:10:12.263895 1119 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.40:22000
I0122 20:10:30.025861 1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 13% Complete (5 out of 36)
I0122 20:10:30.759806 1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 16% Complete (6 out of 36)
I0122 20:11:16.298985 1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 25% Complete (9 out of 36)
I0122 20:11:21.365696 1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 27% Complete (10 out of 36)
I0122 20:11:21.532614 1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 33% Complete (12 out of 36)
I0122 20:11:26.448179 1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 36% Complete (13 out of 36)
I0122 20:11:26.581250 1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 41% Complete (15 out of 36)
I0122 20:11:26.614269 1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 47% Complete (17 out of 36)
I0122 20:11:31.624903 1129 progress-updater.cc:55] Query 67692cad49234453:8cf581505134c8ca:50% Complete (18 out of 36)
I0122 20:11:31.693733 1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 52% Complete (19 out of 36)
I0122 20:11:36.556718 1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 55% Complete (20 out of 36)
I0122 20:11:36.770860 1560 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 58% Complete (21 out of 36)
I0122 20:11:36.770910 1560 coordinator.cc:904] Backend 0 completed,3 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:11:36.770932 1560 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.41:22000
I0122 20:11:36.829344 1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 63% Complete (23 out of 36)
I0122 20:11:37.189350 1129 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 69% Complete (25 out of 36)
I0122 20:11:37.189409 1129 coordinator.cc:904] Backend 3 completed,2 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:11:37.189440 1129 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.41:22000
I0122 20:11:37.485113 1119 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 77% Complete (28 out of 36)
I0122 20:11:42.486377 1119 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 83% Complete (30 out of 36)
I0122 20:11:46.854956 1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 86% Complete (31 out of 36)
I0122 20:11:49.901679 1127 progress-updater.cc:55] Query67692cad49234453:8cf581505134c8ca: 91% Complete (33 out of 36)
I0122 20:11:49.901748 1127 coordinator.cc:904] Backend 1 completed,1 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:11:49.901774 1127 coordinator.cc:913]query_id=67692cad49234453:8cf581505134c8ca: first in-progress backend:10.28.171.42:22000
I0122 20:12:03.582316 1119 progress-updater.cc:45] Query67692cad49234453:8cf581505134c8ca 100% Complete (36 out of 36)
I0122 20:12:03.582365 1119 coordinator.cc:904] Backend 2 completed,0 remaining: query_id=67692cad49234453:8cf581505134c8ca
I0122 20:12:03.589012 11899data-stream-mgr.cc:190(200)] DeregisterRecvr():fragment_id=67692cad49234453:8cf581505134c8ce, node=4
I0122 20:12:03.761831 1031 plan-fragment-executor.cc:335] Finishedexecuting fragment query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cb
I0122 20:12:03.763402 1031 plan-fragment-executor.cc:335] Finished executingfragment query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cb
I0122 20:12:03.763422 1031 coordinator.cc:481] Coordinator waitingfor backends to finish, 0 remaining
I0122 20:12:03.763437 1031 coordinator.cc:486] All backendsfinished or error.
生成查询过程分析文件
I0122 20:12:03.765897 1031 coordinator.cc:1027] Final profile forquery_id=67692cad49234453:8cf581505134c8ca
Query67692cad49234453:8cf581505134c8ca:(228ms 0.00%)
Aggregate Profile:
协调器PlanFragment统计
Coordinator Fragment:(1m58s 0.00%)
- RowsProduced: 20
CodeGen:
- CodegenTime: 2ms
- CompileTime: 55ms
- LoadTime: 5ms
- ModuleFileSize: 34.37 KB
AGGREGATION_NODE (id=6):(1m58s 0.02%)
- BuildBuckets: 1.02K
- BuildTime: 180K clock cycles
- GetResultsTime: 23K clock cycles
- MemoryUsed: 33.44 KB
- RowsReturned: 20
- RowsReturnedRate: 0
EXCHANGE_NODE (id=5):(1m58s 99.98%)
- MemoryUsed: 0.00
- RowsReturned: 80
- RowsReturnedRate: 0
PlanFragment 2平均分析
Averaged Fragment 2:(1s193ms 0.00%)
completion times: min:2ms max:6s764ms mean: 4s431ms stddev:2s662ms
execution rates: min:0.00 /sec max:6.23 MB/sec mean:3.61MB/sec stddev:2.31 MB/sec
split sizes: min: 0.00 , max:42.17 MB, avg: 21.54 MB, stddev: 14.93 MB
- RowsProduced: 10.80K
CodeGen:
- CodegenTime: 440K clock cycles
- CompileTime: 22ms
- LoadTime: 3ms
- ModuleFileSize: 34.37 KB
HDFS_SCAN_NODE (id=1):(1s193ms 15.89%)
- BytesRead: 21.54 MB
- DelimiterParseTime: 193ms
- MaterializeTupleTime: 3ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 6.67 MB/sec
- RowsReturned: 10.80K
- RowsReturnedRate: 7.93 K/sec
- ScanRangesComplete: 1
- ScannerThreadsReadTime: 2s621ms
- TotalReadThroughput: 4.07 MB/sec
PlanFragment 1平均分析
Averaged Fragment 1:(1m41s 0.00%)
completion times: min:1m31s max:1m58s mean: 1m41s stddev:11s004ms
execution rates: min:4.28 MB/sec max:5.01 MB/sec mean:4.65MB/sec stddev:353.17 KB/sec
split sizes: min: 447.30 MB, max:512.00 MB, avg: 468.49 MB, stddev: 25.49 MB
- RowsProduced: 20
CodeGen:
- CodegenTime: 5ms
- CompileTime: 114ms
- LoadTime: 5ms
- ModuleFileSize: 34.37 KB
AGGREGATION_NODE (id=3):(1m41s 1.13%)
- BuildBuckets: 1.02K
- BuildTime: 1s043ms
- GetResultsTime: 21K clock cycles
- MemoryUsed: 33.44 KB
- RowsReturned: 20
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=2):(1m40s 1.75%)
- BuildBuckets: 32.77K
- BuildRows: 43.22K
- BuildTime: 21ms
- MemoryUsed: 0.00
- ProbeRows: 3.75M
- ProbeTime: 1s608ms
- RowsReturned: 3.68M
- RowsReturnedRate: 36.93 K/sec
EXCHANGE_NODE (id=4):(6s737ms 7.37%)
- MemoryUsed: 0.00
- RowsReturned: 43.22K
- RowsReturnedRate: 6.41 K/sec
HDFS_SCAN_NODE (id=0):(1m31s 89.76%)
- BytesRead: 468.49 MB
- DelimiterParseTime: 4s923ms
- MaterializeTupleTime: 458ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 5.26 MB/sec
- RowsReturned: 3.75M
- RowsReturnedRate: 41.12 K/sec
- ScanRangesComplete: 8
- ScannerThreadsReadTime: 1m29s
- TotalReadThroughput: 4.62MB/sec
Plan Fragment 1分析
Fragment 1:
Backend 0分析
Instance 67692cad49234453:8cf581505134c8cc:(1m31s 0.00%)
Hdfs split stats (:<# splits>/): 0:8/477.50M
- RowsProduced: 20
CodeGen:
- CodegenTime: 5ms
- CompileTime: 114ms
- LoadTime: 6ms
- ModuleFileSize: 34.37 KB
AGGREGATION_NODE (id=3):(1m31s 1.13%)
- BuildBuckets: 1.02K
- BuildTime: 1s006ms
- GetResultsTime: 21K clock cycles
- MemoryUsed: 33.44 KB
- RowsReturned: 20
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=2):(1m30s 1.75%)
- BuildBuckets: 32.77K
- BuildRows: 43.22K
- BuildTime: 21ms
- MemoryUsed: 0.00
- ProbeRows: 3.63M
- ProbeTime: 1s555ms
- RowsReturned: 3.61M
- RowsReturnedRate: 39.97 K/sec
EXCHANGE_NODE (id=4):(6s727ms 7.37%)
- MemoryUsed: 0.00
- RowsReturned: 43.22K
- RowsReturnedRate: 6.42 K/sec
HDFS_SCAN_NODE (id=0):(1m21s 89.76%)
- BytesRead: 455.38 MB
- DelimiterParseTime: 4s503ms
- MaterializeTupleTime: 428ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 5.67 MB/sec
- RowsReturned: 3.63M
- RowsReturnedRate: 44.30 K/sec
- ScanRangesComplete: 8
- ScannerThreadsReadTime: 1m20s
- TotalReadThroughput: 4.98 MB/sec
Backend 1分析
Instance 67692cad49234453:8cf581505134c8cd:(1m44s 0.00%)
Hdfs split stats (:<# splits>/): 0:8/469.03M
- RowsProduced: 20
CodeGen:
- CodegenTime: 5ms
- CompileTime: 114ms
- LoadTime: 6ms
- ModuleFileSize: 34.37 KB
AGGREGATION_NODE (id=3):(1m44s 0.98%)
- BuildBuckets: 1.02K
- BuildTime: 994ms
- GetResultsTime: 21K clock cycles
- MemoryUsed: 33.44 KB
- RowsReturned: 20
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=2):(1m43s 1.52%)
- BuildBuckets: 32.77K
- BuildRows: 43.22K
- BuildTime: 21ms
- MemoryUsed: 0.00
- ProbeRows: 3.59M
- ProbeTime: 1s550ms
- RowsReturned: 3.56M
- RowsReturnedRate: 34.42 K/sec
EXCHANGE_NODE (id=4):(6s739ms 6.45%)
- MemoryUsed: 0.00
- RowsReturned: 43.22K
- RowsReturnedRate: 6.41 K/sec
HDFS_SCAN_NODE (id=0):(1m35s 91.05%)
- BytesRead: 447.30 MB
- DelimiterParseTime: 4s519ms
- MaterializeTupleTime: 462ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 4.90 MB/sec
- RowsReturned: 3.59M
- RowsReturnedRate: 37.79 K/sec
- ScanRangesComplete: 8
- ScannerThreadsReadTime: 1m31s
- TotalReadThroughput: 4.20 MB/sec
Backend 2分析
Instance 67692cad49234453:8cf581505134c8ce:(1m58s 0.00%)
Hdfs split stats (:<# splits>/): 0:8/536.87M
- RowsProduced: 20
CodeGen:
- CodegenTime: 5ms
- CompileTime: 115ms
- LoadTime: 2ms
- ModuleFileSize: 34.37 KB
AGGREGATION_NODE (id=3):(1m58s 1.00%)
- BuildBuckets: 1.02K
- BuildTime: 1s144ms
- GetResultsTime: 21K clock cycles
- MemoryUsed: 33.44 KB
- RowsReturned: 20
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=2):(1m56s 1.52%)
- BuildBuckets: 32.77K
- BuildRows: 43.22K
- BuildTime: 22ms
- MemoryUsed: 0.00
- ProbeRows: 4.11M
- ProbeTime: 1s750ms
- RowsReturned: 4.06M
- RowsReturnedRate: 34.71 K/sec
EXCHANGE_NODE (id=4):(6s739ms 5.71%)
- MemoryUsed: 0.00
- RowsReturned: 43.22K
- RowsReturnedRate: 6.41 K/sec
HDFS_SCAN_NODE (id=0):(1m48s 91.78%)
- BytesRead: 512.00 MB
- DelimiterParseTime: 5s083ms
- MaterializeTupleTime: 515ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 4.82 MB/sec
- RowsReturned: 4.11M
- RowsReturnedRate: 37.90 K/sec
- ScanRangesComplete: 8
- ScannerThreadsReadTime: 1m46s
- TotalReadThroughput: 4.32 MB/sec
Backend 3分析
Instance 67692cad49234453:8cf581505134c8cf:(1m31s 0.00%)
Hdfs split stats (:<# splits>/): 0:8/481.61M
- RowsProduced: 20
CodeGen:
- CodegenTime: 5ms
- CompileTime: 114ms
- LoadTime: 6ms
- ModuleFileSize: 34.37 KB
AGGREGATION_NODE (id=3):(1m31s 1.15%)
- BuildBuckets: 1.02K
- BuildTime: 1s030ms
- GetResultsTime: 20K clock cycles
- MemoryUsed: 33.44 KB
- RowsReturned: 20
- RowsReturnedRate: 0
HASH_JOIN_NODE (id=2):(1m30s 1.77%)
- BuildBuckets: 32.77K
- BuildRows: 43.22K
- BuildTime: 21ms
- MemoryUsed: 0.00
- ProbeRows: 3.66M
- ProbeTime: 1s577ms
- RowsReturned: 3.50M
- RowsReturnedRate: 38.62 K/sec
EXCHANGE_NODE (id=4):(6s741ms 7.35%)
- MemoryUsed: 0.00
- RowsReturned: 43.22K
- RowsReturnedRate: 6.41 K/sec
HDFS_SCAN_NODE (id=0):(1m22s 89.73%)
- BytesRead: 459.30 MB
- DelimiterParseTime: 5s587ms
- MaterializeTupleTime: 426ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 5.64 MB/sec
- RowsReturned: 3.66M
- RowsReturnedRate: 44.49 K/sec
- ScanRangesComplete: 8
- ScannerThreadsReadTime: 1m21s
- TotalReadThroughput: 4.99 MB/sec
Plan Fragment 2分析
Fragment 2:
Backend 4分析
Instance 67692cad49234453:8cf581505134c8d0:(11K clock cycles 84.11%)
- RowsProduced: 0
CodeGen:
- CodegenTime: 358K clock cycles
- CompileTime: 24ms
- LoadTime: 2ms
- ModuleFileSize: 34.37 KB
HDFS_SCAN_NODE (id=1):(1K clock cycles 15.89%)
- BytesRead: 0.00
- MaterializeTupleTime: 0K clock cycles
- MemoryUsed: 0.00
- PerDiskReadThroughput: 0.00 /sec
- RowsReturned: 0
- RowsReturnedRate: 0
- ScanRangesComplete: 0
- ScannerThreadsReadTime: 0K clock cycles
- TotalReadThroughput: 0.00 /sec
Backend 5分析
Instance 67692cad49234453:8cf581505134c8d1:(1s026ms 0.01%)
Hdfs split stats (:<# splits>/): 0:1/23.96M
- RowsProduced: 11.35K
CodeGen:
- CodegenTime: 321K clock cycles
- CompileTime: 21ms
- LoadTime: 2ms
- ModuleFileSize: 34.37 KB
HDFS_SCAN_NODE (id=1):(1s026ms 99.99%)
- BytesRead: 22.85 MB
- DelimiterParseTime: 196ms
- MaterializeTupleTime: 3ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 8.94 MB/sec
- RowsReturned: 11.35K
- RowsReturnedRate: 11.05 K/sec
- ScanRangesComplete: 1
- ScannerThreadsReadTime: 2s554ms
- TotalReadThroughput: 6.53 MB/sec
Backend 6分析
Instance 67692cad49234453:8cf581505134c8d2:(2s957ms 0.03%)
Hdfs split stats (:<# splits>/): 0:2/44.22M
- RowsProduced: 21.19K
CodeGen:
- CodegenTime: 766K clock cycles
- CompileTime: 22ms
- LoadTime: 6ms
- ModuleFileSize: 34.37 KB
HDFS_SCAN_NODE (id=1):(2s957ms 99.97%)
- BytesRead: 42.17 MB
- DelimiterParseTime: 358ms
- MaterializeTupleTime: 6ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 7.09 MB/sec
- RowsReturned: 21.19K
- RowsReturnedRate: 7.17 K/sec
- ScanRangesComplete: 2
- ScannerThreadsReadTime: 5s948ms
- TotalReadThroughput: 6.49 MB/sec
Backend 7分析
Instance 67692cad49234453:8cf581505134c8d3:(790ms 0.03%)
Hdfs split stats (:<# splits>/): 0:1/22.16M
- RowsProduced: 10.67K
CodeGen:
- CodegenTime: 314K clock cycles
- CompileTime: 21ms
- LoadTime: 2ms
- ModuleFileSize: 34.37 KB
HDFS_SCAN_NODE (id=1):(790ms 99.97%)
- BytesRead: 21.13 MB
- DelimiterParseTime: 219ms
- MaterializeTupleTime: 3ms
- MemoryUsed: 0.00
- PerDiskReadThroughput: 10.67 MB/sec
- RowsReturned: 10.67K
- RowsReturnedRate: 13.50 K/sec
- ScanRangesComplete: 1
- ScannerThreadsReadTime: 1s981ms
- TotalReadThroughput: 3.25 MB/sec
查询处理完成，注销服务
I0122 20:12:03.767083 1031 impala-server.cc:1393] close():query_id=67692cad49234453:8cf581505134c8ca
I0122 20:12:03.767102 1031 impala-server.cc:1008]UnregisterQuery(): query_id=67692cad49234453:8cf581505134c8ca
I0122 20:12:03.770297 1031 data-stream-mgr.cc:190]DeregisterRecvr(): fragment_id=67692cad49234453:8cf581505134c8cb, node=5
处理节点1日志
本节点开始处理PlanFragment 1 Backend 0的任务
I0122 20:10:13.667968 26470impala-server.cc:1655] ExecPlanFragment()instance_id=67692cad49234453:8cf581505134c8cc coord=10.28.171.42:22000backend#=0
I0122 20:10:13.668203 26470plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cc
I0122 20:10:13.674814 26470plan-fragment-executor.cc:95] descriptor table forfragment=67692cad49234453:8cf581505134c8cc
I0122 20:10:13.797652 4113 plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8cc
本节点开始处理PlanFragment 2 Backend 4的任务
I0122 20:10:13.800835 26470impala-server.cc:1655] ExecPlanFragment()instance_id=67692cad49234453:8cf581505134c8d0 coord=10.28.171.42:22000 backend#=4
I0122 20:10:13.800878 26470plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8d0
I0122 20:10:13.803349 26470plan-fragment-executor.cc:95] descriptor table for fragment=67692cad49234453:8cf581505134c8d0
I0122 20:10:13.816920 4113 aggregation-node.cc:166]AggregationNode(node_id=3) using llvm codegend functions.
I0122 20:10:13.820927 4113 hash-join-node.cc:157]HashJoinNode(node_id=2) using llvm codegend function for building hash table.
I0122 20:10:13.828485 4118 plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8d0
I0122 20:10:13.830157 4113 hash-join-node.cc:170]HashJoinNode(node_id=2) using llvm codegend function for probing hash table.
查询任务全部处理完成，注销服务
I0122 20:11:45.129060 4113 data-stream-mgr.cc:190]DeregisterRecvr(): fragment_id=67692cad49234453:8cf581505134c8cc, node=4
处理节点2日志
本节点开始处理PlanFragment 1 Backend 1的任务
I0122 20:10:12.942119 22235impala-server.cc:1655] ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8cdcoord=10.28.171.42:22000 backend#=1
I0122 20:10:12.942455 22235plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cd
I0122 20:10:12.949337 22235plan-fragment-executor.cc:95] descriptor table forfragment=67692cad49234453:8cf581505134c8cd
I0122 20:10:13.072387 31215plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8cd
本节点开始处理PlanFragment 2 Backend 5的任务
I0122 20:10:13.073715 22235impala-server.cc:1655] ExecPlanFragment()instance_id=67692cad49234453:8cf581505134c8d1 coord=10.28.171.42:22000backend#=5
I0122 20:10:13.073760 22235plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8ca instance_id=67692cad49234453:8cf581505134c8d1
I0122 20:10:13.076200 22235plan-fragment-executor.cc:95] descriptor table forfragment=67692cad49234453:8cf581505134c8d1
I0122 20:10:13.091857 31215aggregation-node.cc:166] AggregationNode(node_id=3) using llvm codegendfunctions.
I0122 20:10:13.095860 31215hash-join-node.cc:157] HashJoinNode(node_id=2) using llvm codegend function forbuilding hash table.
I0122 20:10:13.098786 31218plan-fragment-executor.cc:194] Open():instance_id=67692cad49234453:8cf581505134c8d1
I0122 20:10:13.105170 31215hash-join-node.cc:170] HashJoinNode(node_id=2) using llvm codegend function forprobing hash table.
查询任务全部处理完成，注销服务
I0122 20:11:57.536412 31215data-stream-mgr.cc:190] DeregisterRecvr():fragment_id=67692cad49234453:8cf581505134c8cd, node=4
处理节点4日志
本节点开始处理PlanFragment 1 Backend 3的任务
I0122 20:10:06.177486 1448 impala-server.cc:1655]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8cfcoord=10.28.171.42:22000 backend#=3
I0122 20:10:06.177809 1448 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8cf
I0122 20:10:06.184216 1448 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8cf
tuples:
I0122 20:10:06.306742 15883plan-fragment-executor.cc:194] Open(): instance_id=67692cad49234453:8cf581505134c8cf
本节点开始处理PlanFragment 2 Backend 7的任务
I0122 20:10:06.308400 1448 impala-server.cc:1655]ExecPlanFragment() instance_id=67692cad49234453:8cf581505134c8d3coord=10.28.171.42:22000 backend#=7
I0122 20:10:06.308454 1448 plan-fragment-executor.cc:82] Prepare():query_id=67692cad49234453:8cf581505134c8cainstance_id=67692cad49234453:8cf581505134c8d3
I0122 20:10:06.311002 1448 plan-fragment-executor.cc:95] descriptortable for fragment=67692cad49234453:8cf581505134c8d3
I0122 20:10:06.326146 15883aggregation-node.cc:166] AggregationNode(node_id=3) using llvm codegendfunctions.
I0122 20:10:06.330163 15883hash-join-node.cc:157] HashJoinNode(node_id=2) using llvm codegend function forbuilding hash table.
I0122 20:10:06.333781 15885 plan-fragment-executor.cc:194]Open(): instance_id=67692cad49234453:8cf581505134c8d3
I0122 20:10:06.339367 15883hash-join-node.cc:170] HashJoinNode(node_id=2) using llvm codegend function forprobing hash table.
查询任务全部处理完成，注销服务
I0122 20:11:38.036321 15883data-stream-mgr.cc:190] DeregisterRecvr():fragment_id=67692cad49234453:8cf581505134c8cf, node=4
原创文章，转载请注明出处：http://blog.csdn.net/wind5shy/article/details/8563355