Spark大型电商项目实战-及其改良(4) 单独运行程序发现的问题

　　之前的运行结果比对发现,有1个函数的作用在2个job里面是相同的,但是对应的计算时间却差太远

　　于是把4个job分开运行.虽说使用的数据不同,但是生成数据的生成器是相同的,数据排布差距不大,数据量也是相同的.

　　以下是这4个job的运行时间表

.content{position:relative}
.vis.timeline .axis{position:absolute;width:100%;height:0;left:0;z-index:1}
.vis.timeline .item{position:absolute;color:#1A1A1A;border-color:#97B0F8;border-width:1px;background-color:#D5DDF6;display:inline-block;padding:5px}
.vis.timeline .item.range{border-style:solid;border-radius:2px;box-sizing:border-box}
.vis.timeline .item.stage {cursor: pointer;}
.vis.timeline .item.stage.succeeded {background-color: #A0DFFF;border-color: #3EC0FF;}
.vis.timeline .item.range .content{position:relative;display:inline-block;max-width:100%;overflow:hidden}
.vis.timeline .item.range .content {position: unset;}
.vis.timeline .timeaxis{position:relative;overflow:hidden}
.vis.timeline .timeaxis .text.measure{position:absolute;padding-left:0;padding-right:0;margin-left:0;margin-right:0;visibility:hidden}
.vis.timeline .timeaxis.foreground{top:0;left:0;width:100%}
.vis.timeline .timeaxis.background{position:absolute;top:0;left:0;width:100%;height:100%}
.vis.timeline .foreground {cursor: move;}
.vis.timeline .foreground .group{position:relative;box-sizing:border-box;border-bottom:1px solid #bfbfbf}
.vis.timeline .foreground .group:last-child{border-bottom:none}
.vis.timeline .timeaxis.background{position:absolute;top:0;left:0;width:100%;height:100%}
.vis.timeline .timeaxis .text{position:absolute;color:#4d4d4d;padding:3px;white-space:nowrap}
.vis.timeline .labelset{position:relative;overflow:hidden;box-sizing:border-box}
.vis.timeline .vispanel{position:absolute;padding:0;margin:0;box-sizing:border-box}
.vis.timeline .vispanel .shadow{position:absolute;width:100%;height:1px;box-shadow:0 0 10px rgba(0,0,0,.8)}
.vis.timeline .vispanel .shadow.top{top:-1px;left:0}
.vis.timeline .vispanel .shadow.bottom{bottom:-1px;left:0}
.vis.timeline .vispanel.bottom,.vis.timeline .vispanel.center,.vis.timeline .vispanel.left,.vis.timeline .vispanel.right,.vis.timeline .vispanel.top{border:1px #bfbfbf}
.vis.timeline .vispanel.center,.vis.timeline .vispanel.left,.vis.timeline .vispanel.right{border-top-style:solid;border-bottom-style:solid;overflow:hidden}
.vis.timeline .vispanel.bottom,.vis.timeline .vispanel.center,.vis.timeline .vispanel.top{border-left-style:solid;border-right-style:solid}
.vis.timeline .background{overflow:hidden}
.vis.timeline .labelset .vlabel{position:relative;left:0;top:0;width:100%;color:#4d4d4d;box-sizing:border-box;border-bottom:1px solid #bfbfbf}
.vis.timeline .labelset .vlabel .inner{display:inline-block;padding:5px}
.vis.timeline .labelset .vlabel:last-child{border-bottom:none}
.vis.timeline .timeaxis .grid.vertical{position:absolute;border-left:1px solid}
.vis.timeline .timeaxis .grid.minor{border-color:#e5e5e5}
#application-timeline div.legend-area,.my-job-timeline div.legend-area {margin-top: 5px;}
.vispanel.center {font-size: 12px;line-height: 12px;}
.legend-area rect.completed-stage-legend {fill: #A0DFFF;stroke: #3EC0FF;}
.legend-area rect.failed-stage-legend {fill: #FFA1B0;stroke: #FF4D6D;}
.legend-area rect.active-stage-legend {fill: #A2FCC0;stroke: #36F572;}
.legend-area rect.executor-added-legend {fill: #A0DFFF;stroke: #3EC0FF;}
.legend-area rect.executor-removed-legend {fill: #FFA1B0;stroke: #FF4D6D;}
div#application-timeline, div.my-job-timeline {margin-bottom: 30px;}
[class*="span"]{float:left;min-height:1px;margin-left:20px;}
table.sortable thead {cursor: pointer;}
table{max-width:100%;background-color:transparent;border-collapse:collapse;border-spacing:0;}
.table{width:100%;margin-bottom:20px;}
.table th,.table td{padding:8px;line-height:20px;text-align:left;vertical-align:top;border-top:1px solid #dddddd;}
.table th{font-weight:bold;}
.table caption+thead tr:first-child th,.table caption+thead tr:first-child td,.table colgroup+thead tr:first-child th,.table colgroup+thead tr:first-child td,.table thead:first-child tr:first-child th,.table thead:first-child tr:first-child td{border-top:0;}
table.sortable td {word-wrap: break-word;max-width: 600px;}
.table-striped tbody>tr:nth-child(odd)>td,.table-striped tbody>tr:nth-child(odd)>th{background-color:#f9f9f9;}
.table thead th{vertical-align:bottom;}
.table-condensed th,.table-condensed td{padding:4px 5px;}
.table-bordered{border:1px solid #dddddd;border-collapse:separate;*border-collapse:collapse;border-left:0;-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;}
.table-bordered th,.table-bordered td{border-left:1px solid #dddddd;}
.table-bordered caption+thead tr:first-child th,.table-bordered caption+tbody tr:first-child th,.table-bordered caption+tbody tr:first-child td,.table-bordered colgroup+thead tr:first-child th,.table-bordered colgroup+tbody tr:first-child th,.table-bordered colgroup+tbody tr:first-child td,.table-bordered thead:first-child tr:first-child th,.table-bordered tbody:first-child tr:first-child th,.table-bordered tbody:first-child tr:first-child td{border-top:0;}
.table-bordered thead:first-child tr:first-child>th:first-child,.table-bordered tbody:first-child tr:first-child>td:first-child,.table-bordered tbody:first-child tr:first-child>th:first-child{-webkit-border-top-left-radius:4px;-moz-border-radius-topleft:4px;border-top-left-radius:4px;}
.table-bordered thead:first-child tr:first-child>th:last-child,.table-bordered tbody:first-child tr:first-child>td:last-child,.table-bordered tbody:first-child tr:first-child>th:last-child{-webkit-border-top-right-radius:4px;-moz-border-radius-topright:4px;border-top-right-radius:4px;}
.table{width:100%;margin-bottom:20px;}
.progress{overflow:hidden;height:20px;margin-bottom:20px;background-color:#f7f7f7;background-image:-moz-linear-gradient(top, #f5f5f5, #f9f9f9);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#f5f5f5), to(#f9f9f9));background-image:-webkit-linear-gradient(top, #f5f5f5, #f9f9f9);background-image:-o-linear-gradient(top, #f5f5f5, #f9f9f9);background-image:linear-gradient(to bottom, #f5f5f5, #f9f9f9);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#fff5f5f5', endColorstr='#fff9f9f9', GradientType=0);-webkit-box-shadow:inset 0 1px 2px rgba(0, 0, 0, 0.1);-moz-box-shadow:inset 0 1px 2px rgba(0, 0, 0, 0.1);box-shadow:inset 0 1px 2px rgba(0, 0, 0, 0.1);-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;}
.progress {margin-bottom: 0px; position: relative}
.progress-completed .bar,.progress .bar{width:0%;height:100%;color:#ffffff;float:left;font-size:12px;text-align:center;text-shadow:0 -1px 0 rgba(0, 0, 0, 0.25);background-color:#0e90d2;background-image:-moz-linear-gradient(top, #149bdf, #0480be);background-image:-webkit-gradient(linear, 0 0, 0 100%, from(#149bdf), to(#0480be));background-image:-webkit-linear-gradient(top, #149bdf, #0480be);background-image:-o-linear-gradient(top, #149bdf, #0480be);background-image:linear-gradient(to bottom, #149bdf, #0480be);background-repeat:repeat-x;filter:progid:DXImageTransform.Microsoft.gradient(startColorstr='#ff149bdf', endColorstr='#ff0480be', GradientType=0);-webkit-box-shadow:inset 0 -1px 0 rgba(0, 0, 0, 0.15);-moz-box-shadow:inset 0 -1px 0 rgba(0, 0, 0, 0.15);box-shadow:inset 0 -1px 0 rgba(0, 0, 0, 0.15);-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box;-webkit-transition:width 0.6s ease;-moz-transition:width 0.6s ease;-o-transition:width 0.6s ease;transition:width 0.6s ease;}
.progress .bar-completed {background-color: #3EC0FF;background-image: -moz-linear-gradient(top, #44CBFF, #34B0EE);background-image: -webkit-gradient(linear, 0 0, 0 100%, from(#44CBFF), to(#34B0EE));background-image: -webkit-linear-gradient(top, #44CBFF, #34B0EE);background-image: -o-linear-gradient(top, #44CBFF, #34B0EE);background-image: linear-gradient(to bottom, #64CBFF, #54B0EE);background-repeat: repeat-x;filter: progid:dximagetransform.microsoft.gradient(startColorstr='#FF44CBFF', endColorstr='#FF34B0EE', GradientType=0);}
tr.corresponding-item-hover > td, tr.corresponding-item-hover > th {background-color: #D6FFE4 !important;}
.tooltip{position:absolute;z-index:1030;display:block;visibility:visible;font-size:11px;line-height:1.4;opacity:0;filter:alpha(opacity=0);}.tooltip.in{opacity:0.8;filter:alpha(opacity=80);}
.tooltip.top{margin-top:-3px;padding:5px 0;}
.tooltip.right{margin-left:3px;padding:0 5px;}
.tooltip.bottom{margin-top:3px;padding:5px 0;}
.tooltip.left{margin-left:-3px;padding:0 5px;}
.tooltip-inner{max-width:200px;padding:8px;color:#ffffff;text-align:center;text-decoration:none;background-color:#000000;-webkit-border-radius:4px;-moz-border-radius:4px;border-radius:4px;}
.tooltip-arrow{position:absolute;width:0;height:0;border-color:transparent;border-style:solid;}
.tooltip.top .tooltip-arrow{bottom:0;left:50%;margin-left:-5px;border-width:5px 5px 0;border-top-color:#000000;}
.tooltip.right .tooltip-arrow{top:50%;left:0;margin-top:-5px;border-width:5px 5px 5px 0;border-right-color:#000000;}
.tooltip.left .tooltip-arrow{top:50%;right:0;margin-top:-5px;border-width:5px 0 5px 5px;border-left-color:#000000;}
.tooltip.bottom .tooltip-arrow{top:0;left:50%;margin-left:-5px;border-width:0 5px 5px;border-bottom-color:#000000;}
.fade{opacity:0;-webkit-transition:opacity 0.15s linear;-moz-transition:opacity 0.15s linear;-o-transition:opacity 0.15s linear;transition:opacity 0.15s linear;}
.fade.in{opacity:1;}
.tooltip{font-weight: normal;}
.vis.timeline .item .tooltip-inner {max-width: unset !important;}
.vis.timeline .item.dot {position: absolute;padding: 0;border-width: 4px;border-style: solid;border-radius: 4px;}
.vis.timeline .item.box {text-align: center;border-style: solid;border-radius: 2px;}
.vis.timeline .item.line{padding:0;position:absolute;width:0;border-left-width:1px;border-left-style:solid}
.vis.timeline .item.executor.added {background-color: #A0DFFF;border-color: #3EC0FF;}
-->

Details for pure RDD job

• Status: SUCCEEDED
• Completed Stages: 7

Event Timeline

Enable zooming

Completed Stages (7)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Output	Shuffle Read	Shuffle Write
6	collect at AreaTop3ProductRDD.java:353	2019/01/30 15:58:43	94 ms	41/41			235.4 KB
5	mapToPair at AreaTop3ProductRDD.java:259	2019/01/30 15:58:42	0.4 s	41/41			382.9 KB	235.4 KB
4	mapToPair at AreaTop3ProductRDD.java:251	2019/01/30 15:58:42	0.1 s	41/41			99.2 KB	246.0 KB
2	mapToPair at AreaTop3ProductRDD.java:230	2019/01/30 15:58:41	1 s	41/41			765.8 KB	99.2 KB
1	mapToPair at AreaTop3ProductRDD.java:128	2019/01/30 15:58:38	3 s	41/41				750.1 KB
0	mapToPair at AreaTop3ProductRDD.java:164	2019/01/30 15:58:38	3 s	1/1				15.7 KB
3	mapToPair at AreaTop3ProductRDD.java:248	2019/01/30 15:58:38	4 s	41/41				137.0 KB

　　可以看到,产品信息被转换为pairRDD要花4秒,城市信息和点击信息要花3秒.而之前的实验的运行时间却是零点几秒.说明这里可能有自动缓存,把之前的运行结果直接拿来用了

　　这3个步骤是并行的,花的时间也缩小了.运行时间:5秒

Details for pure RDD job with map join

• Status: SUCCEEDED
• Completed Stages: 3

Event Timeline

Enable zooming

Completed Stages (3)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Output	Shuffle Read	Shuffle Write
3	collect at AreaTop3ProductRDDSpeedUp.java:371	2019/01/30 16:00:23	0.2 s	41/41			246.7 KB
2	mapToPair at AreaTop3ProductRDDSpeedUp.java:284	2019/01/30 16:00:22	0.5 s	41/41			477.6 KB	246.8 KB
1	mapToPair at AreaTop3ProductRDDSpeedUp.java:218	2019/01/30 16:00:17	5 s	41/41				478.2 KB

　　估计是map join很占内存的理由,承载城市信息和点击记录的mapToPair运行时间被延长了.运行时间:6秒

Details for original job

• Status: SUCCEEDED
• Completed Stages: 7

Event Timeline

Enable zooming

Completed Stages (7)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Output	Shuffle Read	Shuffle Write
6	collect at AreaTop3ProductSpark.java:108	2019/01/30 16:04:04	0.8 s	200/200			865.5 KB
5	javaRDD at AreaTop3ProductSpark.java:401	2019/01/30 16:03:58	6 s	200/200 (2 failed)			899.9 KB	869.3 KB
3	javaRDD at AreaTop3ProductSpark.java:401	2019/01/30 16:03:56	1 s	200/200			224.2 KB	733.2 KB
2	javaRDD at AreaTop3ProductSpark.java:401	2019/01/30 16:03:55	2 s	41/41			766.0 KB	224.3 KB
4	javaRDD at AreaTop3ProductSpark.java:401	2019/01/30 16:03:50	3 s	41/41				159.9 KB
1	mapToPair at AreaTop3ProductSpark.java:131	2019/01/30 16:03:49	6 s	41/41				750.3 KB
0	mapToPair at AreaTop3ProductSpark.java:178	2019/01/30 16:03:49	3 s	1/1				15.7 KB

　　数据量最多的点击记录mapToPair耗费时间最长,为6秒

　　其他的对应操作耗时都不低于纯RDD版本对应操作,特别是collect前面2个操作,纯RDD程序不用1秒就能跑完.

　　据前面的too many open files错误,可以推定SQL操作是在本地创建文件读写的,加上某些SQL语句对业务处理步骤不如RDD简洁,严重拖慢了运行时间,运行时间:16秒

Details for pure sparkSQL job

• Status: SUCCEEDED
• Completed Stages: 7

Event Timeline

Enable zooming

Completed Stages (7)

Stage Id	Description	Submitted	Duration	Tasks: Succeeded/Total	Input	Output	Shuffle Read	Shuffle Write
6	collect at AreaTop3ProductSql.java:99	2019/01/30 16:08:23	0.8 s	200/200			869.0 KB
5	javaRDD at AreaTop3ProductSql.java:306	2019/01/30 16:08:21	2 s	200/200 (1 failed)			894.1 KB	870.2 KB
3	javaRDD at AreaTop3ProductSql.java:306	2019/01/30 16:08:20	1 s	200/200			224.2 KB	733.4 KB
2	javaRDD at AreaTop3ProductSql.java:306	2019/01/30 16:08:18	1 s	200/200			405.2 KB	224.6 KB
4	javaRDD at AreaTop3ProductSql.java:306	2019/01/30 16:08:01	4 s	41/41				159.9 KB
1	javaRDD at AreaTop3ProductSql.java:306	2019/01/30 16:08:01	17 s	1/1				4.0 KB
0	javaRDD at AreaTop3ProductSql.java:306	2019/01/30 16:08:01	6 s	41/41 (1 failed)				401.8 KB

　　本身sparkSQL就很慢,前面2步操作被SQL化之后更慢了...运行时间:22秒