一个执行器在火花流任务中运行的时间比所有其他执行器长得多

I'm integrating spark streaming with kafka, in one of the stages, one executor runs much slower than the other..

我正在将火花流与kafka集成,在其中一个阶段,一个执行器运行速度比另一个慢得多。

you could find in the picture, h10.zw runs 2.6 min, and the "task time" is 52min, which is much longer than other executors. But the shuffle read size/ shuffle write size is the same as others.

你可以在图片中找到,h10.zw运行2.6分钟,“任务时间”是52分钟,这比其他执行器长得多。但是随机读取大小/随机写入大小与其他大小相同。

I wonder what is the "task time"? What is the h10.zw executor doing? How to balance the running time of all executors to avoid time skew？

我想知道什么是“任务时间”?什么是h10.zw执行者在做什么?如何平衡所有执行程序的运行时间以避免时间偏差?

1 个解决方案

#1

This may, depending on your exact processing, be due to data skew. Try speculative execution and changing your partitioning to smaller partitions. This should help determining if this is the case.

根据您的确切处理,这可能是由于数据偏差造成的。尝试推测执行并将分区更改为较小的分区。这应该有助于确定是否是这种情况。