【异常】SparkSession执行action操作时:

时间:2025-04-12 12:04:41

问题描述:

在之前Spark程序一直没有问题,今天跑的时候就出现找不到类的错误,具体异常栈如下:

写了简单的测试发现,是因为执行了Dataset的action操作会出现下面,并且这个找不到的类属于spark-sql所依赖的jar包,发现3.0.0里面确实没有这个类,然后只好在pom文件里面加上2.7.8的commons-compiler的版本。

<dependency>
  <groupId></groupId>
  <artifactId>commons-compiler</artifactId>
  <version>2.7.8</version>
</dependency>

后面发现是其他同事继承了spring的一些包导致的


org.spark_project.: : org/codehaus/commons/compiler/UncheckedCompileException

	at org.spark_project.$(:2261)
	at org.spark_project.(:4000)
	at org.spark_project.(:4004)
	at org.spark_project.$(:4874)
	at $.compile(:890)
	at (:357)
	at $$anonfun$execute$(:114)
	at $$anonfun$execute$(:114)
	at $$anonfun$executeQuery$(:135)
	at $.withScope(:151)
	at (:132)
	at (:113)
	at (:225)
	at (:272)
	at $$anonfun$collectAsList$1$$anonfun$apply$(:2364)
	at $$anonfun$collectAsList$1$$anonfun$apply$(:2363)
	at $.withNewExecutionId(:57)
	at (:2765)
	at $$anonfun$collectAsList$(:2363)
	at $$anonfun$collectAsList$(:2362)
	at (:2778)
	at (:2362)
	at .initSpark1(:148)
	at .invoke0(Native Method)
	at (:62)
	at (:43)
	at (:498)
	at $(:50)
	at (:12)
	at (:47)
	at (:17)
	at (:26)
	at (:325)
	at .(:78)
	at .(:57)
	at $(:290)
	at $(:71)
	at (:288)
	at $000(:58)
	at $(:268)
	at (:363)
	at (:137)
	at .junit4.(:68)
	at $(:51)
	at (:237)
	at (:70)
Caused by: : org/codehaus/commons/compiler/UncheckedCompileException
	at (:380)
	at (:315)
	at (:233)
	at (:192)
	at (:91)
	at $.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(:935)
	at $$anon$(:998)
	at $$anon$(:995)
	at org.spark_project.$(:3599)
	at org.spark_project.$(:2379)
	at org.spark_project.$(:2342)
	at org.spark_project.$(:2257)
	... 45 more
Caused by: : 
	at (:381)
	at (:424)
	at $(:335)
	at (:357)
	... 57 more


相关文章