问题描述:
在之前Spark程序一直没有问题,今天跑的时候就出现找不到类的错误,具体异常栈如下:
写了简单的测试发现,是因为执行了Dataset的action操作会出现下面,并且这个找不到的类属于spark-sql所依赖的jar包,发现3.0.0里面确实没有这个类,然后只好在pom文件里面加上2.7.8的commons-compiler的版本。
<dependency>
<groupId></groupId>
<artifactId>commons-compiler</artifactId>
<version>2.7.8</version>
</dependency>
后面发现是其他同事继承了spring的一些包导致的
org.spark_project.: : org/codehaus/commons/compiler/UncheckedCompileException
at org.spark_project.$(:2261)
at org.spark_project.(:4000)
at org.spark_project.(:4004)
at org.spark_project.$(:4874)
at $.compile(:890)
at (:357)
at $$anonfun$execute$(:114)
at $$anonfun$execute$(:114)
at $$anonfun$executeQuery$(:135)
at $.withScope(:151)
at (:132)
at (:113)
at (:225)
at (:272)
at $$anonfun$collectAsList$1$$anonfun$apply$(:2364)
at $$anonfun$collectAsList$1$$anonfun$apply$(:2363)
at $.withNewExecutionId(:57)
at (:2765)
at $$anonfun$collectAsList$(:2363)
at $$anonfun$collectAsList$(:2362)
at (:2778)
at (:2362)
at .initSpark1(:148)
at .invoke0(Native Method)
at (:62)
at (:43)
at (:498)
at $(:50)
at (:12)
at (:47)
at (:17)
at (:26)
at (:325)
at .(:78)
at .(:57)
at $(:290)
at $(:71)
at (:288)
at $000(:58)
at $(:268)
at (:363)
at (:137)
at .junit4.(:68)
at $(:51)
at (:237)
at (:70)
Caused by: : org/codehaus/commons/compiler/UncheckedCompileException
at (:380)
at (:315)
at (:233)
at (:192)
at (:91)
at $.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(:935)
at $$anon$(:998)
at $$anon$(:995)
at org.spark_project.$(:3599)
at org.spark_project.$(:2379)
at org.spark_project.$(:2342)
at org.spark_project.$(:2257)
... 45 more
Caused by: :
at (:381)
at (:424)
at $(:335)
at (:357)
... 57 more