java.lang.IllegalArgumentException : System memory 259522560 must be at least 471859200.

时间:2024-03-24 14:52:04

Exception in thread "main" java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.


java.lang.IllegalArgumentException : System memory 259522560 must be at least 471859200.



问题:

JVM内存不够, 无法启动sc



解决:

通过SparkContext增加内存 > least 471859200

eg:

val conf = new SparkConf().setAppName("AggregateOperator").setMaster("local").set("spark.testing.memory", "512000000")