Question

运行CDH 5.14.4群集以及Spark 2.3 Release 4 CDS。从Spark应用程序（客户端模式）提交相当大的Spark SQL作业时，在少数执行程序上面临一些java.lang.OutOfMemoryError错误。该错误与基础数据量无关，因为我能够看到100条或100万条记录的错误。 SQL非常庞大，因为它执行大量复杂的条件语句和表达式。

但是，当我以本地模式运行应用程序而不是用yarn作为主服务器时，执行会顺利进行。

在此处请求社区的即时帮助。

#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill %p"
# Executing /bin/sh -c "kill 4505"...
15:20:57.430 [SIGTERM handler] ERROR org.apache.spark.executor.CoarseGrainedExecutorBackend - RECEIVED SIGNAL TERM
15:20:57.435 [Thread-2] INFO org.apache.spark.storage.DiskBlockManager - Shutdown hook called
15:20:57.457 [Executor task launch worker for task 13466] ERROR org.apache.spark.executor.Executor - Exception in task 118.0 in stage 146.1 (TID 13466)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuilder.append(StringBuilder.java:136)
at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:200)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:113)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:112)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:112)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:104)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:137)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:138)
at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.logDebug(RuleExecutor.scala:40)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:134)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:76)


#
# java.lang.OutOfMemoryError: Java heap space
# -XX:OnOutOfMemoryError="kill %p"
# Executing /bin/sh -c "kill 8562"...
15:21:11.409 [SIGTERM handler] ERROR org.apache.spark.executor.CoarseGrainedExecutorBackend - RECEIVED SIGNAL TERM
15:21:11.420 [Thread-2] INFO org.apache.spark.storage.DiskBlockManager - Shutdown hook called
15:21:11.454 [Executor task launch worker for task 13470] ERROR org.apache.spark.executor.Executor - Exception in task 129.0 in stage 146.1 (TID 13470)
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:207)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at scala.collection.mutable.StringBuilder.toString(StringBuilder.scala:430)
at scala.collection.immutable.StringLike$class.$times(StringLike.scala:71)
at scala.collection.immutable.StringOps.$times(StringOps.scala:29)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:113)
at org.apache.spark.sql.catalyst.util.package$$anonfun$sideBySide$1.apply(package.scala:112)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:112)
at org.apache.spark.sql.catalyst.util.package$.sideBySide(package.scala:104)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:137)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$5.apply(RuleExecutor.scala:138)
at org.apache.spark.internal.Logging$class.logDebug(Logging.scala:58)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.logDebug(RuleExecutor.scala:40)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:134)

Answer 1

update：此问题可以标记为已关闭。原来，此问题是驱动程序（客户端应用程序）和执行程序（yarn.staging.dir）之间的火花/混合罐不匹配。错误令人误解。

带有大型SparkSQL查询的Spark 2.3内存泄漏

1 个答案: