断言失败:运行时反射Universe中的不安全符号DeveloperApi

时间:2016-09-21 15:25:56

标签: java scala apache-spark

我在Java 8程序中运行以下行

SparkConf sparkConf = new SparkConf();
sparkConf.setAppName("testJob");
sparkConf.setMaster("spark://blahblah:7077");
SparkSession sparkSession = SparkSession.builder().config(sparkConf).getOrCreate();
SQLContext sqlContext = new SQLContext(sparkSession);
Dataset<Row> ds = sqlContext.sql("SHOW TABLES");

我得到了一个非常神秘的例外(运行sqlContext.sql("SHOW TABLES")行)

java.lang.RuntimeException: error reading Scala signature of
    org.apache.spark.sql.package: assertion failed: unsafe symbol 
    DeveloperApi (child of package annotation) in runtime reflection universe

(在这个问题的最后报告了完整的堆栈跟踪)。

任何人都可以给我一个如何解决这个问题的提示吗?这是什么意思?

[Gogo shell] INFO org.apache.spark.util.Utils - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 51077.
[Gogo shell] INFO org.apache.spark.network.netty.NettyBlockTransferService - Server created on 172.29.30.215:51077
[Gogo shell] INFO org.apache.spark.storage.BlockManagerMaster - Registering BlockManager BlockManagerId(driver, 172.29.30.215, 51077)
[dispatcher-event-loop-0] INFO org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint - Executor updated: app-20160921162135-0005/0 is now RUNNING
[dispatcher-event-loop-3] INFO org.apache.spark.storage.BlockManagerMasterEndpoint - Registering block manager 172.29.30.215:51077 with 2004.6 MB RAM, BlockManagerId(driver, 172.29.30.215, 51077)
[Gogo shell] INFO org.apache.spark.storage.BlockManagerMaster - Registered BlockManager BlockManagerId(driver, 172.29.30.215, 51077)
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@18e8216b{/metrics/json,null,AVAILABLE}
[Gogo shell] INFO org.apache.spark.scheduler.cluster.StandaloneSchedulerBackend - SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
[Gogo shell] WARN org.apache.spark.SparkContext - Use an existing SparkContext, some configuration may not take effect.
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5b855aa1{/SQL,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@5df316cb{/SQL/json,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@78a44f1f{/SQL/execution,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@7e3bbe8{/SQL/execution/json,null,AVAILABLE}
[Gogo shell] INFO org.spark_project.jetty.server.handler.ContextHandler - Started o.s.j.s.ServletContextHandler@1476f413{/static/sql,null,AVAILABLE}
[Gogo shell] INFO org.apache.spark.sql.internal.SharedState - Warehouse path is 'file:/Users/matteorulli/git/nonono/com.nonono.osgi.example.int/com.nonono.osgi.example.int.etl/spark-warehouse'.
[Gogo shell] INFO org.apache.spark.sql.execution.SparkSqlParser - Parsing command: SHOW TABLES
java.lang.AssertionError: assertion failed: unsafe symbol DeveloperApi (child of package annotation) in runtime reflection universe
    at scala.reflect.internal.Symbols$Symbol.<init>(Symbols.scala:184)
    at scala.reflect.internal.Symbols$TypeSymbol.<init>(Symbols.scala:3009)
    at scala.reflect.internal.Symbols$ClassSymbol.<init>(Symbols.scala:3201)
    at scala.reflect.internal.Symbols$StubClassSymbol.<init>(Symbols.scala:3496)
    at scala.reflect.internal.Symbols$Symbol.newStubSymbol(Symbols.scala:498)
    at scala.reflect.internal.pickling.UnPickler$Scan.readExtSymbol$1(UnPickler.scala:258)
    at scala.reflect.internal.pickling.UnPickler$Scan.readSymbol(UnPickler.scala:284)
    at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolRef(UnPickler.scala:649)
    at scala.reflect.internal.pickling.UnPickler$Scan.readType(UnPickler.scala:417)
    at scala.reflect.internal.pickling.UnPickler$Scan$$anonfun$readTypeRef$1.apply(UnPickler.scala:658)
    at scala.reflect.internal.pickling.UnPickler$Scan$$anonfun$readTypeRef$1.apply(UnPickler.scala:658)
    at scala.reflect.internal.pickling.UnPickler$Scan.at(UnPickler.scala:179)
    at scala.reflect.internal.pickling.UnPickler$Scan.readTypeRef(UnPickler.scala:658)
    at scala.reflect.internal.pickling.UnPickler$Scan.readAnnotationInfo(UnPickler.scala:492)
    at scala.reflect.internal.pickling.UnPickler$Scan.readSymbolAnnotation(UnPickler.scala:515)
    at scala.reflect.internal.pickling.UnPickler$Scan.run(UnPickler.scala:97)
    at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:38)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:619)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply$mcV$sp(SymbolLoaders.scala:28)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:25)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.info(SynchronizedSymbols.scala:189)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:286)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
    at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
    at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
    at scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
    at scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
    at scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
    at scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
    at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
    at scala.reflect.internal.Types$Type.member(Types.scala:600)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
    at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:77)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:122)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
    at org.apache.spark.sql.types.StringType$$typecreator1$1.apply(StringType.scala:37)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
    at org.apache.spark.sql.types.AtomicType.<init>(AbstractDataType.scala:136)
    at org.apache.spark.sql.types.StringType.<init>(StringType.scala:32)
    at org.apache.spark.sql.types.StringType$.<init>(StringType.scala:48)
    at org.apache.spark.sql.types.StringType$.<clinit>(StringType.scala)
    at org.apache.spark.sql.execution.command.ShowTablesCommand.<init>(tables.scala:556)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:123)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:53)
    at org.apache.spark.sql.catalyst.parser.SqlBaseParser$ShowTablesContext.accept(SqlBaseParser.java:844)
    at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:63)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:54)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:82)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:46)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
    at com.nonono.osgi.example.int.etl.command.EtlCommand.etl(EtlCommand.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Console.run(Console.java:62)
    at org.apache.felix.gogo.shell.Shell.console(Shell.java:203)
    at org.apache.felix.gogo.shell.Shell.gosh(Shell.java:128)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Activator.run(Activator.java:75)
    at java.lang.Thread.run(Thread.java:745)
java.lang.ExceptionInInitializerError
    at org.apache.spark.sql.execution.command.ShowTablesCommand.<init>(tables.scala:556)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder$$anonfun$visitShowTables$1.apply(SparkSqlParser.scala:124)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:123)
    at org.apache.spark.sql.execution.SparkSqlAstBuilder.visitShowTables(SparkSqlParser.scala:53)
    at org.apache.spark.sql.catalyst.parser.SqlBaseParser$ShowTablesContext.accept(SqlBaseParser.java:844)
    at org.antlr.v4.runtime.tree.AbstractParseTreeVisitor.visit(AbstractParseTreeVisitor.java:42)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.AstBuilder$$anonfun$visitSingleStatement$1.apply(AstBuilder.scala:64)
    at org.apache.spark.sql.catalyst.parser.ParserUtils$.withOrigin(ParserUtils.scala:96)
    at org.apache.spark.sql.catalyst.parser.AstBuilder.visitSingleStatement(AstBuilder.scala:63)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:54)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser$$anonfun$parsePlan$1.apply(ParseDriver.scala:53)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:82)
    at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:46)
    at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parsePlan(ParseDriver.scala:53)
    at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
    at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
    at com.nonono.osgi.example.int.etl.command.EtlCommand.etl(EtlCommand.java:34)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Console.run(Console.java:62)
    at org.apache.felix.gogo.shell.Shell.console(Shell.java:203)
    at org.apache.felix.gogo.shell.Shell.gosh(Shell.java:128)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.felix.gogo.runtime.Reflective.invoke(Reflective.java:137)
    at org.apache.felix.gogo.runtime.CommandProxy.execute(CommandProxy.java:82)
    at org.apache.felix.gogo.runtime.Closure.executeCmd(Closure.java:480)
    at org.apache.felix.gogo.runtime.Closure.executeStatement(Closure.java:406)
    at org.apache.felix.gogo.runtime.Pipe.run(Pipe.java:108)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:182)
    at org.apache.felix.gogo.runtime.Closure.execute(Closure.java:119)
    at org.apache.felix.gogo.runtime.CommandSessionImpl.execute(CommandSessionImpl.java:94)
    at org.apache.felix.gogo.shell.Activator.run(Activator.java:75)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: error reading Scala signature of org.apache.spark.sql.package: assertion failed: unsafe symbol DeveloperApi (child of package annotation) in runtime reflection universe
    at scala.reflect.internal.pickling.UnPickler.unpickle(UnPickler.scala:46)
    at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:619)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply$mcV$sp(SymbolLoaders.scala:28)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter$$anonfun$complete$1.apply(SymbolLoaders.scala:25)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:25)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:189)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$10.info(SynchronizedSymbols.scala:189)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:286)
    at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:341)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply$mcV$sp(SymbolLoaders.scala:74)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType$$anonfun$complete$2.apply(SymbolLoaders.scala:71)
    at scala.reflect.internal.SymbolTable.slowButSafeEnteringPhaseNotLaterThan(SymbolTable.scala:263)
    at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:71)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1514)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.scala$reflect$runtime$SynchronizedSymbols$SynchronizedSymbol$$super$info(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anonfun$info$1.apply(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
    at scala.reflect.runtime.JavaUniverse.gilSynchronized(JavaUniverse.scala:16)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:123)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.gilSynchronizedIfNotThreadsafe(SynchronizedSymbols.scala:174)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$class.info(SynchronizedSymbols.scala:127)
    at scala.reflect.runtime.SynchronizedSymbols$SynchronizedSymbol$$anon$1.info(SynchronizedSymbols.scala:174)
    at scala.reflect.internal.Types$TypeRef.thisInfo(Types.scala:2194)
    at scala.reflect.internal.Types$TypeRef.baseClasses(Types.scala:2199)
    at scala.reflect.internal.tpe.FindMembers$FindMemberBase.<init>(FindMembers.scala:17)
    at scala.reflect.internal.tpe.FindMembers$FindMember.<init>(FindMembers.scala:219)
    at scala.reflect.internal.Types$Type.scala$reflect$internal$Types$Type$$findMemberInternal$1(Types.scala:1014)
    at scala.reflect.internal.Types$Type.findMember(Types.scala:1016)
    at scala.reflect.internal.Types$Type.memberBasedOnName(Types.scala:631)
    at scala.reflect.internal.Types$Type.member(Types.scala:600)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:45)
    at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:66)
    at scala.reflect.internal.Mirrors$RootsBase.staticModuleOrClass(Mirrors.scala:77)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:122)
    at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
    at org.apache.spark.sql.types.StringType$$typecreator1$1.apply(StringType.scala:37)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe$lzycompute(TypeTags.scala:232)
    at scala.reflect.api.TypeTags$WeakTypeTagImpl.tpe(TypeTags.scala:232)
    at org.apache.spark.sql.types.AtomicType.<init>(AbstractDataType.scala:136)
    at org.apache.spark.sql.types.StringType.<init>(StringType.scala:32)
    at org.apache.spark.sql.types.StringType$.<init>(StringType.scala:48)
    at org.apache.spark.sql.types.StringType$.<clinit>(StringType.scala)
    ... 49 more

1 个答案:

答案 0 :(得分:4)

事实证明,类路径中缺少包含DeveloperApi类的spark jar,并且类没有包含在spark包中。

所以,在这种情况下,unsafe symbol DeveloperApi in runtime reflection universe引导到更平凡的ClassNotFoundException DeveloperApi:在例外情况下,消息java接缝比scala更简洁。

相关问题