Question

运行spark-shell时出现以下错误

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/01/30 18:22:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/01/30 18:22:29 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
18/01/30 18:22:29 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.

    java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder':

    Caused by: scala.MatchError: 2.3.0 (of class java.lang.String)
    at 

org.apache.spark.sql.hive.client.IsolatedClientLoader$.hiveVersion(IsolatedClientLoader.scala:89)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:279)
at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:266)
at org.apache.spark.sql.hive.HiveExternalCatalog.client$lzycompute(HiveExternalCatalog.scala:66)
at org.apache.spark.sql.hive.HiveExternalCatalog.client(HiveExternalCatalog.scala:65)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:194)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:194)
at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97)
at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:193)
at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:105)
at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:93)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.externalCatalog(HiveSessionStateBuilder.scala:39)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog$lzycompute(HiveSessionStateBuilder.scala:54)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:52)
at org.apache.spark.sql.hive.HiveSessionStateBuilder.catalog(HiveSessionStateBuilder.scala:35)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1050)
... 61 more
<console>:14: error: not found: value spark
import spark.implicits._
^
<console>:14: error: not found: value spark
import spark.sql
^

Answer 1

今天（Spark 2.3.0 RC）最新支持的版本是2.1。 Relevant code：

def hiveVersion(version: String): HiveVersion = version match {
  case "12" | "0.12" | "0.12.0" => hive.v12
  case "13" | "0.13" | "0.13.0" | "0.13.1" => hive.v13
  case "14" | "0.14" | "0.14.0" => hive.v14
  case "1.0" | "1.0.0" => hive.v1_0
  case "1.1" | "1.1.0" => hive.v1_1
  case "1.2" | "1.2.0" | "1.2.1" | "1.2.2" => hive.v1_2
  case "2.0" | "2.0.0" | "2.0.1" => hive.v2_0
  case "2.1" | "2.1.0" | "2.1.1" => hive.v2_1
}

不支持Hive 2.3。

Apache spark 2.2.0升级hive Metastore后无法连接到Metastore

1 个答案: