线程“main”中的异常java.lang.IllegalArgumentException：实例化'org.apache.spark.sql.hive.HiveSessionState'时出错：

时间：2018-05-03 04:27:28

标签： scala apache-spark hadoop hive apache-spark-2.0

我正在尝试通过Intelliji连接到Hive。我使用的是Scala版本2.11.4，spark-core，spark-hive，spark-sql的版本是2.1.1。这是我用来从我的windows m / c远程连接的代码snippt。在连接时我遇到以下错误，有人可以帮我解决这个问题吗？

注意：当我读取一些线程时，他们提到了检查tmp的权限，在本例中是/ tmp / hive / warehouse。它对我用来连接的用户xyz有适当的权限。使用此功能ID，我可以从其中一个unix服务器手动连接。我甚至尝试过使用spark.sql（“show databases”），但这是同样的错误。

def main(args: Array[String]): Unit = {
    createKerberosTicket()
    val spark: SparkSession = {
          SparkSession
            .builder()
            .master("local")
            .appName("SparkHiveTest")
            .config("hive.exec.dynamic.partition.mode", "nonstrict")
            .config("hive.exec.dynamic.partition", "true")
            .config("mapreduce.job.queuename", "root.XYZ_Pool")
            .enableHiveSupport()
            .getOrCreate()
        }
    spark.sparkContext.hadoopConfiguration.addResource(new Path("core-site.xml"))
    spark.sparkContext.hadoopConfiguration.addResource(new Path("hdfs-site.xml"))
    spark.sparkContext.hadoopConfiguration.addResource(new Path("hive-site.xml"))
    spark.sparkContext.hadoopConfiguration.set("fs.hdfs.impl", classOf[DistributedFileSystem].getName)
    spark.sparkContext.hadoopConfiguration.set("fs.file.impl", classOf[LocalFileSystem].getName)
    val listOfDBs = spark.sqlContext.sql("show databases")
}

18/05/02 23:59:13 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/tmp/hive/warehouse').
18/05/02 23:59:13 INFO SharedState: Warehouse path is '/tmp/hive/warehouse'.
18/05/02 23:59:14 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/05/02 23:59:14 INFO metastore: Trying to connect to metastore with URI thrift://xyz.net:1234
18/05/02 23:59:14 INFO metastore: Connected to metastore.
18/05/02 23:59:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
    at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981)
    at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
    at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878)
    at org.apache.spark.sql.SparkSession$Builder$$anonfun$getOrCreate$5.apply(SparkSession.scala:878)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
    at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
    at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
    at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:878)
    at spark.SparkPlusHive$.main(SparkPlusHive.scala:25)
    at spark.SparkPlusHive.main(SparkPlusHive.scala)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:978)
    ... 12 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
    at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:169)
    at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
    at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
    at org.apache.spark.sql.SparkSession$$anonfun$sharedState$1.apply(SparkSession.scala:101)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
    at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
    at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
    at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
    ... 17 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$$reflect(SharedState.scala:166)
    ... 25 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:264)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:358)
    at org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:262)
    at org.apache.spark.sql.hive.HiveExternalCatalog.<init>(HiveExternalCatalog.scala:66)
    ... 30 more
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
    ... 38 more
Caused by: java.lang.NullPointerException
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
    at org.apache.hadoop.util.Shell.run(Shell.java:478)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:831)
    at org.apache.hadoop.util.Shell.execCommand(Shell.java:814)
    at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:712)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:470)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:510)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:488)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:309)
    at org.apache.hadoop.hive.ql.session.SessionState.createPath(SessionState.java:639)
    at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:567)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:508)
    ... 39 more

2 个答案:

答案 0 :(得分：0)

  

18/05/02 23:59:13 INFO SharedState：spark.sql.warehouse.dir不是   集，

很明显，您的from tkinter import * root = Tk() root.geometry('200x200') def click(event): print(event.x,event.y) root.bind('<Button-1>', click) root.mainloop() 未设置，请将其设置为删除上述问题

import tkinter as tk
import random

def on_click(event):
    current = event.widget.find_withtag("current")
    if current:
        item = current[0]
        color = canvas.itemcget(item, "fill")
        label.configure(text="you clicked on item with id %s (%s)" % (item, color))
    else:
        label.configure(text="You didn't click on an item")

root = tk.Tk()
label = tk.Label(root, anchor="w")
canvas = tk.Canvas(root, background="bisque", width=400, height=400)
label.pack(side="top", fill="x")
canvas.pack(fill="both", expand=True)

for color in ("red", "orange", "yellow", "green", "blue", "violet"):
    x0 = random.randint(50, 350)
    y0 = random.randint(50, 350)
    canvas.create_rectangle(x0, y0, x0+50, y0+50, outline="black", fill=color)
    canvas.bind('<ButtonPress-1>', on_click)

root.mainloop()

希望这可以帮助你

答案 1 :(得分：0)

在我的情况下，它是通过遵循here来完成的。

  • 下载winutils.exe。
  • 设置环境变量HADOOP_HOME。
  • 将dir'C：\ tmp \ hive'用作Spark属性值'spark.sql.warehouse.dir'。
  • 将目录“ C：\ tmp \ hive”的权限更改为777。
相关问题
最新问题