Question

我正在尝试为给定链接中的spark本身提供的Spark kinesis流scala应用程序编译和创建一个jar。 Kinesis word count sample

以下是我的sbt文件，它具有所有依赖项，并且对于简单程序可以编译。

name := "stream parser"

version := "1.0"

val sparkVersion = "2.0.2"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core"  % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming"  % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql"  % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming-kinesis-asl"  % sparkVersion % "provided"
)

但是在我的Ubuntu系统中编译时，kinesis样本会抛出以下错误。

trait Logging in package internal cannot be accessed in package org.apache.spark.internal
[error] object KinesisWordCountASL extends Logging {
[error]                                    ^

记录类导入

import org.apache.spark.internal.Logging

任何想法可能是什么问题？

Answer 1

根据spark repo 2.0及以上版本记录类包含在package org.apache.spark.internal spark中。在以前的版本中它位于package org.apache.spark我建议您运行示例设置spark 2.0.0或后来。或者根据适当的库的使用情况进行更改。

与build.sbt编译时没有任何问题的相同示例

scalaVersion := "2.10.6"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.0.2",
  "org.apache.spark" %% "spark-sql" % "2.0.2",
  "org.apache.spark" %% "spark-streaming" % "2.0.2",
  "org.apache.spark" %% "spark-streaming-kinesis-asl" % "2.0.2",
  "com.amazonaws" % "amazon-kinesis-producer" % "0.10.2"
)

使用SBT工具编译Spark流kinesis示例应用程序

1 个答案: