Question

我无法在Intellij for Spark scala项目中导入spark mllib 库。我收到解决方案例外。

下面是我的sbt.build

name := "ML_Spark"  

version := "0.1" 

scalaVersion := "2.11.12"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.1"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.2.1" % "runtime"

Answer 1

我试图复制/粘贴您提供的相同build.sbt文件，但出现以下错误：

[error] [/Users/pc/testsbt/build.sbt]:3: ';' expected but string literal found.

实际上，build.sbt无效： intellij error

将版本和Scala版本放在不同的行为我解决了这个问题：

name := "ML_Spark"  

version := "0.1" 
scalaVersion := "2.11.12"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.1"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1"
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.2.1" % "runtime"

我不确定这是您要面对的问题（能否请您分享您遇到的例外情况？），这可能是您在主目录中.sbt文件夹下指定的存储库有问题。

Answer 2

我以前遇到过同样的问题。为了解决这个问题，我只使用了mllib的编译的版本，而不是运行时的版本。这是我的conf：

name := "SparkDemo"

version := "0.1"

scalaVersion := "2.11.12"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-mllib
libraryDependencies += "org.apache.spark" %% "spark-mllib" % "2.3.0"

Answer 3

我遇到了类似的问题，但是找到了解决方法。即，您必须手动将spark-mllib jar文件添加到您的项目中。确实，尽管我的build.sbt文件是

name := "example_project"

version := "0.1"
scalaVersion := "2.12.10"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "3.0.0",
  "org.apache.spark" %% "spark-sql" % "3.0.0",
  "org.apache.spark" %% "spark-mllib" % "3.0.0" % "runtime"
)

我无法使用以下命令导入spark库

import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.ml._

对我有用的解决方案是手动添加jar文件。具体来说，

下载所需的ml库的jar文件（例如，对于spark 3，请使用https://mvnrepository.com/artifact/org.apache.spark/spark-mllib_2.12/3.0.0）。
点击此链接可将jar文件添加到您的intelliJ项目：Correct way to add external jars (lib/*.jar) to an IntelliJ IDEA project
还要添加本地mlib jar（https://mvnrepository.com/artifact/org.apache.spark/spark-mllib-local）

如果由于某种原因再次编译build.sbt，则需要再次重新导入jar文件。

无法在IntelliJ中导入spark mllib

3 个答案: