Question

我是Spark Machine Learning（2天）的新手我在Spark Shell中执行以下代码我试图预测一些值我看到Stackoverflow中提供的这个错误帖子但是我无法修复我的代码正确的解决方案，所以再次发布问题为同样道歉

输入数据：

1.00,1.00,9.00
1.00,2.00,10.00
1.00,3.00,9.00
1.00,4.00,9.00
1.00,5.00,9.00
1.00,6.00,9.45
1.00,7.00,9.45
1.00,8.00,9.45
1.00,9.00,9.45

代码：

val df = spark.read.csv("/root/Predictiondata.csv").toDF("Userid", "Date", "Intime")
import org.apache.spark.sql.types.DoubleType
val featureDf = df.select( df("Userid").cast(DoubleType).as("Userid"),df("Date").cast(DoubleType).as("Date"),df("Intime").cast(DoubleType).as("Intime")).toDF()
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
val data = featureDf.select("Userid","Date","Intime").map(r => LabeledPoint(r(0).toString.toDouble,Vectors.dense(r(1).toString.toDouble,r(2).toString.toDouble))).toDF()
import org.apache.spark.ml.regression.LinearRegression
val lr = new LinearRegression()
val lrModel = lr.fit(data)

错误：

 scala> val lrModel = lr.fit(data)
 java.lang.IllegalArgumentException: requirement failed: Column features must be of type org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7 but was actually org.apache.spark.mllib.linalg.VectorUDT@f71b0bce.
 at scala.Predef$.require(Predef.scala:224)
 at org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:42)
 at org.apache.spark.ml.PredictorParams$class.validateAndTransformSchema(Predictor.scala:51)
 at org.apache.spark.ml.Predictor.validateAndTransformSchema(Predictor.scala:72)
 at org.apache.spark.ml.Predictor.transformSchema(Predictor.scala:122)
 at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
 at org.apache.spark.ml.Predictor.fit(Predictor.scala:90)
 ... 48 elided

非常感谢任何帮助或建议。

提前致谢

Answer 1

请将Spark 2+与 DataFrame API 一起用于VectorAssembler

像这样的东西（没有经过测试）：

import spark.implicits._

val data = spark.read
    .option("inferSchema", true)
    .csv("/root/Predictiondata.csv")
    .toDF("Userid", "Date", "Intime")

val dataWithFeatures = new VectorAssembler()
    .setInputCols(Array("Date", "Intime"))
    .transform(data)

val dataWithLabelFeatures = dataWithFeatures        
    .withColumn("label", $"Userid")

val lrModel = new LinearRegression().fit(dataWithLabelFeatures)

另外，请查看Pipeline

Answer 2

如果你的 Spark是＆gt; 2.x 导入

request

而不是

org.apache.spark.ml.linalg.VectorUDT

java.lang.IllegalArgumentException：要求失败：列要素的类型必须为org.apache.spark.ml.linalg.VectorUDT

2 个答案: