Question

我正在解析没有换行符号的CSV文件：

"line1field1", "line1field2", "line1field3", "line2field1", "line2field2", "line2field3", "line3field1", "line3field2", "line3field3"

是否有可能在Spark中有效地做到这一点？（我想获得3行的DataSet，每行3个字段）

Answer 1

如果我理解您的问题，并且您输入的数据没有行分隔符

"line1field1", "line1field2", "line1field3", "line2field1", "line2field2", "line2field3", "line3field1", "line3field2", "line3field3"

你希望输出为

+-------------+-------------+-------------+
|Column1      |Column2      |Column3      |
+-------------+-------------+-------------+
|"line1field1"|"line1field2"|"line1field3"|
|"line2field1"|"line2field2"|"line2field3"|
|"line3field1"|"line3field2"|"line3field3"|
+-------------+-------------+-------------+

以下代码可帮助您实现

val data = sc.textFile("path to the input file")
val todf = data
  .map(line => line.split(",")).map(array => {
      val list = new util.ArrayList[Array[String]]()
      for(index <- 0 to array.length-1 by 3){
        list.add(Array(Try(array(index)) getOrElse "", Try(array(index+1)) getOrElse "", Try(array(index+2)) getOrElse ""))
      }
      list
  })
  .flatMap(a => a.toArray())
  .map(arr => arr.asInstanceOf[Array[String]])
  .map(row => Row.fromSeq(Seq(row(0).trim, row(1).trim, row(2).trim)))

val schema = StructType(Array(StructField("Column1", StringType, true), StructField("Column2", StringType, true),StructField("Column3", StringType, true)))
sqlContext.createDataFrame(todf, schema).show(false)

我希望答案很有帮助

Answer 2

如果你想做一个Spark相关的方式，这应该有效。我刚刚通过resources文件夹导入了csv文件，但是为你存放了它所在的字符串。

import sqlContext.implicits._

val columnNames: Seq[String] = Seq("Col1","Col2","Col3")

sparkContext.textFile(this.getClass.getResource("/test.csv").toString) // your string location here
  .map(x => x.split(',').sliding(3, 3))
  .flatMap(x => x)
  .map(x => x.toList)
  .map { case List(a, b, c) => (a, b, c) } //cleanup needed here to convert to Tuple
  .toDF(columnNames: _*)
  .show(truncate = false)

这会产生：

+-----------+-----------+-----------+
|Col1       |Col2       |Col3       |
+-----------+-----------+-----------+
|line1field1|line1field2|line1field3|
|line2field1|line2field2|line2field3|
|line3field1|line3field2|line3field3|
+-----------+-----------+-----------+

更改滑动以匹配列号将适用于其他大小的列长度。您将需要更改元组映射thoug，因此对于大量列，这可能是不可行的。

您可以查看List to Tuple Answer，了解有关未知大小列表元组的映射。

Spark - 阅读CSV没有新的线路标志

2 个答案: