我有一个特殊的用例,其中我为同一位客户有多行,每行对象看起来像:
root
-c1: BigInt
-c2: String
-c3: Double
-c4: Double
-c5: Map[String, Int]
现在我按列c1进行分组,并为同一客户收集所有行作为列表,例如:
c1, [Row1, Row3, Row4]
c2, [Row2, Row5]
我尝试过这种方式
dataset.withColumn("combined", array("c1","c2","c3","c4","c5")).groupBy("c1").agg(collect_list("combined"))
,但出现异常:
Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve 'array(`c1`, `c2`, `c3`, `c4`, `c5`)' due to data type mismatch: input to function array should all be the same type, but it's [bigint, string, double, double, map<string,map<string,double>>];;
答案 0 :(得分:4)
您可以使用array
函数代替struct
来合并列,并使用groupBy
和collect_list
聚合函数作为
import org.apache.spark.sql.functions._
df.withColumn("combined", struct("c1","c2","c3","c4","c5"))
.groupBy("c1").agg(collect_list("combined").as("combined_list"))
.show(false)
,这样您就可以将schema
作为
root
|-- c1: integer (nullable = false)
|-- combined_list: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- c1: integer (nullable = false)
| | |-- c2: string (nullable = true)
| | |-- c3: string (nullable = true)
| | |-- c4: string (nullable = true)
| | |-- c5: map (nullable = true)
| | | |-- key: string
| | | |-- value: integer (valueContainsNull = false)
我希望答案会有所帮助
答案 1 :(得分:0)
如果您希望结果包含Row
的集合,请考虑如下转换为RDD:
import org.apache.spark.sql.functions._
import org.apache.spark.sql.Row
def df = Seq(
(BigInt(10), "x", 1.0, 2.0, Map("a"->1, "b"->2)),
(BigInt(10), "y", 3.0, 4.0, Map("c"->3)),
(BigInt(20), "z", 5.0, 6.0, Map("d"->4, "e"->5))
).
toDF("c1", "c2", "c3", "c4", "c5").
// as[(BigInt, String, Double, Double, Map[String, Int])]
df.rdd.map(r => (r.getDecimal(0), r)).groupByKey.collect
// res1: Array[(java.math.BigDecimal, Iterable[org.apache.spark.sql.Row])] = Array(
// (10,CompactBuffer([10,x,1.0,2.0,Map(a -> 1, b -> 2)], [10,y,3.0,4.0,Map(c -> 3)])),
// (20,CompactBuffer([20,z,5.0,6.0,Map(d -> 4, e -> 5)]))
// )
或者,如果您对DataFrame中的struct
型行的集合比较满意,则可以采用另一种方法:
val cols = ds.columns
df.groupBy("c1").agg(collect_list(struct(cols.head, cols.tail: _*)).as("row_list")).
show(false)
// +---+----------------------------------------------------------------+
// |c1 |row_list |
// +---+----------------------------------------------------------------+
// |20 |[[20,z,5.0,6.0,Map(d -> 4, e -> 5)]] |
// |10 |[[10,x,1.0,2.0,Map(a -> 1, b -> 2)], [10,y,3.0,4.0,Map(c -> 3)]]|
// +---+----------------------------------------------------------------+