Question

我有一个这样的连通图

user1|A,C,B
user2|A,E,B,A
user3|C,B,A,B,E
user4|A,C,B,E,B

其中user是属性名称，并遵循该特定用户的路径。例如

user1 the  path is A->C->B
user2: A->E->B->A
user3: C->B->A->B->E
user4: A->C->B->E->B

现在，我想查找所有从A到达E的用户。输出应为 user2，user3，user4（因为所有这些用户最终都从A到达E，无论他们跳了多少跳）。我如何为此写主题。这就是我尝试过的。

val vertices=spark.createDataFrame(List(("A","Billing"),("B","Devices"),("C","Payment"),("D","Data"),("E","Help"))).toDF("id","desc")

val edges = spark.createDataFrame(List(("A","C","user1"),
("C","B","user1"),
("A","E","user2"),
("E","B","user2"),
("B","A","user2"),
("C","B","user3"),
("B","A","user3"),
("A","B","user3"),
("B","E","user3"),
("A","C","user4"),
("C","B","user4"),
("B","E","user4"),
("E","B","user4"))).toDF("src","dst","user")

val pathAnalysis=GraphFrame(vertices,edges)
pathAnalysis.find("(a)-[]->();()-[]->();()-[]->(d)").filter("a.id='A'").filter("d.id='E'").distinct().show()

但是我遇到了这样的异常

org.apache.spark.sql.AnalysisException: Detected implicit cartesian product for INNER join between logical plans
Join Inner
:- Project [a#355]
:  +- Join Inner, (__tmp-4363599943432734077#353.src = a#355.id)
:     :- LocalRelation [__tmp-4363599943432734077#353]
:     +- Project [named_struct(id, _1#0, desc, _2#1) AS a#355]
:        +- Filter (named_struct(id, _1#0, desc, _2#1).id = A)
:           +- LocalRelation [_1#0, _2#1]
+- LocalRelation
and
LocalRelation [__tmp-1043886091038848698#371]
Join condition is missing or trivial.
Either: use the CROSS JOIN syntax to allow cartesian products between these
relations, or: enable implicit cartesian products by setting the configuration
variable spark.sql.crossJoin.enabled=true;

我不确定我的条件是否正确或如何设置此属性 spark.sql.crossJoin.enabled=true位于火花壳上

我如下调用了spark-shell

spark-shell --packages graphframes:graphframes:0.3.0-spark2.0-s_2.11

Answer 1

我建议的解决方案有点琐碎，但是如果路径相对较短，并且用户数（即数据集中的行数）很大，则可以很好地解决问题。如果不是这种情况，请告诉我，其他实现也是可能的。

case class UserPath(
  userId: String,
  path: List[String])
val dsUsers = Seq(
  UserPath("user1", List("A", "B", "C")), 
  UserPath("user2", List("A", "E", "B", "A")))
.doDF.as[UserPath]

def pathExists(up: UserPath): Option[String] = {
  val prefix = up.path.takeWhile(s => s != "A")
  val len = up.path.length
  if (up.path.takeRight(len - prefix.length).contains("E"))
    Some(up.userId)
  else
    None
}
// Users with path from A -> E.
dsUsers.map(pathExists).filter(opt => !opt.isEmpty)

Answer 2

您还可以对其使用BFS算法：http://graphframes.github.io/graphframes/docs/_site/api/scala/index.html#org.graphframes.lib.BFS 使用数据模型，您将必须遍历用户并为每个用户运行BFS，如下所示：

scala> pathAnalysis.bfs.fromExpr($"id" === "A").toExpr($"id" === "E").edgeFilter($"user" === "user3").run().show
+------------+-------------+------------+-------------+---------+
|        from|           e0|          v1|           e1|       to|
+------------+-------------+------------+-------------+---------+
|[A, Billing]|[A, B, user3]|[B, Devices]|[B, E, user3]|[E, Help]|
+------------+-------------+------------+-------------+---------+

查找两个不直接连接的顶点之间的路径

2 个答案: