您可以使用以下功能删除重复项。

Question

我正在使用Pyspark 2.1.0。

我尝试使用以下方法执行两个数据帧的左外连接：我有2个数据帧，其架构如下所示：

crimes
 |-- CRIME_ID: string (nullable = true)
 |-- YEAR_MTH: string (nullable = true)
 |-- CRIME_TYPE: string (nullable = true)
 |-- CURRENT_OUTCOME: string (nullable = true)

outcomes
 |-- CRIME_ID: string (nullable = true)
 |-- YEAR_MTH: string (nullable = true)
 |-- FINAL_OUTCOME: string (nullable = true)

我需要能够将犯罪加入基于左外部的结果，因为单一犯罪存在许多结果。我想排除两个数据帧共有的列。

我尝试了以下两种方法，但每种方法都会产生各种错误：

cr_outs = crimes.join(outcomes, crimes.CRIME_ID == outcomes.CRIME_ID, 'left_outer')\
 .select(['crimes.'+c for c in crimes.columns] + ['outcomes.FINAL_OUTCOME'])

 from pyspark.sql.functions as fn    
 cr_outs = crimes.alias('a').join(outcomes.alias('b'), fn.col('b.CRIME_ID') = fn.col('a.CRIME_ID') ,'left_outer')\
  .select([fn.col('a.'+ c) for c in a.columns] + b.FINAL_OUTCOME)

任何人都可以建议另一种方式吗？感谢

Answer 1

这就是诀窍，似乎你必须使用别名，类似于GetId，虽然在PySpark 2.1.0中稍微简单一些。

cr_outs = crimes.alias('a')\
  .join(outcomes, crimes.CRIME_ID == outcomes.CRIME_ID, 'left_outer')\
  .select(*[col('a.'+c) for c in crimes.columns] 
          + [outcomes.FINAL_OUTCOME])

cr_outs.show()
cr_outs.printSchema()

--------+-------------------+--------------------+--------------------+--------------------+
|            CRIME_ID|YEAR_MTH|         REPORTED_BY|        FALLS_WITHIN|LONGITUDE| LATITUDE|            LOCATION|LSOA_CODE|          LSOA_NAME|          CRIME_TYPE|     CURRENT_OUTCOME|       FINAL_OUTCOME|
+--------------------+--------+--------------------+--------------------+---------+---------+--------------------+---------+-------------------+--------------------+--------------------+--------------------+
|426085c2ed33af598...| 2017-01|City of London Po...|City of London Po...|-0.086051| 51.51357|On or near Finch ...|E01032739|City of London 001F|         Other theft|Investigation com...|Investigation com...|
|33a3ddb8160a854a4...| 2017-01|City of London Po...|City of London Po...|-0.077777|51.518047|On or near Sandy'...|E01032
..
..
..
root
 |-- CRIME_ID: string (nullable = true)
 |-- YEAR_MTH: string (nullable = true)
 |-- REPORTED_BY: string (nullable = true)
 |-- FALLS_WITHIN: string (nullable = true)
 |-- LONGITUDE: float (nullable = true)
 |-- LATITUDE: float (nullable = true)
 |-- LOCATION: string (nullable = true)
 |-- LSOA_CODE: string (nullable = true)
 |-- LSOA_NAME: string (nullable = true)
 |-- CRIME_TYPE: string (nullable = true)
 |-- CURRENT_OUTCOME: string (nullable = true)
 |-- FINAL_OUTCOME: string (nullable = true)

正如您所看到的，列中的列数远远多于原始帖子，但没有重复的列，也没有重命名列： - ）

Answer 2

您可以临时重命名公共列以消除歧义

crimes = crimes\
.withColumnRenamed('CRIME_ID','CRIME_ID_1')\
.withColumnRenamed('YEAR_MTH','YEAR_MTH_1)


required_columns = [c for c in crimes.columns] + ['FINAL_OUTCOME']

cr_outs = crimes\
.join(outcomes, crimes.CRIME_ID_1 == outcomes.CRIME_ID, 'left_outer')\
.select(required_columns)

Answer 3

您可以使用以下功能删除重复项。

def dropDupeDfCols(df):
   newcols = []
   dupcols = []

for i in range(len(df.columns)):
    if df.columns[i] not in newcols:
        newcols.append(df.columns[i])
    else:
        dupcols.append(i)

df = df.toDF(*[str(i) for i in range(len(df.columns))])
for dupcol in dupcols:
    df = df.drop(str(dupcol))

return df.toDF(*newcols)## Heading ##

pyspark左外连接有多列

3 个答案:

您可以使用以下功能删除重复项。