pyspark：向数据框添加新字段行元素

Question

我有以下元素：

a = Row(ts=1465326926253, myid=u'1234567', mytype=u'good') 

Row是火花数据框的Row类。我可以在a中添加一个新字段，所以看起来像：

a = Row(ts=1465326926253, myid=u'1234567', mytype=u'good', name = u'john') 

谢谢！

Answer 1

这是一个有效的更新答案。首先，您必须创建一个字典，然后更新dict，然后将其写入pyspark Row。

代码如下：

from pyspark.sql import Row

#Creating the pysql row
row = Row(field1=12345, field2=0.0123, field3=u'Last Field')

#Convert to python dict
temp = row.asDict()

#Do whatever you want to the dict. Like adding a new field or etc.
temp["field4"] = "it worked!"

# Save or output the row to a pyspark rdd
output = Row(**temp)

#How it looks
output

In [1]:
Row(field1=12345, field2=0.0123, field3=u'Last Field', field4='it worked!')

Answer 2

您无法向Row添加新字段。 Row is a subclass of tuple

from pyspark.sql import Row

issubclass(Row, tuple)
## True

isinstance(Row(), tuple)
## True

和Python tuples是不可变的。你所能做的就是创建一个新的：

row = Row(ts=1465326926253, myid=u'1234567', mytype=u'good') 

# In legacy Python: Row(name=u"john", **row.asDict())
Row(**row.asDict(), name=u"john") 
## Row(myid='1234567', mytype='good', name='john', ts=1465326926253)

请注意Row保持字段按名称排序。