DATEADD的Pyspark实施

Question

我有如下所示的T-SQL代码

cast( dateadd(minute, - 240, tmp_view_tos_lenelgate_qry11.eventdate) as date

如何在PYSPARK中实现DATE_ADD功能？

Answer 1

# Creating the DataFrame
df = spark.createDataFrame([('2014-02-13 12:36:52.721',),('2018-01-01 00:30:50.001',)], ['eventdate'])
df = df.withColumn('eventdate', col('eventdate').cast('timestamp'))
df.show(truncate=False)
+-----------------------+
|eventdate              |
+-----------------------+
|2014-02-13 12:36:52.721|
|2018-01-01 00:30:50.001|
+-----------------------+
df.printSchema()
root
 |-- eventdate: timestamp (nullable = true)

# Subtract 240 minutes/240*60=14400 seconds from 'eventdate'
from pyspark.sql.functions import col, unix_timestamp
df = df.withColumn('eventdate_new', (unix_timestamp('eventdate') - 240*60).cast('timestamp'))
df.show(truncate=False)
+-----------------------+-------------------+
|eventdate              |eventdate_new      |
+-----------------------+-------------------+
|2014-02-13 12:36:52.721|2014-02-13 08:36:52|
|2018-01-01 00:30:50.001|2017-12-31 20:30:50|
+-----------------------+-------------------+