Question

我需要将列表传递给UDF，该列表将确定距离的分数/类别。就目前而言，我很难将所有距离编码为第4分。

// The below line is equivalent to writing:
// position: new google.maps.LatLng(-34.397, 150.644)
position: {lat: -34.397, lng: 150.644},

当我尝试这样的事情时，我得到了这个错误。

a= spark.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])

from pyspark.sql.functions import udf
def cate(label, feature_list):
    if feature_list == 0:
        return label[4]
label_list = ["Great", "Good", "OK", "Please Move", "Dead"]
udf_score=udf(cate, StringType())
a.withColumn("category", udf_score(label_list,a["distances"])).show(10)

Answer 1

希望这有帮助！

from pyspark.sql.functions import udf, col

#sample data
a= sqlContext.createDataFrame([("A", 20), ("B", 30), ("D", 80)],["Letter", "distances"])
label_list = ["Great", "Good", "OK", "Please Move", "Dead"]

def cate(label, feature_list):
    if feature_list == 0:
        return label[4]
    else:  #you may need to add 'else' condition as well otherwise 'null' will be added in this case
        return 'I am not sure!'

def udf_score(label_list):
    return udf(lambda l: cate(l, label_list))
a.withColumn("category", udf_score(label_list)(col("distances"))).show()

输出是：

+------+---------+--------------+
|Letter|distances|      category|
+------+---------+--------------+
|     A|       20|I am not sure!|
|     B|       30|I am not sure!|
|     D|       80|I am not sure!|
+------+---------+--------------+

Answer 2

尝试使用该函数，以便DataFrame调用中唯一的参数是您希望函数执行的列的名称：

udf_score=udf(lambda x: cate(label_list,x), StringType())
a.withColumn("category", udf_score("distances")).show(10)

Answer 3

我认为这可能有助于将list作为变量的默认值

传递

bestnews1 = Best.objects.filter(select_reason="左一").first()
bestnews1_new = None if bestnew1 is None else bestnews1.select_news
return render(request, 'index.html', {
            'all_news': news,
             'bestnews1_new':bestnews1_new,
             'bestnews2_new':bestnews2_new,
        })

输出：

from pyspark.sql.functions import udf, col

#sample data
a= sqlContext.createDataFrame([("A", 20), ("B", 30), ("D", 80),("E",0)],["Letter", "distances"])
label_list = ["Great", "Good", "OK", "Please Move", "Dead"]

#Passing List as Default value to a variable
def cate( feature_list,label=label_list):
    if feature_list == 0:
        return label[4]
    else:  #you may need to add 'else' condition as well otherwise 'null' will be added in this case
        return 'I am not sure!'

udfcate = udf(cate, StringType())

a.withColumn("category", udfcate("distances")).show()

PySpark - 将列表作为参数传递给UDF

3 个答案: