Question

我将下面的pyspark函数转换为Spark-Scala。在pyspark函数中，我们作为参数传递，例如（plat，metrics，perc）值并为Input导出一些roximateQuantile值。

两个代码（即Pyspark和scala）的输入数据相同。

pyspark代码：

vkCmdPipelineBarrier

调用方法：

//////////////////////////////////////////////////////
// First layout transition
//////////////////////////////////////////////////////
VkImageMemoryBarrier vkGraphicsImageMemoryBarrier
{
    VkStructureType::VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER,
    nullptr,
    0,
    VkAccessFlagBits::VK_ACCESS_COLOR_ATTACHMENT_WRITE_BIT,
    VkImageLayout::VK_IMAGE_LAYOUT_UNDEFINED,
    VkImageLayout::VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL,
    VK_QUEUE_FAMILY_IGNORED,
    VK_QUEUE_FAMILY_IGNORED,
    this->sSwapchain.vulkanImageList()[nImageIndex],
    VkImageSubresourceRange
    {
        VkImageAspectFlagBits::VK_IMAGE_ASPECT_COLOR_BIT,
        0,
        VK_REMAINING_MIP_LEVELS,
        0,
        VK_REMAINING_ARRAY_LAYERS
    }
};

vkCmdPipelineBarrier(
    this->sGraphicsCommandBufferList[nImageIndex],
    VkPipelineStageFlagBits::VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT,
    VkPipelineStageFlagBits::VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT,
    0,
    0, nullptr,
    0, nullptr,
    1, &vkGraphicsImageMemoryBarrier);

//
//  Draws a triangle.
//

//////////////////////////////////////////////////////
// Second layout transition and ownership transfer
//////////////////////////////////////////////////////
VkImageMemoryBarrier vkGraphicsImageMemoryBarrier
{
    VkStructureType::VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER,
    nullptr,
    VkAccessFlagBits::VK_ACCESS_COLOR_ATTACHMENT_WRITE_BIT,
    0,
    VkImageLayout::VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL,
    VkImageLayout::VK_IMAGE_LAYOUT_PRESENT_SRC_KHR,
    this->nGraphicsFamily,
    this->nPresentFamily,
    this->sSwapchain.vulkanImageList()[nImageIndex],
    VkImageSubresourceRange
    {
        VkImageAspectFlagBits::VK_IMAGE_ASPECT_COLOR_BIT,
        0,
        VK_REMAINING_MIP_LEVELS,
        0,
        VK_REMAINING_ARRAY_LAYERS
    }
};
VkImageMemoryBarrier vkPresentImageMemoryBarrier
{
    VkStructureType::VK_STRUCTURE_TYPE_IMAGE_MEMORY_BARRIER,
    nullptr,
    0,
    0,
    VkImageLayout::VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL,
    VkImageLayout::VK_IMAGE_LAYOUT_PRESENT_SRC_KHR,
    this->nGraphicsFamily,
    this->nPresentFamily,
    this->sSwapchain.vulkanImageList()[nImageIndex],
    VkImageSubresourceRange
    {
        VkImageAspectFlagBits::VK_IMAGE_ASPECT_COLOR_BIT,
        0,
        VK_REMAINING_MIP_LEVELS,
        0,
        VK_REMAINING_ARRAY_LAYERS
    }
};

//
//  Below call produces the above validation message.
//
vkCmdPipelineBarrier(
    this->sGraphicsCommandBufferList[nImageIndex],
    VkPipelineStageFlagBits::VK_PIPELINE_STAGE_COLOR_ATTACHMENT_OUTPUT_BIT,
    VkPipelineStageFlagBits::VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT,
    0,
    0, nullptr,
    0, nullptr,
    1, &vkGraphicsImageMemoryBarrier);
vkCmdPipelineBarrier(
    this->sPresentCommandBufferList[nImageIndex],
    VkPipelineStageFlagBits::VK_PIPELINE_STAGE_TOP_OF_PIPE_BIT,
    VkPipelineStageFlagBits::VK_PIPELINE_STAGE_BOTTOM_OF_PIPE_BIT,
    0,
    0, nullptr,
    0, nullptr,
    1, &vkPresentImageMemoryBarrier);

输出值，如：

def give_percentile(plat,metrics,perc):
    df_perc = df_final_main.filter(df_final_main.platform.like('%' + plat + '%'))
    df_perc = df_perc.filter(df_final_main[metrics]!=0)
    percentile_val = df_perc.approxQuantile(metrics, [perc],0.05)
    if len(percentile_val)>0:
        percentile_val = float(percentile_val[0])
    else:
        percentile_val = float(0)
    return percentile_val

火花标量代码：

df_agg = sqlContext.createDataFrame(
[Row(platform='iOS',percentile_page_load_50=give_percentile("iOS","page_load",0.5)),
 Row(platform='Android',percentile_page_load_50=give_percentile("Android","page_load",0.5)),
 Row(platform='Web',percentile_page_load_50=give_percentile("Web","page_load",0.5))
 ]
)

调用方法：

+-----------------------+--------+
|percentile_page_load_50|platform|
+-----------------------+--------+
|                    0.0|     iOS|
|                    0.0| Android|
|      6.956999778747559||    Web|
+-----------------------+--------+

但是它在这里给出了不同的输出：

输出值，例如：

def give_percentile(plat: String, metrics:String, perc: Double): Double = {
    var df_perc = df_final_main_join.filter(df_final_main_join.col("platform").like('%' + plat + '%'))
    var df_perc1 = df_perc.filter(df_perc.col(metrics) =!= 0)
    var apQuantile = df_perc.stat.approxQuantile(b, Array(c), 0.05)
    if (apQuantile.length > 0){
        var ret = apQuantile(0)
        return ret
    }
    else {
        var ret :Double=(0)
        return ret
    }
}

那么您能在这里帮助我，告诉我上面的Scala代码有什么问题吗？

Answer 1

我认为这是因为您的Scala代码中有错字：

错误：

var df_perc1 = df_perc.filter(df_perc.col(metrics) =!= 0)

右：

var df_perc1 = df_perc.filter(df_perc.col(metrics) != 0)

从pyspark转换为scala函数时面临问题

1 个答案: