Spark数据框连接时出现错误结果

时间:2019-12-31 06:54:03

标签: scala apache-spark join apache-spark-sql

我正在使用spark-1.6.0,我想加入2个数据框,它们显示在YARN日志中,如下所示。

一个是df_train_raw:

+------------+-----------+-----+
|subscriberid|   objectid|label|
+------------+-----------+-----+
|    80755258|11030733889|    0|
|    81405858|11030733889|    0|
|    83486458|11030733889|    0|
|    81867258|11030733889|    0|
|    83077858|11030733889|    0|
|    80278458|11030733889|    0|
|    80044458|11030733889|    0|
|    81079858|11030733889|    0|
|    83418658|11030733889|    0|
|    83105658|11030733889|    0|
|    83105658|    2157122|    0|
|    83077858|11030780536|    0|
|    83105658|11030797977|    0|
|    83418658|11030714577|    0|
|    83077858|11030714577|    0|
|    79752658|11030714577|    0|
|    83105658|11028639583|    0|
|    79752658|11030549822|    0|
|    83105658|11028975426|    0|
|    83105658|11030686035|    0|
|    81079858|11030686035|    0|
|    79752658|11030504648|    0|
|    83486458|11030696858|    0|
|    81867258|11030696858|    0|
|    83105658|11030696858|    0|
|    81079858|11030696858|    0|
|    83418658|11030696858|    0|
|    80044458|11030696858|    0|
|    80278458|11030696858|    0|
|    81405858|11030696858|    0|
|    80755258|11030696858|    0|
|    83486458|11030434056|    0|
|    80278458|11030434056|    0|
|    81405858|11030434056|    0|
|    80044458|11030434056|    0|
|    80755258|11030434056|    0|
|    81867258|11030434056|    0|
|   108920274|11022029789|    1|
+------------+-----------+-----+

另一个是df_user_clicks_info:

+------------+-------------------------------+-------------------------------+--------------------------------+---------------------------------+---------------------------------+---------------------------------+--------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------------+----------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+----------------------------------+
|subscriberid|user_clicks_avg_everyday_a_week|user_clicks_sum_time_1_9_a_week|user_clicks_sum_time_9_14_a_week|user_clicks_sum_time_14_17_a_week|user_clicks_sum_time_17_19_a_week|user_clicks_sum_time_19_23_a_week|user_clicks_sum_time_23_1_a_week|user_clicks_avg_everyday_weekday|user_clicks_sum_time_1_9_weekday|user_clicks_sum_time_9_14_weekday|user_clicks_sum_time_14_17_weekday|user_clicks_sum_time_17_19_weekday|user_clicks_sum_time_19_23_weekday|user_clicks_sum_time_23_1_weekday|user_clicks_avg_everyday_weekdend|user_clicks_sum_time_1_9_weekdend|user_clicks_sum_time_9_14_weekdend|user_clicks_sum_time_14_17_weekdend|user_clicks_sum_time_17_19_weekdend|user_clicks_sum_time_19_23_weekdend|user_clicks_sum_time_23_1_weekdend|
+------------+-------------------------------+-------------------------------+--------------------------------+---------------------------------+---------------------------------+---------------------------------+--------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------------+----------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+----------------------------------+
|    83486458|                           1.71|                              3|                               3|                                3|                                0|                                3|                               0|                             2.4|                               3|                                3|                                 3|                                 0|                                 3|                                0|                              0.0|                                0|                                 0|                                  0|                                  0|                                  0|                                 0|
|    81867258|                           0.43|                              0|                               3|                                0|                                0|                                0|                               0|                             0.0|                               0|                                0|                                 0|                                 0|                                 0|                                0|                              1.5|                                0|                                 3|                                  0|                                  0|                                  0|                                 0|
|    80044458|                           0.43|                              3|                               0|                                0|                                0|                                0|                               0|                             0.6|                               3|                                0|                                 0|                                 0|                                 0|                                0|                              0.0|                                0|                                 0|                                  0|                                  0|                                  0|                                 0|
|    79752658|                           2.57|                              0|                               9|                                0|                                3|                                6|                               0|                             3.0|                               0|                                9|                                 0|                                 3|                                 3|                                0|                              1.5|                                0|                                 0|                                  0|                                  0|                                  3|                                 0|
|    83105658|                           59.0|                             63|                             105|                               49|                               70|                              126|                               0|                            61.6|                              56|                               98|                                49|                                42|                                63|                                0|                             52.5|                                7|                                 7|                                  0|                                 28|                                 63|                                 0|
|    80278458|                           3.43|                              0|                               9|                                3|                                0|                               12|                               0|                             3.6|                               0|                                9|                                 3|                                 0|                                 6|                                0|                              3.0|                                0|                                 0|                                  0|                                  0|                                  6|                                 0|
|    81405858|                           1.29|                              3|                               0|                                3|                                0|                                3|                               0|                             1.2|                               0|                                0|                                 3|                                 0|                                 3|                                0|                              1.5|                                3|                                 0|                                  0|                                  0|                                  0|                                 0|
|   108920274|                            2.0|                              1|                               5|                                4|                                4|                                0|                               0|                             2.2|                               0|                                5|                                 3|                                 3|                                 0|                                0|                              1.5|                                1|                                 0|                                  1|                                  1|                                  0|                                 0|
|    80755258|                           2.14|                              3|                               0|                                0|                                6|                                6|                               0|                             0.0|                               0|                                0|                                 0|                                 0|                                 0|                                0|                              7.5|                                3|                                 0|                                  0|                                  6|                                  6|                                 0|
+------------+-------------------------------+-------------------------------+--------------------------------+---------------------------------+---------------------------------+---------------------------------+--------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------------+----------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+----------------------------------+

————————————

我试图用代码将它们内部连接:

val df_tmp_tmp_0 = df_train_raw.join(df_user_clicks_info, Seq("subscriberid"))

df_tmp_tmp_0.show()

我得到的结果完全没有!天哪!

+------------+--------+-----+------------+-------------------------------+-------------------------------+--------------------------------+---------------------------------+---------------------------------+---------------------------------+--------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------------+----------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+----------------------------------+
|subscriberid|objectid|label|subscriberid|user_clicks_avg_everyday_a_week|user_clicks_sum_time_1_9_a_week|user_clicks_sum_time_9_14_a_week|user_clicks_sum_time_14_17_a_week|user_clicks_sum_time_17_19_a_week|user_clicks_sum_time_19_23_a_week|user_clicks_sum_time_23_1_a_week|user_clicks_avg_everyday_weekday|user_clicks_sum_time_1_9_weekday|user_clicks_sum_time_9_14_weekday|user_clicks_sum_time_14_17_weekday|user_clicks_sum_time_17_19_weekday|user_clicks_sum_time_19_23_weekday|user_clicks_sum_time_23_1_weekday|user_clicks_avg_everyday_weekdend|user_clicks_sum_time_1_9_weekdend|user_clicks_sum_time_9_14_weekdend|user_clicks_sum_time_14_17_weekdend|user_clicks_sum_time_17_19_weekdend|user_clicks_sum_time_19_23_weekdend|user_clicks_sum_time_23_1_weekdend|
+------------+--------+-----+------------+-------------------------------+-------------------------------+--------------------------------+---------------------------------+---------------------------------+---------------------------------+--------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------------+----------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+----------------------------------+
+------------+--------+-----+------------+-------------------------------+-------------------------------+--------------------------------+---------------------------------+---------------------------------+---------------------------------+--------------------------------+--------------------------------+--------------------------------+---------------------------------+----------------------------------+----------------------------------+----------------------------------+---------------------------------+---------------------------------+---------------------------------+----------------------------------+-----------------------------------+-----------------------------------+-----------------------------------+----------------------------------+

我不知道为什么?这里好像没错吗?希望有帮助〜谢谢〜

0 个答案:

没有答案
豫ICP备18024241号-1