Question

在执行看似两级分组时，我对Pig有疑问。举个例子，假设我有一些示例输入数据，如：

email_id:chararray    from:chararray        to:bag{recipients:tuple(recipient:chararray)}
e1                    user1@example.com     {(friend1@example.com),(friend2@example.com),(friend3@myusers.com)}
e2                    user1@example.com     {(friend1@example.com),(friend4@example.com)}
e3                    user1@example.com     {(friend5@example.com)}
e4                    user2@example.com     {(friend2@example.com),(friend4@example.com)}

因此，每一行都是来自用户“从”到用户“到”的电子邮件。

我最终想要一份所有发件人和他们发送电子邮件的所有人的列表，包括为每个人发送的电子邮件数量，从最高到最低排序，例如：

user1@example.com     {(friend1@example.com, 2), (friend2@example.com, 1), (friend3@example.com, 1), (friend4@example.com, 1), (friend5@example.com, 1)}
user2@example.com     {(friend2@example.com, 1), (friend4@example.com, 1)}

在猪的最佳解决方法的想法将不胜感激！

Answer 1

以下是该脚本的一个版本：

inpt = load '/pig_data/pig_fun/input/from_senders.txt' as (email_id:chararray, from:chararray, to:bag{recipients:tuple(recipient:chararray)});

pivot = foreach inpt generate from, FLATTEN(to);
pivot = foreach pivot generate from, to::recipient as recipient;
dump pivot;
/*
(user1@example.com,friend1@example.com)
(user1@example.com,friend2@example.com)
(user1@example.com,friend3@myusers.com)
(user1@example.com,friend1@example.com)
(user1@example.com,friend4@example.com)
(user1@example.com,friend5@example.com)
(user2@example.com,friend2@example.com)
(user2@example.com,friend4@example.com)
*/

grp = group pivot by (from, recipient);
with_count = foreach grp generate FLATTEN(group), COUNT(pivot) as count;
dump with_count;
/*
(user1@example.com,friend1@example.com,2)
(user1@example.com,friend2@example.com,1)
(user1@example.com,friend3@myusers.com,1)
(user1@example.com,friend4@example.com,1)
(user1@example.com,friend5@example.com,1)
(user2@example.com,friend2@example.com,1)
(user2@example.com,friend4@example.com,1)
*/

to_bag = group with_count by from;
result = foreach to_bag {
    order_by_count = order with_count by count desc;
    generate group as from, order_by_count.(recipient, count);
};
dump result;
/*
(user1@example.com,{(friend1@example.com,2),(friend2@example.com,1),(friend3@myusers.com,1),(friend4@example.com,1),(friend5@example.com,1)})
(user2@example.com,{(friend2@example.com,1),(friend4@example.com,1)})
*/

希望它有所帮助。

在Pig中对袋子的不同值进行计数

1 个答案: