Redshift中的分组查询需要大量时间

Question

我有以下要求：我有一个以下格式的表。

这就是我希望将其转换为：

基本上我希望具有各种活动组合的用户数量

我想使用这种格式，因为我想用它来创建TreeMap可视化。

这是我到目前为止所做的。 首先找出具有活动分组的用户数

WITH lookup AS
(
  SELECT listagg(name,',') AS groupings,
         processed_date,
         guid
  FROM warehouse.test
  GROUP BY processed_date,
           guid
)
SELECT groupings AS activity_groupings,
       LENGTH(groupings) -LENGTH(REPLACE(groupings,',','')) + 1 AS count,
       processed_date,
       COUNT(           guid) AS users
FROM lookup
GROUP BY processed_date,
         groupings

我将结果放在单独的表中

然后，我像这样进行拆分和合并：

SELECT NULLIF(SPLIT_PART(groupings,',', 1),'') AS grouping_1,
          COALESCE(NULLIF(SPLIT_PART(groupings,',', 2),''), grouping_1) AS grouping_2,
          COALESCE(NULLIF(SPLIT_PART(groupings,',', 3),''), grouping_2, grouping_1) AS grouping_3,
          num_users
   FROM warehouse.groupings) AS expr_qry
GROUP BY grouping_1,
         grouping_2,
         grouping_3

问题在于，由于我有超过2.5亿行，因此第一个查询需要执行超过90分钟的时间。

必须有一种更好，更有效的方法来解决这个问题。 敬请谅解。

谢谢

Answer 1

您不需要使用复杂的字符串操作功能（LISTAGG()，SPLIT_PART()）。您可以使用ROW_NUMBER()函数和简单的聚合来实现自己的目标。

-- Create sample data
CREATE TEMP TABLE test_data (id, guid, name) 
AS        SELECT 1::INT, 1::INT, 'cooking'
UNION ALL SELECT 2::INT, 1::INT, 'cleaning'
UNION ALL SELECT 3::INT, 2::INT, 'washing'
UNION ALL SELECT 4::INT, 4::INT, 'cooking'
UNION ALL SELECT 6::INT, 5::INT, 'cooking'
UNION ALL SELECT 7::INT, 3::INT, 'cooking'
UNION ALL SELECT 8::INT, 3::INT, 'cleaning'
;
-- Assign a row number to each name per guid
WITH name_order AS (
    SELECT guid
         , name
         , ROW_NUMBER() OVER(PARTITION BY guid ORDER BY id) row_n
    FROM test_data
) -- Use MAX() to collapse each guid's data to 1 row
, groupings AS (
    SELECT guid
         , MAX(CASE WHEN row_n = 1 THEN name END) grouping_1
         , MAX(CASE WHEN row_n = 2 THEN name END) grouping_2
    FROM name_order
    GROUP BY guid
) -- Count the guids per each grouping
SELECT grouping_1
     , COALESCE(grouping_2, grouping_1) AS grouping_2
     , COUNT(guid) num_users
   FROM groupings
GROUP BY 1,2
;
-- Output
 grouping_1 | grouping_2 | num_users
------------+------------+-----------
 washing    | washing    |         1
 cooking    | cleaning   |         2
 cooking    | cooking    |         2