加快向PostgreSQL插入新数据

Question

我目前的工作流程如下。

但是，它非常慢。它每天只能处理数百万个数据

我想加快速度。有什么想法吗？

query_expression（由Ruby rake文件生成，然后将生成的SQL表达式放入ActiveRecord::Base.connection.execute）

• sample-1-query.sql的第1步：按小时，小时，......汇总数据

      LEFT JOIN
      (
        SELECT  DISTINCT ON (1)
          date_trunc('#{frequence}', ticktime) AS ticktime ,
  

• sample-1-query.sql的第2步：填空/差距

      FROM 
      (
        SELECT DISTINCT ON (1) generate_series
        (
          date_trunc('second', min(ticktime)::TIMESTAMP),
          max(ticktime)::TIMESTAMP,
  

• query_expression

      SELECT DISTINCT ON (time)
         time_series.ticktime AS time,
         t.high,
         t.low,
         t.open,
         t.close,
         t.volume,
         t.product_type,
         t.contract_month
      FROM 
      (
        SELECT DISTINCT ON (1) generate_series
        (
          date_trunc('second', min(ticktime)::TIMESTAMP),
          max(ticktime)::TIMESTAMP,
          '1 #{frequence}'::interval
        ) AS ticktime FROM #{market} WHERE product_type ='#{product_type}' AND contract_month = '#{contract_month}'::timestamp
  
      ) time_series
      LEFT JOIN
      (
        SELECT  DISTINCT ON (1)
          date_trunc('#{frequence}', ticktime) AS ticktime ,
          first_value(last_price) OVER w AS open,
          max(last_price) OVER w AS high ,
          min(last_price) OVER w AS low,
          last_value(last_price)  OVER w AS close,
          sum(last_volume) OVER w AS volume,
          product_type,
          contract_month
          FROM #{market}
          WHERE product_type ='#{product_type}' 
          AND contract_month = '#{contract_month}'::timestamp
          WINDOW w AS (PARTITION BY date_trunc('#{frequence}', ticktime) ORDER BY ticktime
                       ROWS BETWEEN UNBOUNDED PRECEDING
                       AND UNBOUNDED FOLLOWING)
      ) t USING (ticktime)
      WHERE time_series.ticktime::time >= '#{market_begin_end_time[market]["begin_at"]}'::time 
      AND time_series.ticktime::time < '#{market_begin_end_time[market]["end_at"]}'::time
      AND time_series.ticktime > '#{sampling_begin_time}'::TIMESTAMP 
      ORDER BY 1       
  

然后

Rake文件

ActiveRecord::Base.connection.execute(query_expression).each_with_index do |raw_record, j|
     Model.create(raw_record)                     
end