KDB喜欢加入pandas中的时间序列数据?

时间:2012-09-07 16:49:21

标签: python join time-series pandas kdb

kdb +有一个aj函数,通常用于沿时间列连接表。

这是一个我有交易和报价表的例子,我得到每笔交易的现行报价。

q)5# t
time         sym  price size 
-----------------------------
09:30:00.439 NVDA 13.42 60511
09:30:00.439 NVDA 13.42 60511
09:30:02.332 NVDA 13.42 100  
09:30:02.332 NVDA 13.42 100  
09:30:02.333 NVDA 13.41 100  

q)5# q
time         sym  bid   ask   bsize asize
-----------------------------------------
09:30:00.026 NVDA 13.34 13.44 3     16   
09:30:00.043 NVDA 13.34 13.44 3     17   
09:30:00.121 NVDA 13.36 13.65 1     10   
09:30:00.386 NVDA 13.36 13.52 21    1    
09:30:00.440 NVDA 13.4  13.44 15    17

q)5# aj[`time; t; q]
time         sym  price size  bid   ask   bsize asize
-----------------------------------------------------
09:30:00.439 NVDA 13.42 60511 13.36 13.52 21    1    
09:30:00.439 NVDA 13.42 60511 13.36 13.52 21    1    
09:30:02.332 NVDA 13.42 100   13.34 13.61 1     1    
09:30:02.332 NVDA 13.42 100   13.34 13.61 1     1    
09:30:02.333 NVDA 13.41 100   13.34 13.51 1     1  

如何使用pandas进行相同的操作?我正在使用交易和报价数据框,其中索引是datetime64。

In [55]: quotes.head()
Out[55]: 
                              bid    ask  bsize  asize
2012-09-06 09:30:00.026000  13.34  13.44      3     16
2012-09-06 09:30:00.043000  13.34  13.44      3     17
2012-09-06 09:30:00.121000  13.36  13.65      1     10
2012-09-06 09:30:00.386000  13.36  13.52     21      1
2012-09-06 09:30:00.440000  13.40  13.44     15     17

In [56]: trades.head()
Out[56]: 
                            price   size
2012-09-06 09:30:00.439000  13.42  60511
2012-09-06 09:30:00.439000  13.42  60511
2012-09-06 09:30:02.332000  13.42    100
2012-09-06 09:30:02.332000  13.42    100
2012-09-06 09:30:02.333000  13.41    100

我看到pandas有一个asof函数,但是没有在DataFrame上定义,只在Series对象上定义。我猜一个人可以循环遍历每个系列并逐个对齐它们,但我想知道是否有更好的方法?

3 个答案:

答案 0 :(得分:13)

我前段时间写了一篇宣传不足的ordered_merge函数:

In [27]: quotes
Out[27]: 
                        time    bid    ask  bsize  asize
0 2012-09-06 09:30:00.026000  13.34  13.44      3     16
1 2012-09-06 09:30:00.043000  13.34  13.44      3     17
2 2012-09-06 09:30:00.121000  13.36  13.65      1     10
3 2012-09-06 09:30:00.386000  13.36  13.52     21      1
4 2012-09-06 09:30:00.440000  13.40  13.44     15     17

In [28]: trades
Out[28]: 
                        time  price   size
0 2012-09-06 09:30:00.439000  13.42  60511
1 2012-09-06 09:30:00.439000  13.42  60511
2 2012-09-06 09:30:02.332000  13.42    100
3 2012-09-06 09:30:02.332000  13.42    100
4 2012-09-06 09:30:02.333000  13.41    100

In [29]: ordered_merge(quotes, trades)
Out[29]: 
                        time    bid    ask  bsize  asize  price   size
0 2012-09-06 09:30:00.026000  13.34  13.44      3     16    NaN    NaN
1 2012-09-06 09:30:00.043000  13.34  13.44      3     17    NaN    NaN
2 2012-09-06 09:30:00.121000  13.36  13.65      1     10    NaN    NaN
3 2012-09-06 09:30:00.386000  13.36  13.52     21      1    NaN    NaN
4 2012-09-06 09:30:00.439000    NaN    NaN    NaN    NaN  13.42  60511
5 2012-09-06 09:30:00.439000    NaN    NaN    NaN    NaN  13.42  60511
6 2012-09-06 09:30:00.440000  13.40  13.44     15     17    NaN    NaN
7 2012-09-06 09:30:02.332000    NaN    NaN    NaN    NaN  13.42    100
8 2012-09-06 09:30:02.332000    NaN    NaN    NaN    NaN  13.42    100
9 2012-09-06 09:30:02.333000    NaN    NaN    NaN    NaN  13.41    100

In [32]: ordered_merge(quotes, trades, fill_method='ffill')
Out[32]: 
                        time    bid    ask  bsize  asize  price   size
0 2012-09-06 09:30:00.026000  13.34  13.44      3     16    NaN    NaN
1 2012-09-06 09:30:00.043000  13.34  13.44      3     17    NaN    NaN
2 2012-09-06 09:30:00.121000  13.36  13.65      1     10    NaN    NaN
3 2012-09-06 09:30:00.386000  13.36  13.52     21      1    NaN    NaN
4 2012-09-06 09:30:00.439000  13.36  13.52     21      1  13.42  60511
5 2012-09-06 09:30:00.439000  13.36  13.52     21      1  13.42  60511
6 2012-09-06 09:30:00.440000  13.40  13.44     15     17  13.42  60511
7 2012-09-06 09:30:02.332000  13.40  13.44     15     17  13.42    100
8 2012-09-06 09:30:02.332000  13.40  13.44     15     17  13.42    100
9 2012-09-06 09:30:02.333000  13.40  13.44     15     17  13.41    100

可以很容易地(对于熟悉代码的人)扩展为模仿KDB的“左连接”。在这种情况下,我意识到向前填充贸易数据是不合适的;只是说明了这个功能。

答案 1 :(得分:9)

正如您在问题中提到的,循环遍历每一列应该适合您:

df1.apply(lambda x: x.asof(df2.index))

我们可以创建一个更快的NaN-naive版本的DataFrame.asof来一次性完成所有列。但就目前而言,我认为这是最直接的方式。

答案 2 :(得分:7)

pandas 0.19 has introduced an asof join

render "new"

语义与q / kdb +中的功能非常相似。

相关问题