Question

我有一个按日期时间列排序的熊猫数据框。几行将具有相同的日期时间，但是“报告类型”列的值不同。我只需要根据首选报告类型列表选择这些行之一。该列表按优先顺序排列。因此，如果这些行中的一个在列表中具有第一个元素，那么该行即被选择附加到新的数据框。

我尝试了GroupBy和速度如此慢的Python for循环，以处理每个组以查找首选的报告类型，并将该行附加到新的数据框中。我想到了numpy vectorize（），但是我不知道如何在其中合并该组。我确实对数据帧了解不多，但是正在学习。关于如何使其更快的任何想法？我可以加入这个小组吗？

示例数据框

OBSERVATIONTIME       REPTYPE   CIGFT
2000-01-01 00:00:00 AUTO    73300
2000-01-01 00:00:00 FM-15   25000
2000-01-01 00:00:00 FM-12   3000
2000-01-01 01:00:00 SAO 9000
2000-01-01 01:00:00 FM-16   600
2000-01-01 01:00:00 FM-15   5000
2000-01-01 01:00:00 AUTO    5000
2000-01-01 02:00:00 FM-12   12000
2000-01-01 02:00:00 FM-15   15000
2000-01-01 02:00:00 FM-16   8000
2000-01-01 03:00:00 SAO 700
2000-01-01 04:00:00 SAO 3000
2000-01-01 05:00:00 FM-16   5000
2000-01-01 06:00:00 AUTO    15000
2000-01-01 06:00:00 FM-12   12500
2000-01-01 06:00:00 FM-16   12000
2000-01-01 07:00:00 FM-15   20000

#################################################
# The function to loop through and find the row
################################################
    def select_the_one_ob(df):
    ''' select the preferred observation '''
    tophour_df = pd.DataFrame()
    preferred_order = ['FM-15', 'AUTO', 'SAO', 'FM-16', 'SAOSP', 'FM-12', 
'SY-MT', 'SY-SA']
    grouped = df.groupby("OBSERVATIONTIME", as_index=False)

    for name, group in grouped:
        a_group_df = pd.DataFrame(grouped.get_group(name))

        for reptype in preferred_order:
            preferred_found = False
            for i in a_group_df.index.values:
                if a_group_df.loc[i, 'REPTYPE'] == reptype:
                    tophour_df = 
tophour_df.append(a_group_df.loc[i].transpose())
                    preferred_found = True
                    break

            if preferred_found:
                break

        del a_group_df



    return tophour_df

################################################
### The function which calls the above function
################################################
def process_ceiling(plat, network):
    platformcig.data_pull(CONNECT_SRC, PULL_CEILING)
    data_df = platformcig.df

    data_df = select_the_one_ob(data_df)

具有300,000行的完整数据集，该功能需要4个小时以上。我需要它要快得多。我可以在numpy vectorize（）中合并该组吗？

Answer 1

您可以避免使用<SwitchToggle onPress={this.onPress1} />。一种方法是使用pd.Categorical然后再用sort_values和drop_duplicates对列“ REPTYPE”进行分类，例如：

groupby

您将获得示例：

def select_the_one_ob(df):
    preferred_order = ['FM-15', 'AUTO', 'SAO', 'FM-16', 'SAOSP', 'FM-12', 'SY-MT', 'SY-SA']
    df.REPTYPE = pd.Categorical(df.REPTYPE, categories=preferred_order, ordered=True)
    return (df.sort_values(by=['OBSERVATIONTIME','REPTYPE'])
              .drop_duplicates(subset='OBSERVATIONTIME', keep='first'))

Answer 2

发现创建一个单独的形状相同的数据框时，每个小时的观察时间都有填充，我可以使用pandas数据框merge（），在第一遍之后使用pandas数据框Combine_first（）。这只花了几分钟而不是几小时。

def select_the_one_ob（df）： '''选择首选观察参数： df（熊猫对象），一个熊猫数据框

Returns Pandas Dataframe
'''
dshelldict = {'DateTime': pd.date_range(BEG_POR, END_POR, freq='H')}
dshell = pd.DataFrame(data = dshelldict)

dshell['YEAR'] = dshell['DateTime'].dt.year  
dshell['MONTH'] = dshell['DateTime'].dt.month
dshell['DAY'] = dshell['DateTime'].dt.day
dshell['HOUR'] = dshell['DateTime'].dt.hour
dshell = dshell.set_index(['YEAR','MONTH','DAY','HOUR'])
df = df.set_index(['YEAR','MONTH','DAY','HOUR'])

#tophour_df = pd.DataFrame()
preferred_order = ['FM-15', 'AUTO', 'SAO', 'FM-16', 'SAOSP', 'FM-12', 'SY-MT', 'SY-SA']
reptype_list = list(df.REPTYPE.unique())

# remove the preferred report types from the unique ones
for rep in preferred_order:
    if rep in reptype_list:
        reptype_list.remove(rep)

# If there are any unique report types left, append them to the preferred list
if len(reptype_list) > 0:
    preferred_order = preferred_order + reptype_list

## i is flag to make sure a report type is used to transfer columns to new DataFrame
## (Merge has to happen before combine first)
first_pass = True
for reptype in preferred_order:
    if first_pass:
        ## if there is data in dataframe
        if df[(df['MINUTE']==00)&(df['REPTYPE']==reptype)].shape[0]>0:
            first_pass = False
            # Merge shell with first df with data, the dataframe is sorted by original 
            # obstime and drop any dup's keeping first aka. first report chronologically
            tophour_df = dshell.merge( df[ (df['MINUTE']==00)&(df['REPTYPE']==reptype) ].sort_values(['OBSERVATIONTIME'],ascending=True).drop_duplicates(subset=['ROLLED_OBSERVATIONTIME'],keep='first'),how ='left',left_index = True,right_index=True ).drop('DateTime',axis=1)
    else:
        # combine_first takes the original dataframe and fills any nan values with data 
        # of another identical shape dataframe
        # ex. if value df.loc[2,col1] is nan df2.loc[2,col1] would fill it if not nan
        tophour_df = tophour_df.combine_first(df[(df['MINUTE']==00)&(df['REPTYPE']==reptype)].sort_values(['OBSERVATIONTIME'],ascending=True).drop_duplicates(subset=['ROLLED_OBSERVATIONTIME'],keep='first'))

tophour_df = tophour_df.reset_index()

return tophour_df

强调文字

向量化GroupBy Pandas数据框的功能

2 个答案: