Question

我有很多数据集需要排序并可能合并。数据由来自不同时间点的2D（x，y）数据组成，我想要做的是：

读取不同的时间点，看看我是否有来自同一时间点的多个（x，y）数据集。如果是这种情况，我想比较X轴，看看数据是否可以合并。

所有这一切我都可以在python中轻松完成 - 它可能不是很漂亮的代码，但似乎有效。问题在于，由于迭代数据集的方式，合并后的数据包含已汇总的数据的副本。这有点难以解释，所以让我用这个示例代码说明：

# define test data
times = ['10ms', '20ms','30ms','10ms', '10ms']
x_axis = np.atleast_2d(np.linspace(1,5,5)).T
data_sets = [np.concatenate((x_axis, np.random.rand(5,5)),axis=1) for num in range(5)]


def mergeData(times, data_sets):
    data = []
    pooled_times = []
    repetitions_strings = ['','_2nd', '_3rd'] +  ['_%ith' % num for num in range(3,30,1) ]

    for num, item in enumerate(times):
        if times.count(item) == 1:
            # only one data-set with the current time point exist
            pooled_times.append(item)
            data.append(data_sets[num])

        elif times.count(item) > 1:
            # more than one occurence of this time point
            # extract all the occurences and compare them 
            idx_different = 0 # dummy variable used to keep track of the numbers of different X-axis
            idx_repetitions = item == np.array(times)

            # lists do not accept lists of boolean index argument. find the number indices
            num_repititions = np.linspace(0, len(idx_repetitions)-1, len(idx_repetitions))[idx_repetitions]

            # *** get the data ***
            temporary_data = [data_sets[int(num)] for num in num_repititions]
            # either round off X-axis or change both tolerances in np.allclose()
            X_axis_round = [np.round(temporary_data[int(num)][:,0],decimals=4) for num in range(len(temporary_data))]


            # *** THIS IS WHERE THINGS GO BAD :((
            # loop over X-axis and compare - note that last X-axis is NOT considered
            # Deal with last X-axis separatetly 
            for idx1 in range(len(X_axis_round)-1):
                pool = temporary_data[idx1]
                removal_counter = int(0)
                for idx2 in range(idx1+1,len(X_axis_round),len(X_axis_round)):
                    if len(X_axis_round[idx1]) == len(X_axis_round[idx2]) and np.allclose(X_axis_round[idx1],X_axis_round[idx2]):
                        # pool the data because the X-axis and time point is the same
                        pool = np.concatenate((pool, temporary_data[idx2][:,1:]),axis=1)
                        removal_counter += 1

                        # remove the time points included in the pool so they are not dublicated
                        # !!! TIME POINTS SEEMS TO BE REMOVED BUT DUPPLICATES ARE STILL OCCURING?!? !!!
                        index = int(num_repititions[idx2])
                        print 'Removing index: %i, delay %s' % (index, item)
                        times = [times[int(num)] for num in range(len(times)) if num is not index]

                time_string = item + repetitions_strings[idx_different]
                pooled_times.append(time_string)
                data.append(pool)
                idx_different += 1

           # deal with last X-axis in case it is not pooled
           if removal_counter + 1 < len(times):  # True if last data-set could not be pooled
               time_string = item + repetitions_strings[idx_different]
               pooled_times.append(time_string)
               data.append(temporary_data[-1])
               index = num_repititions[-1]
               times = [times[int(num)] for num in range(len(times)) if num is not index]

    return pooled_times, data

正如你所看到的，我正在从列表中删除我正在迭代的条目（次），直觉上这听起来像是一个非常糟糕的主意。从我的测试看起来循环迭代原始Times列表中的所有条目，因此在循环期间删除不起作用 - 但我想不出更好的方法来做到这一点 - 输入将非常感激！

任何更聪明的方法来进行这种池化/合并，或者制作

for num, item in enumerate(times):

使用当前的'时间'列表而不是原始列表？

任何帮助都会非常受欢迎:)

在函数中进行多重比较的Pythonic方法

0 个答案: