如何在Python中解释这种内存使用情况?

时间:2018-08-23 08:16:01

标签: python memory jupyter-notebook

出于深度学习目的,我已经在Jupyter Notebook中编写了以下代码:

X_train_imbalanced=[]
Y_train_imbalanced=[]
# create sequences
look_back=50
for k in range (0, len (train_list)):
    print (k)
    #print (len(train_list[k].iloc[:,1:nb_features-1].values))
    temp_data, temp_target= create_dataset (train_list[k].iloc[:,1:nb_features-1].values, train_list[k].iloc[:,nb_features-1].values, look_back)
#     print (len(temp_data))
#     print (len (temp_data[0]))
#     print (len (temp_target))
    if (k==0) :
        X_train_imbalanced=temp_data
        Y_train_imbalanced=temp_target
    else :
        %memit X_train_imbalanced=np.concatenate((X_train_imbalanced,temp_data), axis=0)
        Y_train_imbalanced=np.concatenate((Y_train_imbalanced,temp_target), axis=0)

其中temp_data是大小为(?, 50,22)的3-D数组,其中?一次迭代之间有所不同,但很大(大约30000)。

当我运行代码时,它在第20次迭代时变得非常慢,这就是%memit给出的:

1
peak memory: 1221.69 MiB, increment: 309.11 MiB
2
peak memory: 1313.96 MiB, increment: 366.71 MiB
3
peak memory: 1370.51 MiB, increment: 358.19 MiB
4
peak memory: 1396.97 MiB, increment: 370.20 MiB
5
peak memory: 1666.82 MiB, increment: 509.88 MiB
6
peak memory: 1803.96 MiB, increment: 517.50 MiB
7
peak memory: 1738.25 MiB, increment: 583.27 MiB
8
peak memory: 1788.33 MiB, increment: 781.89 MiB
9
peak memory: 2107.31 MiB, increment: 966.23 MiB
10
peak memory: 2060.38 MiB, increment: 903.95 MiB
11
peak memory: 2210.77 MiB, increment: 1000.03 MiB
12
peak memory: 2179.66 MiB, increment: 915.61 MiB
13
peak memory: 2267.32 MiB, increment: 1683.86 MiB
14
peak memory: 2455.46 MiB, increment: 1156.93 MiB
15
peak memory: 2508.21 MiB, increment: 938.65 MiB
16
peak memory: 3160.77 MiB, increment: 1485.71 MiB
17
peak memory: 3416.46 MiB, increment: 1600.22 MiB
18
peak memory: 3512.63 MiB, increment: 1616.89 MiB
19
peak memory: 3725.57 MiB, increment: 2344.89 MiB
20
peak memory: 3776.23 MiB, increment: 1777.52 MiB
21
peak memory: 3835.47 MiB, increment: 1702.66 MiB
22
peak memory: 3949.54 MiB, increment: 3490.01 MiB

但是,在实践中我很难解释……我有6GB的RAM,以上结果是否可以解释为什么代码运行缓慢并几乎使机器崩溃?

如何避免这种情况?有没有优化代码的方法?还是不是在笔记本电脑上而是在更大的基础架构上运行代码更好?

我对应该做什么感到困惑...

0 个答案:

没有答案