Question

有人知道如何在python中使用巨大的矩阵吗？我必须使用形状的邻接矩阵（10 ^ 6,10 ^ 6）并执行包括加法，缩放和点积的操作。使用numpy数组我遇到了ram的问题。

Answer 1

这样的事情怎么样......

import numpy as np

# Create large arrays x and y.
# Note they are 1e4 not 1e6 b/c of memory issues creating random numpy matrices (CookieOfFortune) 
# However, the same principles apply to larger arrays
x = np.random.randn(10000, 10000)
y = np.random.randn(10000, 10000)

# Create memory maps for x and y arrays
xmap = np.memmap('xfile.dat', dtype='float32', mode='w+', shape=x.shape)
ymap = np.memmap('yfile.dat', dtype='float32', mode='w+', shape=y.shape)

# Fill memory maps with data
xmap[:] = x[:]
ymap[:] = y[:]

# Create memory map for out of core dot product result
prodmap = np.memmap('prodfile.dat', dtype='float32', mode='w+', shape=x.shape)

# Due out of core dot product and write data
prodmap[:] = np.memmap.dot(xmap, ymap)

# Create memory map for out of core addition result
addmap = np.memmap('addfile.dat', dtype='float32', mode='w+', shape=x.shape)

# Due out of core addition and write data
addmap[:] = xmap + ymap

# Create memory map for out of core scaling result
scalemap = np.memmap('scalefile.dat', dtype='float32', mode='w+', shape=x.shape)

# Define scaling constant
scale = 1.3

# Do out of core  scaling and write data
scalemap[:] = scale * xmap

此代码将创建包含二进制格式数组的文件xfile.dat，yfile.dat等。要在以后访问它们，您只需执行np.memmap(filename)。 np.memmap的其他参数是可选的，但是被推荐（dtype，shape等参数）。

在Python中使用巨大矩阵的矩阵运算

1 个答案: