我有两个矩阵，我需要用它来创建一个更大的矩阵。每个矩阵只是一个以制表符分隔的文本文件，可以读取。每个矩阵有48个列，每个矩阵具有相同的标识符，具有不同的行数。第一个矩阵是108887x48，第二个矩阵是55482x48。每个矩阵的每个位置的条目可以是0或1，所以二进制。最终输出应该有第一个矩阵行id作为行，第二个矩阵行id作为cols，所以最终矩阵是55482x10887。
这里需要发生的是，对于第一个矩阵中每行中的每个pos，对于第二个矩阵中的每一行，如果每个矩阵的pos（col）为1，那么最终矩阵计数将增加1最终矩阵中任何位置的最高值可以是48，并且预计会剩下0。
示例：
mat1
A B C D
1id1 0 1 0 1
1id2 1 1 0 0
1id3 1 1 1 1
1id4 0 0 1 0
mat2
A B C D
2id1 1 1 0 0
2id2 0 1 1 0
2id3 1 1 1 1
2id4 1 0 1 0
final
2id1 2id2 2id3 2id4
1id1 1 1 2 0
1id2 2 1 2 1
1id3 2 2 4 2
1id4 0 1 1 1
我有代码可以做到这一点，但是它很慢，这是我主要寻求帮助的地方。我试图尽可能加快算法速度。它已经运行了24小时，而且只有25％左右。我让它在之前运行，最终输出文件是20GB。我没有数据库的经验，可以在这里实现，如果osomeone可以帮助我如何这样做，给出下面的代码片段。
#!/usr/bin/env python
import sys
mat1in = sys.argv[1]
mat2in = sys.argv[2]
print '\n######################################################################################'
print 'Generating matrix by counts from smaller matrices.'
print '########################################################################################\n'
with open(mat1in, 'r') as f:
cols = [''] + next(f).strip().split('\t') # First line of matrix is composed of 48 cols
mat1 = [line.strip().split('\t') for line in f] # Each line in matrix = 'ID': 0 or 1 per col id
with open(mat2in, 'r') as f:
next(f) # Skip first row, col IDs are taken from mat1
mat2 = [line.strip().split('\t') for line in f] # Each line in matrix = 'ID': 0 or 1 per col id
out = open('final_matrix.txt', 'w') # Output file
#matrix = []
header = [] # Final matrix header
header.append('') # Add blank as first char in large matrix header
for i in mat2:
header.append(i[0]) # Composed of all mat2 row ids
#matrix.append(header)
print >> out, '\t'.join(header) # First print header to output file
print '\nTotal mat1 rows: ' + str(len(mat1)) # Get total mat1 rows
print 'Total mat2 rows: ' + str(len(mat2)), '\n' # Get total mat2 rows
print 'Progress: ' # Progress updated as each mat1 id is read
length = len(header) # Length of header, i.e. total number of mat2 ids
totmat1 = len(mat1) # Length of rows (-header), i.e. total number of mat1 ids
total = 0 # Running total - for progress meter
for h in mat1: # Loop through all mat1 ids - each row in the HC matrix
row = [] # Empty list for new row for large matrix
row.append(h[0]) # Append mat1 id, as first item in each row
for i in xrange(length-1): # For length of large matrix header (add 0 to each row) - header -1 for first '\t'
row.extend('0')
for n in xrange(1,49): # Loop through each col id
for k in mat2: # For every row in mat2
if int(h[n]) == 1 and int(k[n]) == 1: # If the pos (count for that particular col id) is 1 from mat1 and mat2 matrix;
pos = header.index(k[0]) # Get the position of the mat2 id
row[pos] = str(int(row[pos]) + 1) # Add 1 to current position in row - [i][j] = [mat1_id][mat2_id]
print >> out, '\t'.join(row) # When row is completed (All columns are compared from both mat1 and mat2 matrices; print final row to large matrix
total += 1 # Update running total
sys.stdout.write('\r\t' + str(total) + '/' + str(tvh)) # Print progress to screen
sys.stdout.flush()
print '\n######################################################################################'
print 'Matrix complete.'
print '########################################################################################\n'
这是在mat1中对ids的前30次迭代进行概况分析：
######################################################################################
Generating matrix by counts from smaller matrices.
########################################################################################
Total mat1 rows: 108887
Total mat2 rows: 55482
Progress:
30/108887^C 2140074 function calls in 101.234 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 70.176 70.176 101.234 101.234 build_matrix.py:3(<module>)
4 0.000 0.000 0.000 0.000 {len}
55514 0.006 0.000 0.006 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
1719942 1.056 0.000 1.056 0.000 {method 'extend' of 'list' objects}
30 0.000 0.000 0.000 0.000 {method 'flush' of 'file' objects}
35776 29.332 0.001 29.332 0.001 {method 'index' of 'list' objects}
31 0.037 0.001 0.037 0.001 {method 'join' of 'str' objects}
164370 0.589 0.000 0.589 0.000 {method 'split' of 'str' objects}
164370 0.033 0.000 0.033 0.000 {method 'strip' of 'str' objects}
30 0.000 0.000 0.000 0.000 {method 'write' of 'file' objects}
2 0.000 0.000 0.000 0.000 {next}
3 0.004 0.001 0.004 0.001 {open}
我也计时每次迭代，每个mat1 id大约需要2.5-3s，如果我正确的话需要大约90小时来完成整个事情。这是关于整个脚本一直运行所需的内容。
我已经编辑了一些主要部分，通过追加和xrange来删除行，通过乘以＆＃39; 0＆＃39;按标题的长度。另外我用索引作为值来制作mat2 id的字典，认为dict查找会比索引更快。
headdict = {}
for k,v in enumerate(header):
headdict[v] = k
total = 0 # Running total - for progress meter
for h in mat1: # Loop through all mat1 ids - each row in the HC matrix
timestart = time.clock()
row = [h[0]] + ['0']*(length-1) # Empty list for new row for large matrix
#row.append(h[0]) # Append mat1 id, as first item in each row
#for i in xrange(length-1): # For length of large matrix header (add 0 to each row) - header -1 for first '\t'
# row.append('0')
for n in xrange(1,49): # Loop through each col id
for k in mat2: # For every row in mat2
if int(h[n]) == 1 and int(k[n]) == 1: # If the pos (count for that particular col id) is 1 from mat1 and mat2 matrix;
pos = headdict[k[0]] #header.index(k[0]) # Get the position of the mat2 id
row[pos] = str(int(row[pos]) + 1) # Add 1 to current position in row - [i][j] = [mat1_id][mat2_id]
print >> out, '\t'.join(row) # When row is completed (All columns are compared from both mat1 and mat2 matrices; print final row to large matrix
total += 1 # Update running total
sys.stdout.write('\r\t' + str(total) + '/' + str(totmat1)) # Print progress to screen
#sys.stdout.flush()
timeend = time.clock()
print timestart - timeend
答案 0 :(得分：2)
这只是一个矩阵乘法。您希望将第一个矩阵乘以第二个矩阵的转置。要获得有效的矩阵运算，请获取NumPy。
如果你将两个输入矩阵读入dtype numpy.int8的NumPy数组，那么计算就是
m1.dot(m2.T)
它需要几分钟，顶部。
答案 1 :(得分：0)
我不太明白这段代码的作用（单字母变量名称没有帮助）。
我的建议：尝试减少在最里面的循环中执行的操作数量。例如，您是否需要在内层重新计算pos？
pos = header.index(k[0])
如果可以对嵌套循环k，h和n进行重新排序，您可以减少代价高昂的list.index，这是O（n）操作。