因此,我开始尝试使用Python的多处理库。我的目标是加快一个慢速函数的运行速度,该函数将字符串与其他字符串的大型数据库进行比较,并返回最相似的匹配项。为此,我尝试编写一个函数,使用共享变量捕获结果,以将该任务拆分为不同的Process对象,并使它们运行。
cores = cpu_count() # Number of cores in this computer, i.e. 4
sublistList = chunks(tasks,cores) # Split tasks into subprocessing arrays of evenly-sized chunks, the number of chunks equal to how many cores we have to process them
# Create a multiprocessing function, since this is a large function that will take time and splitting it across cores will ease the load
if __name__ == '__main__':
freeze_support() # Make sure multiple applications don't spawn, this is necessary for Windows
jobs = [] # Array of processes
manager = Manager() # Create a manager
returns = manager.list() # Shared list variable we use to get return results
for i in range(0,cores): # For number of cores...
p = Process(target=workerFunction,args=(w,sublistList[i],returns))
jobs.append(p) # Add to array of processes to run
p.start()
for p in jobs:
p.join()
但是,当我运行此代码时,它将创建一个新的应用程序窗口,然后无限期挂起,这完全是奇怪的行为,根本不是我想要的。是什么在我的代码中引起这种情况?我的工作人员功能是否在默默地崩溃并且没有提醒我?我查看了各种其他答案,但是没有一个建议的答案似乎可以解决此问题。
(如果与问题相关,我是一名入门级软件工程师,具有几年使用其他语言的编程经验,但是对于Python来说还比较陌生。这是我的一个小型独立游戏项目。)
答案 0 :(得分:1)
(目前)还没有答案,但我将其发布给您以展示可运行 Minimal, Complete, and Verifiable Example的示例。
代码基于您的问题中的当前内容,以及使它可运行所需的所有其他内容。毫不奇怪,由于所有这些只是猜测,它不会重现您说的问题,但这很可能是由于我的一个或多个猜测在某些重要方面有所不同...这就是为什么您真的应该提供所有代码的原因。
一个观察结果:最后的p.join()
调用将使主进程等待每个子进程完成。这将导致在等待每个主进程时主进程似乎“挂起”。
from multiprocessing import *
from time import sleep
tasks = None
def chunks(tasks, cores):
return [[i for _ in range(8)] for i in range(cores)]
def workerFunction(w, sublist, returns):
print('starting workerFunction:', w)
result = [value+100 for value in sublist]
returns.append(result)
sleep(3)
print('exiting workerFunction:', w)
if __name__ == '__main__':
# Only do in main process.
freeze_support()
cores = cpu_count()
sublistList = chunks(tasks, cores)
manager = Manager()
returns = manager.list()
jobs = []
for i in range(cores):
w = i
p = Process(target=workerFunction, args=(w, sublistList[i], returns))
jobs.append(p)
p.start()
for i, p in enumerate(jobs, 1):
print('joining job[{}]'.format(i))
p.join()
# Display results.
for sublist in returns:
print(sublist)
print('done')
输出:
joining job[1]
starting workerFunction: 2
starting workerFunction: 1
starting workerFunction: 0
starting workerFunction: 5
starting workerFunction: 7
starting workerFunction: 3
starting workerFunction: 4
starting workerFunction: 6
exiting workerFunction: 2
exiting workerFunction: 0
exiting workerFunction: 1
joining job[2]
exiting workerFunction: 5
joining job[3]
joining job[4]
exiting workerFunction: 7
exiting workerFunction: 3
exiting workerFunction: 4
joining job[5]
exiting workerFunction: 6
joining job[6]
joining job[7]
joining job[8]
[102, 102, 102, 102, 102, 102, 102, 102]
[101, 101, 101, 101, 101, 101, 101, 101]
[100, 100, 100, 100, 100, 100, 100, 100]
[105, 105, 105, 105, 105, 105, 105, 105]
[107, 107, 107, 107, 107, 107, 107, 107]
[103, 103, 103, 103, 103, 103, 103, 103]
[104, 104, 104, 104, 104, 104, 104, 104]
[106, 106, 106, 106, 106, 106, 106, 106]
done
Press any key to continue . . .