Question

我有一个python代码，它使用子进程包在shell中运行：

subprocess.call(mycode.py, shell=inshell)

当我执行top命令时，我发现我只使用了~30％或更少的CPU。我意识到一些命令可能是使用磁盘而不是cpu，因此我计时速度。在Linux系统上运行它的速度似乎比mac 2核心系统慢。

如何将其与线程或多处理程序包并行化，以便在所述linux系统上使用多个CPU内核？

Answer 1

要并行化mycode.py中完成的工作，您需要组织代码以使其适合这种基本模式：

# Import the kind of pool you want to use (processes or threads).
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

# Collect work items as an iterable of single values (eg tuples, 
# dicts, or objects). If you can't hold all items in memory,
# define a function that yields work items instead.
work_items = [
    (1, 'A', True),
    (2, 'X', False),
    ...
]

# Define a callable to do the work. It should take one work item.
def worker(tup):
    # Do the work.
    ...

    # Return any results.
    ...

# Create a ThreadPool (or a process Pool) of desired size.
# What size? Experiment. Slowly increase until it stops helping.
pool = ThreadPool(4)

# Do work and collect results.
# Or use pool.imap() or pool.imap_unordered().
work_results = pool.map(worker, work_items)

# Wrap up.
pool.close()
pool.join()

---------------------

# Or, in Python 3.3+ you can do it like this, skipping the wrap-up code.
with ThreadPool(4) as pool:
    work_results = pool.map(worker, work_items)

Answer 2

好吧，您可以先创建一个线程，然后将要并行化的函数传递给它。在函数内部你有子进程。

import threading
import subprocess

def worker():
    """thread worker function"""
    print 'Worker'
    subprocess.call(mycode.py, shell=inshell)
    return

threads = []
for i in range(5):
    t = threading.Thread(target=worker)
    threads.append(t)
    t.start()

Answer 3

对FMc的回答略有改动，

work_items = [(1, 'A', True), (2, 'X', False), (3, 'B', False)]
def worker(tup):
 for i in range(5000000):
     print(work_items)
 return

pool = Pool(processes = 8)
start = time.time()
work_results = pool.map(worker, work_items)
end = time.time()
print(end-start)
pool.close()
pool.join()

上面的代码需要53.60秒。然而，下面的技巧需要27.34秒。

from multiprocessing import Pool
import functools
import time

work_items = [(1, 'A', True), (2, 'X', False), (3, 'B', False)]

def worker(tup):
    for i in range(5000000):
        print(work_items)
    return

def parallel_attribute(worker):
    def easy_parallelize(worker, work_items):
        pool = Pool(processes = 8)
        work_results = pool.map(worker, work_items)
        pool.close()
        pool.join()
    from functools import partial 
    return partial(easy_parallelize, worker)

start = time.time()
worker.parallel = parallel_attribute(worker(work_items))
end = time.time()
print(end - start)

两条评论： 1）我没有看到使用多处理虚拟的差异 2）使用Python的部分功能（带嵌套的范围）就像一个精彩的包装器，将计算时间减少1/2。参考：https://www.binpress.com/tutorial/simple-python-parallelism/121

另外，谢谢FMc！

python在多个CPU核心上传播subprocess.call

3 个答案: