Question

如何在以下代码中将开放线程的最大值限制为20？我知道过去曾提出过一些类似的问题，但我特别想知道如何使用队列以及如果可能的工作示例最好的方法。

    # b is a list with 10000 items
    threads = [threading.Thread(target=targetFunction, args=(ptf,anotherarg)) for ptf in b]
    for thread in threads:
        thread.start()

    for thread in threads:
        thread.join()

Answer 1

执行此操作的简单方法是使用queue.Queue进行工作并使用for _ in range(MAXTHREADS): threading.Thread(target=f, args=(the_queue,)).start()启动线程。但是，我发现通过子类化Thread更容易阅读。您的里程可能会有所不同。

import threading
import queue

class Worker(threading.Thread):
    def __init__(self, q, other_arg, *args, **kwargs):
        self.q = q
        self.other_arg = other_arg
        super().__init__(*args, **kwargs)
    def run(self):
        while True:
            try:
                work = self.q.get(timeout=3)  # 3s timeout
            except queue.Empty:
                return
            # do whatever work you have to do on work
            self.q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    Worker(q, otherarg).start()
q.join()  # blocks until the queue is empty.

如果您坚持使用某个功能，我建议您使用知道如何从队列中获取的内容包装您的targetFunction。

def wrapper_targetFunc(f, q, somearg):
    while True:
        try:
            work = q.get(timeout=3)  # or whatever
        except queue.Empty:
            return
        f(work, somearg)
        q.task_done()

q = queue.Queue()
for ptf in b:
    q.put_nowait(ptf)
for _ in range(20):
    threading.Thread(target=wrapper_targetFunc,
                     args=(targetFunction, q, otherarg)).start()
q.join()

线程队列工作示例

1 个答案: