Question

有人可以向我解释为什么最后一个示例（示例3. Net-SNMP的多进程包装器）在下一页中：https://www.ibm.com/developerworks/aix/library/au-multiprocessing/不会引发PicklingError？

我尝试使用自己的绑定方法更新并返回实例属性（类似于更新并返回实例属性的示例），并引发以下错误：

 Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
PicklingError: Can't pickle <type 'thread.lock'>: attribute lookup thread.lock failed

这是我的代码：

from multiprocessing import Process, Queue
import requests

class MyClass(object):

    def do_request(self, url):
        try:
            self.response = requests.get(url)
        except:
            self.response = None
        return self.response

def make_request(url):
    s = MyClass()
    return s.do_request(url)

# Function run by worker processes
def worker(input, output):
    for func in iter(input.get, 'STOP'):
        result = make_request(func)
        output.put(result)

def main():
    """Runs everything"""

    #clients
    urls = ['http://www.google.com', 'http://www.amazon.com']
    NUMBER_OF_PROCESSES = len(urls)

    # Create queues
    task_queue = Queue()
    done_queue = Queue()

    #submit tasks
    for url in urls:
        task_queue.put(url)

    #Start worker processes
    for i in range(NUMBER_OF_PROCESSES):
        Process(target=worker, args=(task_queue, done_queue)).start()

     # Get and print results
    print 'Unordered results:'
    for i in range(len(urls)):
        print '\t', done_queue.get()

    # Tell child processes to stop
    for i in range(NUMBER_OF_PROCESSES):
        task_queue.put('STOP')
        print "Stopping Process #%s" % i

if __name__ == "__main__":
    main()

Answer 1

问题是来自requests.get（）的返回不是可选对象。您需要提取所需的信息并将其返回给父对象。就个人而言，我喜欢保留简单类型以及此类事物的列表和词汇 - 它可以将可能发生的坏事数量保持在最低限度。

这是一个在我的linux + python 2.7机器上抛出一个更混乱的异常的实验，但是让你知道这个问题：

>>> import requests
>>> import pickle
>>> r=requests.get('http://google.com')
>>> pickle.dumps(r)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)

  ... many lines removed

  File "/usr/lib/python2.7/copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

您可以将self.response = requests.get(url)替换为self.response = "hello"来测试我是否正确。

Python多处理：不引发PicklingError的示例

1 个答案: