Question

我是ipyparallel的新手，我想使用这个软件包来实现我的机器学习应用程序的并行计算。

以下是对ipyparallel的测试，我在func.py文件中定义了一个名为add的函数，在test.py文件中定义了一个函数。

func.py的代码是：

#!/usr/bin/env python
# coding=utf-8

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers

test.py的代码是：

#!/usr/bin/env python
# coding=utf-8

from func import add
from ipyparallel import Client

if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3]
    print rc[0].map_sync(add, [1, 2, 3, 4])

由于您知道map可以正常运行，但在运行map_sync时，命令行会返回：

☁  test  python test.py 
[[2], [3], [4]]
Traceback (most recent call last):
  File "test.py", line 14, in <module>
    print rc[0].map_sync(add, [1, 2, 3, 4])
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 353, in map_sync
    return self.map(f,*sequences,**kwargs)
  File "<string>", line 2, in map
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 54, in sync_results
    ret = f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/view.py", line 618, in map
    return pf.map(*sequences)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 268, in map
    ret = self(*sequences)
  File "<string>", line 2, in __call__
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 75, in sync_view_results
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/remotefunction.py", line 251, in __call__
    return r.get()
  File "/usr/local/lib/python2.7/dist-packages/ipyparallel/client/asyncresult.py", line 104, in get
    raise self._exception
ipyparallel.error.CompositeError: one or more exceptions from call to method: add
[0:apply]: ImportError: No module named func

如果我在test.py文件中定义函数，map_sync可以运行：

#!/usr/bin/env python
# coding=utf-8

#from func import add
from ipyparallel import Client

def add(*numbers):
    numbers = list(numbers)
    for i, n in enumerate(numbers):
        numbers[i] = n + 1
    return numbers


if __name__ == '__main__':
    rc = Client(
        '/home/fit/.ipython/profile_default/security/ipcontroller-client.json')

    print map(add, [1, 2, 3])

    print rc[0].map_sync(add, [1, 2, 3, 4])

结果是：

☁  test  python test.py
[[2], [3], [4]]
[[2], [3], [4], [5]]

我想知道map_sync如何在其他文件中使用函数define？以及如何导入这些功能？由于from py_file import func不适用于map_sync。

Answer 1

所需的模块应该被复制（或者你可以推送或模块）到engine machines，并且engine machines上应该安装三方软件包，如果没有，ImportError将是出错。

但是，在运行程序时，您应该运行：

$ ipcontroller --ip=client_ip --reuse

client machine上的

，/home/user/.ipython/profile_default/security目录

中将生成2个文件

$ ls /home/user/.ipython/profile_default/security 
ipcontroller-client.json  ipcontroller-engine.json

因此，必须将ipcontroller-client.json和ipcontroller-engine.json复制到engine machines并运行

$ ipengine --file=/path/to/ipcontroller-engine.json

在engine machines上，因此已设置parallel computing环境。

接下来，您可以定义parallel computing任务并运行程序。

IPython ipyparallel map_sync ImportError

1 个答案: