我有一个简单的并行程序，我成功地在 Dask 本地运行。好极了！现在我想将它移动到一个集群并扩大问题的规模。在这种情况下，我使用 GCP。我尝试了两种方法，GCPCluster() 和 HelmCluster()，每种方法都提供了不同的失败路径。 （我之前已经成功实例化了 GCE 计算。因此，我们可以假设我已经解决了所有的安全/登录凭据。网络可能是一个不同的故事。）以下是主要例程：
from dask import delayed
from dask.distributed import Client, wait, as_completed, LocalCluster
from dask_kubernetes import HelmCluster
from dask_cloudprovider.gcp import GCPCluster
from problem.loop import inner_loop
from problem.problemSpec import problemInit
# gRange = 99
gRange = 12
def phase_transition(client: Client):
p = problemInit()
m = p.m
loop = delayed(inner_loop)
loops = [loop(int(m[i])) for i in range(gRange)]
# visualize(loops, filename='delayed_results', format='svg')
futures = client.compute(loops)
wait(futures)
for future, result in as_completed(futures, with_results=True):
print(result)
if __name__ == "__main__":
# with LocalCluster(dashboard_address='localhost:8787') as cluster:
with GCPCluster(projectid='random-words-654321', machine_type='n1-standard-4', n_workers=2) as cluster:
with Client(cluster) as client:
phase_transition(client)
使用 GCPCluster() 时，系统等待调度程序的响应。以下是日志消息：
Launching cluster with the following configuration:
Source Image: projects/ubuntu-os-cloud/global/images/ubuntu-minimal-1804-bionic-v20201014
Docker Image: daskdev/dask:latest
Machine Type: n1-standard-4
Filesytsem Size: 50
Disk Type: pd-standard
N-GPU Type:
Zone: us-east1-c
Creating scheduler instance
dask-837e1ad1-scheduler
Internal IP: 10.142.0.4
External IP: 35.237.42.13
Waiting for scheduler to run at 35.237.42.13:8786
scheduler 系统已启动，我可以SSH 进入。看起来像一些网络问题。 （顺便说一句，我正在使用类似于 daskdev/dask:latest 调用的图像的 Conda 图像从 PyCharm 运行它。）显然，我们甚至还没有开始在云上安装本地代码。
这是某种问题，Dask 和 GCP 的经验将解决，我还没有经验。因此，请允许我通过文档尝试不同的路径并启动由 Helm 管理的 k8s 集群。对我的代码的唯一更改是：
if __name__ == "__main__":
cluster = HelmCluster(release_name='gke-dask')
with Client(cluster) as client:
phase_transition(client)
这运行得更好。现在在我的本地机器上的子目录 problem 中查找代码时出现问题。以下是日志：
Forwarding from 127.0.0.1:65410 -> 8786
Forwarding from [::1]:65410 -> 8786
Handling connection for 65410
Handling connection for 65410
/Users/awd/opt/anaconda3/envs/dask-cvxpy/lib/python3.8/site-packages/distributed/client.py:1140: VersionMismatchWarning: Mismatched versions found
+---------+---------------+---------------+---------------+
| Package | client | scheduler | workers |
+---------+---------------+---------------+---------------+
| blosc | None | 1.9.2 | 1.9.2 |
| lz4 | 3.1.3 | 3.1.1 | 3.1.1 |
| msgpack | 1.0.2 | 1.0.0 | 1.0.0 |
| numpy | 1.20.2 | 1.18.1 | 1.18.1 |
| python | 3.8.8.final.0 | 3.8.0.final.0 | 3.8.0.final.0 |
+---------+---------------+---------------+---------------+
Notes:
- msgpack: Variation is ok, as long as everything is above 0.6
warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Handling connection for 65410
Handling connection for 65410
Handling connection for 65410
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "/Users/awd/Projects/Stats285/ExamplePhaseTransition/main_func.py", line 39, in <module>
phase_transition(client)
File "/Users/awd/Projects/Stats285/ExamplePhaseTransition/main_func.py", line 28, in phase_transition
for future, result in as_completed(futures, with_results=True):
File "/Users/awd/opt/anaconda3/envs/dask-cvxpy/lib/python3.8/site-packages/distributed/client.py", line 4336, in __next__
return self._get_and_raise()
File "/Users/awd/opt/anaconda3/envs/dask-cvxpy/lib/python3.8/site-packages/distributed/client.py", line 4327, in _get_and_raise
raise exc.with_traceback(tb)
File "/opt/conda/lib/python3.8/site-packages/distributed/protocol/pickle.py", line 75, in loads
ModuleNotFoundError: No module named 'problem'
在实践中，我正在寻求有关任一问题的帮助。我对 GCPCluster() 解决方案略有偏好。