如何将ipython集群控制器和引擎子类化以添加选项?

时间:2015-04-21 19:14:29

标签: parallel-processing ipython pbs ipython-parallel

我有基本的PBS控制器和EngineSet Launchers与我的PBS /扭矩集群一起工作。但现在我想通过配置文件和/或命令行传递其他选项,以使用{}模板格式化程序语法在batch_template文件中呈现。

举个例子,我想跑:

ipcluster start -n 10 --pmem=10gb

然后,在我的模板中:

#PBS -l pmem={pmem}

我想我必须继承PBSControllerLauncher& PBSEngineSetLauncher类?文档中提到了这一点,但我找不到如何做的明确示例。

只需从这些类继承并添加pmem = Unicode(...)作为类属性,在调用write_batch_script时,模板的上下文中就不会显示pmem。

我可以访问配置选项并将其插入上下文,如下所示。这段代码有效,但很尴尬,似乎不是正确的方法:

from IPython.parallel.apps.launcher import PBSControllerLauncher, PBSEngineSetLauncher, BatchClusterAppMixin, ipcontroller_cmd_argv, ipengine_cmd_argv
from IPython.utils.traitlets import (
        Any, Integer, CFloat, List, Unicode, Dict, Instance, HasTraits, CRegExp
        )
import pipes

class MyPBSLauncherExtension(object):
    def update_context(self):
        cfg = self.config
        ctx = self.context
        cls = type(self).__name__
        for c in ('extra_pbs_directives', 'extra_modules'):
            if cls in cfg:
                if c in cfg[cls]:
                    ctx[c] = cfg[cls][c]
                    continue
            if c in cfg:
                ctx[c] = cfg[c]
            else:
                ctx[c] = getattr(self, c)


class MyPBSControllerLauncher(PBSControllerLauncher, MyPBSLauncherExtension):
    extra_pbs_directives = Unicode(u'', config=True, help='')
    extra_modules = Unicode(u'', config=True, help='')


    def write_batch_script(self, n):
        self.update_context()
        super(MyPBSControllerLauncher, self).write_batch_script(n)

class MyPBSEngineSetLauncher(PBSEngineSetLauncher, MyPBSLauncherExtension):
    extra_pbs_directives = Unicode(u'', config=True, help='')
    extra_modules = Unicode(u'', config=True, help='')


    def write_batch_script(self, n):
        self.update_context()
        super(MyPBSEngineSetLauncher, self).write_batch_script(n)

非常感谢为启动器子类添加附加选项的简单示例。

0 个答案:

没有答案