如何使用多达3400台机器快速编写脚本?

时间:2016-12-12 06:33:42

标签: python multiprocessing fabric

我有3400台机器需要将数据发送到石墨,但我的脚本需要5分钟才能完成,如何将其减少到3分钟或2分钟,谢谢,代码如下

def sendmsg(hostname,metric):
        a = mntdir + hostname + '/' + metric + '_median'
        with settings(hide('running'), warn_only=True):
            valueMedian = local(("cat %s|grep %s|awk '{print $2}'") % (a,now),capture=True)
            local(("echo %s.%s_Median_90days %s %s >/dev/udp/20.26.2.18/2001 ") % (hostname,metric,valueMedian,unixdate))


if __name__=='__main__':
    while True:
        localtime = time.asctime( time.localtime(time.time()) )
        date = datetime.datetime.now()
        now1 = date.strftime("%H:%M")
        print now1
        now2 = date.strftime("%M")
        with settings(hide('running'), warn_only=True):
            unixdate = local("date +%s",capture=True)
        if int(now2) % 5 == 0:
            now = now1 + ':00'
            p=Pool(100)
            print now
            for hostname in host:
                for metric in metrics:
                    p.apply_async(sendmsg, args=(hostname,metric))
            starttime = time.asctime( time.localtime(time.time()) )
            print('Waiting for all job done...%s' % starttime)
            p.close()
            p.join()
            stoptime = time.asctime( time.localtime(time.time()) )
            print('sending completed...%s ' % stoptime)
        time.sleep(60)

1 个答案:

答案 0 :(得分:1)

awk '/foo/ {print $2}' file

与您的

大致相同
cat file | grep foo | awk '{print $2}'

并避免创建两个进程,这可能会有所帮助。