使用python的多处理星图的线程安全计数器

时间:2016-11-10 15:05:30

标签: python-3.x sqlite multiprocessing shared-memory

目前我正试图处理来自非常快速计算的结果。首先,我将每个模拟结果插入到sqlite数据库中,但结果却是整个计算的瓶颈。所以我最终使用了cursor.executemany而不是cursor.execute,这要快得多。

我现在的问题是我无法以某种方式实现线程安全计数器。

每1000次计算应执行executemany任务。因此,我实现了一个带有多处理的初始化程序。值得我尝试了这个解决方案(http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing),但不知何故,计数器的某些值是重复的,最终会将executemany任务运行到常常或根本不运行。

如果有人知道如何解决这个问题,我真的很感激。

以下是最低样本:

import multiprocessing, sqlite3
from multiprocessing import Value, Lock
from itertools import repeat

def worker(Testvalues, TotalValues):
    MP_counter.value += 1
    counter.increment()

    con = sqlite3.connect("Test.db", timeout=30.0)
    cur = con.cursor()
    # Minimum sample:
    Helper = list(range(5))
    Helper = [x * Testvalues for x in Helper]
    GList.append(Helper)

    Execute_Every = 10
    print("Counter class: %d" % (counter.value()))
    print("MP_counter: %d" % (MP_counter.value))

    if counter.value() % Execute_Every == 0 or counter.value() == TotalValues - 1:
        print("Execute query")
        print("Counter class: %d" % (counter.value()))
        print("MP_counter: %d" % (MP_counter.value))

        Helper = [tuple(row) for row in GList[:Execute_Every]]
        del GList[:Execute_Every]
        cur.executemany(
            "INSERT INTO Test (One, Two, Three, Four, Five) VALUES (?, ?, ?, ?, ?);", Helper)
        con.commit()

    con.close()

def setup(t, g, c):
    global MP_counter
    global GList
    global counter
    MP_counter = t
    GList = g
    counter = c

class Counter(object):
    def __init__(self, initval=0):
        self.val = Value('i', initval)
        self.lock = Lock()

    def increment(self):
        with self.lock:
            self.val.value += 1

    def value(self):
        with self.lock:
            return self.val.value

if __name__ == '__main__':
    m = multiprocessing.Manager()
    CPUS = multiprocessing.cpu_count()
    MP_counter = multiprocessing.Value('i', 0)
    GList = m.list([])
    thread_safe_counter = Counter(0)

    l = multiprocessing.Lock()
    WORKERS = multiprocessing.Pool(initializer=setup, initargs=[MP_counter, GList, thread_safe_counter],processes=CPUS)

    con = sqlite3.connect("Test.db", timeout=30.0)
    cur = con.cursor()
    cur.execute('PRAGMA journal_mode=wal')
    SQLCommand = "CREATE TABLE IF NOT EXISTS Test (One INT, Two INT, Three INT, Four INT, Five INT);"
    cur.execute(SQLCommand)
    con.close()

    TotalValues = 100
    Testvalues = list(range(TotalValues))

    WORKERS.starmap(worker, zip(Testvalues, repeat(TotalValues)))
    WORKERS.close()
    WORKERS.join()
    #Check if list is empty
    print(GList)

谢谢你们:)

1 个答案:

答案 0 :(得分:1)

你的计数器有一个value()和一个increment()方法,需要单独调用,所以为了保证安全,你必须在持有锁的同时调用这两个操作。您的value()方法应该在递增后返回新值,并且您应该在不再调用class Counter(object): def __init__(self, initval=0): self.val = Value('i', initval) self.lock = Lock() def increment(self): with self.lock: self.val.value += 1 return self.val.value ... def worker(Testvalues, TotalValues): counter_value = counter.increment() # use only counter_value from here on ... 的情况下使用它,例如:

RLock

此外,已经使用默认class Counter(object): def __init__(self, initval=0): self.val = Value('i', initval) # or Value('i', initval, lock=Lock()) def increment(self): with self.val.get_lock(): self.val.value += 1 return self.val.value 创建了Value,如果需要,可以在构造函数调用中使用不同的锁类型覆盖{{3}}。所以你真的不需要分配自己的锁,你可以使用:

cat kernel/config-x86_64-4.4.11 | grep MMC
# CONFIG_PCI_MMCONFIG is not set
CONFIG_MMC=y
CONFIG_MMC_DEBUG=y
# MMC/SD/SDIO Card Drivers
CONFIG_MMC_BLOCK=m
CONFIG_MMC_BLOCK_MINORS=8
CONFIG_MMC_BLOCK_BOUNCE=y
CONFIG_MMC_TEST=y
# MMC/SD/SDIO Host Controller Drivers
CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_PCI=y
# CONFIG_MMC_RICOH_MMC is not set
CONFIG_MMC_SDHCI_ACPI=y
CONFIG_MMC_SDHCI_PLTFM=y
CONFIG_MMC_WBSD=y
CONFIG_MMC_TIFM_SD=y
CONFIG_MMC_SDRICOH_CS=y
CONFIG_MMC_CB710=y
CONFIG_MMC_VIA_SDMMC=y
CONFIG_MMC_VUB300=y
CONFIG_MMC_USHC=y
CONFIG_MMC_USDHI6ROL0=y
# CONFIG_MMC_REALTEK_USB is not set
CONFIG_MMC_TOSHIBA_PCI=y
CONFIG_MMC_MTK=y

CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_PCI=y
CONFIG_MMC_SDHCI_ACPI=y
CONFIG_MMC_SDHCI_PLTFM=y