Question

我有一个带有很长行的1GB json文件，当我尝试从文件中加载一行时，我从PyCharm控制台收到此错误：

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2017.3.3\helpers\pydev\pydev_run_in_console.py", line 53, in run_file
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "......... .py", line 26, in <module>
    for line in f:
MemoryError
PyDev console: starting.
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:19:30) [MSC v.1500 32 bit (Intel)] on win32

我在拥有64GB内存的Windows Server计算机上。

我的代码是：

import numpy as np
import json
import sys
import re

idRegEx = re.compile(r".*ID=")
endElRegEx = re.compile(r"'.*")

ratingsFile = sys.argv[1]
tweetsFile = sys.argv[2]
outputFile = sys.argv[3]

tweetsMap = {}
with open(tweetsFile, "r") as f:

    for line in f:
        tweetData = json.loads(line)
        tweetsMap[tweetData["key"]] = tweetData

output = open(outputFile, "w")

with open(ratingsFile, "r") as f:
    header = f.next()

    for line in f:
        topicData = line.split("\t")

        topicKey = topicData[0]
        topicTerms = topicData[1]
        ratings = topicData[2]
        reasons = topicData[3]

        ratings = map(lambda x: int(x.strip().replace("'", "")), ratings.replace("[", "").replace("]", "").split(","))
        ratings = np.array(ratings)

        tweetsMap[topicKey]["ratings"] = ratings.tolist()
        tweetsMap[topicKey]["mean"] = ratings.mean()

        topicMap = tweetsMap[topicKey]

        print topicMap["key"], topicMap["mean"]

        json.dump(topicMap, output, sort_keys=True)
        output.write("\n")

output.close()

错误消息中的第26行引用

tweetData = json.loads(line)

而第53行指的是

json.dump(topicMap, output, sort_keys=True)

奇怪的是，我从GitHub分叉了这段代码，所以我认为它应该可行。

Answer 1

看起来您正在使用32位版本的Python：

Python 2.7.14 (...) [MSC v.1500 32 bit (Intel)] on win32

Windows上每个进程的内存限制为2GB，这就是为什么即使你有足够的RAM也会出现内存错误的原因。如果您不想更改脚本，切换到64位版本的Python应该可以解决您的问题。

使用Python在json文件中读取非常长的行时出现内存错误

1 个答案: