Question

我有下面的代码，我使用subprocess.Popen执行4个命令。我正在使用下面的代码处理日志文件。当我使用下面的代码顺序处理文件时，它工作正常。现在我创建了一个线程，一个用于每个文件的并行性，并将函数绑定到每个线程。但其中一些给了我想要的输出和一些抛出错误。

代码：

def process_log_file(file):
    proc= subprocess.Popen(['python27', 'countmapper.py',"C:\\pythonPrograms\\04-03-2014\\17IL\\"+file],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, err = proc.communicate()
    sortedop= subprocess.Popen(['sort'],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE,stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, err = sortedop.communicate(out)
    countReducer= subprocess.Popen(['python27', 'countreducer.py'],cwd="C:\pythonPrograms\\",stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, err = countReducer.communicate(out)
    countpostprocesser= subprocess.Popen(['python27', 'countpostprocesser.py'],cwd="C:\pythonPrograms\\",stdin=subprocess.PIPE,stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    out, err = countpostprocesser.communicate(out)
    jsondata2=json.loads(out)
    fd=open(file+".json","w")
    json.dump(jsondata2,fd,sort_keys=True,indent=2)
    fd.close()
    return

收到错误：

Exception in thread Thread-42:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\pythonPrograms\counts_batch_threading.py", line 45, in process_log_file
    jsondata2=json.loads(out)
  File "C:\Python27\lib\json\__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "C:\Python27\lib\json\decoder.py", line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Python27\lib\json\decoder.py", line 383, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

用于创建线程的代码：

for file in glob.glob("SAMPLE*.log"):
        thread1 = threading.Thread(target=process_log_file,args=(str(file),))
        threads.append(thread1)
        thread1.start()        

# Wait for all threads to complete
for t in threads:
    t.join()

有人可以帮我这个吗？

Answer 1

您的错误可能来自Popen无阻塞这一事实，因此他们会立即尝试读取结果，而不是等待程序终止。请尝试使用check_call，这是一个阻止功能，让您等待获得结果。

在线程函数内捕获subprocess.Popen的输出

1 个答案: