Question

我想知道如何从单个文件夹中读取多个json文件（不指定文件名，只是它们是json文件）。

此外，可以将它们变成pandas DataFrame？

你能给我一个基本的例子吗？

Answer 1

一个选项是使用os.listdir列出目录中的所有文件，然后仅查找以'.json'结尾的文件：

import os, json
import pandas as pd

path_to_json = 'somedir/'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]
print(json_files)  # for me this prints ['foo.json']

现在你可以使用pandas DataFrame.from_dict将json（此时是一个python字典）读入一个pandas数据帧：

montreal_json = pd.DataFrame.from_dict(many_jsons[0])
print montreal_json['features'][0]['geometry']

打印：

{u'type': u'Point', u'coordinates': [-73.6051013, 45.5115944]}

在这种情况下，我将一些jsons附加到列表many_jsons。我列表中的第一个json实际上是geojson，其中包含蒙特利尔的一些地理数据。我已经熟悉了这些内容，所以我打印出了'几何'，它给了我蒙特利尔的lon / lat。

以下代码总结了以上所有内容：

import os, json
import pandas as pd

# this finds our json files
path_to_json = 'json/'
json_files = [pos_json for pos_json in os.listdir(path_to_json) if pos_json.endswith('.json')]

# here I define my pandas Dataframe with the columns I want to get from the json
jsons_data = pd.DataFrame(columns=['country', 'city', 'long/lat'])

# we need both the json and an index number so use enumerate()
for index, js in enumerate(json_files):
    with open(os.path.join(path_to_json, js)) as json_file:
        json_text = json.load(json_file)

        # here you need to know the layout of your json and each json has to have
        # the same structure (obviously not the structure I have here)
        country = json_text['features'][0]['properties']['country']
        city = json_text['features'][0]['properties']['name']
        lonlat = json_text['features'][0]['geometry']['coordinates']
        # here I push a list of data into a pandas DataFrame at row given by 'index'
        jsons_data.loc[index] = [country, city, lonlat]

# now that we have the pertinent json data in our DataFrame let's look at it
print(jsons_data)

对我来说这打印：

  country           city                   long/lat
0  Canada  Montreal city  [-73.6051013, 45.5115944]
1  Canada        Toronto  [-79.3849008, 43.6529206]

知道对于这段代码我在目录名'json'中有两个geojsons可能会有所帮助。每个json都有以下结构：

{"features":
[{"properties":
{"osm_key":"boundary","extent":
[-73.9729016,45.7047897,-73.4734865,45.4100756],
"name":"Montreal city","state":"Quebec","osm_id":1634158,
"osm_type":"R","osm_value":"administrative","country":"Canada"},
"type":"Feature","geometry":
{"type":"Point","coordinates":
[-73.6051013,45.5115944]}}],
"type":"FeatureCollection"}

Answer 2

使用glob模块

可以轻松迭代（平面）目录

from glob import glob

for f_name in glob('foo/*.json'):
    ...

至于将JSON直接读入pandas，请参阅here。

Answer 3

将来自特定目录的所有以* .json结尾的文件加载到字典中：

import os,json

path_to_json = '/lala/'

for file_name in [file for file in os.listdir(path_to_json) if file.endswith('.json')]:
  with open(path_to_json + file_name) as json_file:
    data = json.load(json_file)
    print(data)

亲自尝试： https://repl.it/@SmaMa/loadjsonfilesfromfolderintodict

Answer 4

要读取json文件，

Reduce(`+`,lapply(list(DF2,DF3,DF4),`[`,order(names(DF1))),init=DF1) # here
aggregate(.~Type, rbind(df1,df2,df3,df4), sum)                       # @MrFlick

Answer 5

如果要转换为熊猫数据框，请使用熊猫API。

通常，您可以使用发电机。

def data_generator(my_path_regex):
    for filename in glob.glob(my_path_regex):
        for json_line in open(filename, 'r'):
            yield json.loads(json_line)


my_arr = [_json for _json in data_generator(my_path_regex)]

Python：从文件夹中读取几个json文件

5 个答案: