从单个文件中读取多个数据集

Question

我的文本文件包含每个数据库的表。有没有什么方法可以让pandas读取这个文件并为每个数据库创建单独的数据框？

Database: ABC
+-----------------------------------------------+----------+------------+
|                    Tables                     | Columns  | Total Rows |
+-----------------------------------------------+----------+------------+
| ApplicationUpdateBankLog                      |       13 |          0 |
| ChangeLogTemp                                 |       12 |    1678363 |
| Sheet2$                                       |       10 |        359 |
| tempAllowApplications                         |        1 |          9 |
+-----------------------------------------------+----------+------------+
4 rows in set.


Database: XYZ
+--------------------------------------------------+----------+------------+
|                      Tables                      | Columns  | Total Rows |
+--------------------------------------------------+----------+------------+
| BKP_QualificationDetails_12082014                |       14 |    7959877 |
| BillNotGeneratedCount                            |       11 |       2312 |
| VVshipBenefit                                    |       19 |     197356 |
| VVBenefit_Bkup29012016                           |       19 |     101318 |
+--------------------------------------------------+----------+------------+
4 rows in set.

Answer 1

您可以使用dict comprehension创建dict的{​​{1}}：

DataFrames

import pandas as pd
from pandas.compat import StringIO

temp=u"""Database: ABC
+-----------------------------------------------+----------+------------+
|                    Tables                     | Columns  | Total Rows |
+-----------------------------------------------+----------+------------+
| ApplicationUpdateBankLog                      |       13 |          0 |
| ChangeLogTemp                                 |       12 |    1678363 |
| Sheet2$                                       |       10 |        359 |
| tempAllowApplications                         |        1 |          9 |
+-----------------------------------------------+----------+------------+
4 rows in set.


Database: XYZ
+--------------------------------------------------+----------+------------+
|                      Tables                      | Columns  | Total Rows |
+--------------------------------------------------+----------+------------+
| BKP_QualificationDetails_12082014                |       14 |    7959877 |
| BillNotGeneratedCount                            |       11 |       2312 |
| VVshipBenefit                                    |       19 |     197356 |
| VVBenefit_Bkup29012016                           |       19 |     101318 |
+--------------------------------------------------+----------+------------+
4 rows in set."""
#after testing replace 'StringIO(temp)' to 'filename.csv'
df = pd.read_csv(StringIO(temp), sep="|", names=['a', 'Tables', 'Columns', 'Total Rows'])

#replace NaN in column a created where not 'Database' by forward filing
df.a = df.a.where(df.a.str.startswith('Database')).ffill()
#remove rows where NaN in Tables column 
df = df.dropna(subset=['Tables'])
#remove all whitespaces, set index for selecting in dict comprehension
df = df.apply(lambda x: x.str.strip()).set_index('a')
#convert to numeric columns, replace NaN, convert to int 
df['Columns'] = pd.to_numeric(df['Columns'], errors='coerce').fillna(0).astype(int)
df['Total Rows'] = pd.to_numeric(df['Total Rows'], errors='coerce').fillna(0).astype(int)
#remove rows with value Tables 
df = df[df['Tables'] != 'Tables']
print (df)
                                          Tables  Columns  Total Rows
a                                                                    
Database: ABC           ApplicationUpdateBankLog       13           0
Database: ABC                      ChangeLogTemp       12     1678363
Database: ABC                            Sheet2$       10         359
Database: ABC              tempAllowApplications        1           9
Database: XYZ  BKP_QualificationDetails_12082014       14     7959877
Database: XYZ              BillNotGeneratedCount       11        2312
Database: XYZ                      VVshipBenefit       19      197356
Database: XYZ             VVBenefit_Bkup29012016       19      101318

#select in dict comprehension and reset index to default monotonic index
dfs = {x:df.loc[x].reset_index(drop=True) for x in df.index.unique()}