Question

我使用iterparse解析一个大的xml文件（1.8 gb）。我将所有数据都写到了一个csv文件中。我编写的脚本运行良好，但是由于某种原因，它会随机跳过行。这是我的脚本：

import xml.etree.cElementTree as ET
import csv
xml_data_to_csv =open('Out2.csv','w', newline='', encoding='utf8')
Csv_writer=csv.writer(xml_data_to_csv, delimiter=';')

file_path = "Products_50_producten.xml"
context = ET.iterparse(file_path, events=("start", "end"))

EcommerceProductGuid = ""
ProductNumber = ""
Description = ""
ShopSalesPriceInc = ""
Barcode = ""
AvailabilityStatus = ""
Brand = ""
# turn it into an iterator
#context = iter(context)
product_tag = False
for event, elem in context:
    tag = elem.tag

    if event == 'start' :
        if tag == "Product" :
            product_tag = True

        elif tag == 'EcommerceProductGuid' :
            EcommerceProductGuid = elem.text

        elif tag == 'ProductNumber' :
            ProductNumber = elem.text

        elif tag == 'Description' :
            Description = elem.text

        elif tag == 'SalesPriceInc' :
            ShopSalesPriceInc = elem.text

        elif tag == 'Barcode' :
            Barcode = elem.text

        elif tag == 'AvailabilityStatus' :
            AvailabilityStatus = elem.text


        elif tag == 'Brand' :
            Brand = elem.text

    if event == 'end' and tag =='Product' :
        product_tag = False
        List_nodes = []
        List_nodes.append(EcommerceProductGuid)
        List_nodes.append(ProductNumber)
        List_nodes.append(Description)
        List_nodes.append(ShopSalesPriceInc)
        List_nodes.append(Barcode)
        List_nodes.append(AvailabilityStatus)
        List_nodes.append(Brand)
        Csv_writer.writerow(List_nodes)
        print(EcommerceProductGuid)
        List_nodes.clear()
        EcommerceProductGuid = ""
        ProductNumber = ""
        Description = ""
        ShopSalesPriceInc = ""
        Barcode = ""
        AvailabilityStatus = ""
        Brand = ""

    elem.clear()


xml_data_to_csv.close()

“ Products_50_producten.xml”文件具有以下布局：

<?xml version="1.0" encoding="utf-16" ?>
<ProductExport xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<ExportInfo>
<ExportDateTime>2018-11-07T00:01:03+01:00</ExportDateTime>
<Type>Incremental</Type>
<ExportStarted>Automatic</ExportStarted>
</ExportInfo>
<Products>
<Product><EcommerceProductGuid>4FB8A271-D33E-4501-9EB4-17CFEBDA4177</EcommerceProductGuid><ProductNumber>982301017</ProductNumber><Description>Ducati Jas Radiaal Zwart Xxl Heren Tekst - 982301017</Description><Brand>DUCATI</Brand><ProductVariations><ProductVariation><SalesPriceInc>302.2338</SalesPriceInc><Barcodes><Barcode BarcodeOrder="1">982301017</Barcode></Barcodes></ProductVariation></ProductVariations></Product>
<Product><EcommerceProductGuid>4FB8A271-D33E-4501-9EB4-17CFEBDA4177</EcommerceProductGuid><ProductNumber>982301017</ProductNumber><Description>Ducati Jas Radiaal Zwart Xxl Heren Tekst - 982301017</Description><Brand>DUCATI</Brand><ProductVariations><ProductVariation><SalesPriceInc>302.2338</SalesPriceInc><Barcodes><Barcode BarcodeOrder="1">982301017</Barcode></Barcodes></ProductVariation></ProductVariations></Product>
</Products>

例如，如果我复制“ Product” 300次，则在csv文件中的第155行中，“ EcommerceProductGuid”值保留为空。如果我将产品复制400次，则在155、310和368行上留下一个空值。这怎么可能？

Answer 1

我认为问题出在if event == 'start'。

According to other questions/answers，不能保证定义text属性的内容。

但是，它似乎并不像更改为if event == 'end'那样简单。当我自己尝试时，我得到的空字段多于填充的字段。（更新：如果我从event == 'end'中删除了events=("start", "end")，则使用iterparse确实可行。）

最终起作用的是完全忽略该事件，只是测试看看是否填充了text。

更新的代码...

import xml.etree.cElementTree as ET
import csv

xml_data_to_csv = open('Out2.csv', 'w', newline='', encoding='utf8')
Csv_writer = csv.writer(xml_data_to_csv, delimiter=';')

file_path = "Products_50_producten.xml"
context = ET.iterparse(file_path, events=("start", "end"))

EcommerceProductGuid = ""
ProductNumber = ""
Description = ""
ShopSalesPriceInc = ""
Barcode = ""
AvailabilityStatus = ""
Brand = ""
for event, elem in context:
    tag = elem.tag
    text = elem.text

    if tag == 'EcommerceProductGuid' and text:
        EcommerceProductGuid = text

    elif tag == 'ProductNumber' and text:
        ProductNumber = text

    elif tag == 'Description' and text:
        Description = text

    elif tag == 'SalesPriceInc' and text:
        ShopSalesPriceInc = text

    elif tag == 'Barcode' and text:
        Barcode = text

    elif tag == 'AvailabilityStatus' and text:
        AvailabilityStatus = text

    elif tag == 'Brand' and text:
        Brand = text

    if event == 'end' and tag == "Product":
        product_tag = False
        List_nodes = []
        List_nodes.append(EcommerceProductGuid)
        List_nodes.append(ProductNumber)
        List_nodes.append(Description)
        List_nodes.append(ShopSalesPriceInc)
        List_nodes.append(Barcode)
        List_nodes.append(AvailabilityStatus)
        List_nodes.append(Brand)
        Csv_writer.writerow(List_nodes)
        print(EcommerceProductGuid)
        List_nodes.clear()
        EcommerceProductGuid = ""
        ProductNumber = ""
        Description = ""
        ShopSalesPriceInc = ""
        Barcode = ""
        AvailabilityStatus = ""
        Brand = ""

    elem.clear()

xml_data_to_csv.close()

这似乎与我的300个Product元素的测试文件配合使用。

此外，我认为如果您使用字典和csv.DictWriter，可以简化代码。

示例（与上面的代码产生相同的输出）...

import xml.etree.cElementTree as ET
import csv
from copy import deepcopy

field_names = ['EcommerceProductGuid', 'ProductNumber', 'Description',
               'SalesPriceInc', 'Barcode', 'AvailabilityStatus', 'Brand']

values_template = {'EcommerceProductGuid': "",
                   'ProductNumber': "",
                   'Description': "",
                   'SalesPriceInc': "",
                   'Barcode': "",
                   'AvailabilityStatus': "",
                   'Brand': ""}

with open('Out2.csv', 'w', newline='', encoding='utf8') as xml_data_to_csv:

    csv_writer = csv.DictWriter(xml_data_to_csv, delimiter=';', fieldnames=field_names)

    file_path = "Products_50_producten.xml"
    context = ET.iterparse(file_path, events=("start", "end"))

    values = deepcopy(values_template)

    for event, elem in context:
        tag = elem.tag
        text = elem.text

        if tag in field_names and text:
            values[tag] = text

        if event == 'end' and tag == "Product":
            csv_writer.writerow(values)
            print(values.get('EcommerceProductGuid'))
            values = deepcopy(values_template)

        elem.clear()

Answer 2

对于它的价值以及可能正在搜索的任何人，上述答案也适用于 lxml 库 iterparse()。我在使用 lxml 时遇到了类似的问题，我想我会尝试一下，它的工作原理几乎完全相同。

使用start事件获取xml信息时，随机启动事件还没有拾取到文本项。尝试在结束事件中获取该项目似乎已经用大型 xml 文件解决了我的问题。看起来 Daniel Haley 所做的通过检查文本是否存在增加了另一层保护。

Python iterparse正在跳过值

2 个答案: