Question

背景

我希望编写使用Microsoft.VisualBasic.FileIO.TextFieldParser来解析某些csv数据的代码。我正在生成此数据的系统不理解引号;所以我无法逃脱分界线;而是必须更换它。我使用上面的文本解析器找到了一个解决方案，但我只看到人们使用它来处理来自文件的输入。我不是将我的数据写入文件只是为了再次导入它，而是将内容保存在内存中/利用这个接受流作为输入的类构造函数。

理想情况下，它可以直接从用于管道的任何内存流中获取数据;但我无法弄清楚如何访问它。在我当前的代码中，我创建了自己的内存流并从管道向它提供数据;然后尝试从中读取。不幸的是我错过了什么。

问题

如何在PowerShell中读取/写入内存流？
是否可以直接从正在输入函数管道的流中读取？

代码

clear-host
[Reflection.Assembly]::LoadWithPartialName("System.IO") | out-null
#[Reflection.Assembly]::LoadWithPartialName("Microsoft.VisualBasic") | out-null

function Clean-CsvStream {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true, ValueFromPipeline=$true)]
        [string]$Line
        ,
        [Parameter(Mandatory = $false)]
        [char]$Delimiter = ','
    )
    begin {
        [System.IO.MemoryStream]$memStream = New-Object System.IO.MemoryStream
        [System.IO.StreamWriter]$writeStream = New-Object System.IO.StreamWriter($memStream)
        [System.IO.StreamReader]$readStream = New-Object System.IO.StreamReader($memStream)
        #[Microsoft.VisualBasic.FileIO.TextFieldParser]$Parser = new-object Microsoft.VisualBasic.FileIO.TextFieldParser($memStream)
        #$Parser.SetDelimiters($Delimiter)
        #$Parser.HasFieldsEnclosedInQuotes = $true
        #$writeStream.AutoFlush = $true
    }
    process {
        $writeStream.WriteLine($_)
        #$writeStream.Flush() #maybe we need to flush it before the reader will see it?
        write-output $readStream.ReadLine()
        #("Line: {0:000}" -f $Parser.LineNumber)
        #write-output $Parser.ReadFields()
    }
    end {
        #close streams and dispose (dodgy catch all's in case object's disposed before we call Dispose)
        #try {$Parser.Close(); $Parser.Dispose()} catch{} 
        try {$readStream.Close(); $readStream.Dispose()} catch{} 
        try {$writeStream.Close(); $writeStream.Dispose()} catch{} 
        try {$memStream.Close(); $memStream.Dispose()} catch{} 
    }
}
1,2,3,4 | Clean-CsvStream -$Delimiter ';' #nothing like the real data, but I'm not interested in actual CSV cleansing at this point

解决方法

与此同时，我的解决方案只是替换对象的属性而不是CSV行。

$cols = $objectArray | Get-Member | ?{$_.MemberType -eq 'NoteProperty'} | select -ExpandProperty name
$objectArray | %{$csvRow =$_; ($cols | %{($csvRow.$_ -replace "[`n,]",':')}) -join ',' }

更新

我意识到丢失的代码是$memStream.Seek(0, [System.IO.SeekOrigin]::Begin) | out-null;

然而，这并不完全符合预期;即我的CSV的第一行显示两次，其他输出的顺序错误;所以我可能误解了如何使用Seek。

clear-host
[Reflection.Assembly]::LoadWithPartialName("System.IO") | out-null
[Reflection.Assembly]::LoadWithPartialName("Microsoft.VisualBasic") | out-null

function Clean-CsvStream {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true, ValueFromPipeline=$true)]
        [string]$CsvRow
        ,
        [Parameter(Mandatory = $false)]
        [char]$Delimiter = ','
        ,
        [Parameter(Mandatory = $false)]
        [regex]$InvalidCharRegex 
        ,
        [Parameter(Mandatory = $false)]
        [string]$ReplacementString 

    )
    begin {
        [System.IO.MemoryStream]$memStream = New-Object System.IO.MemoryStream
        [System.IO.StreamWriter]$writeStream = New-Object System.IO.StreamWriter($memStream)
        [Microsoft.VisualBasic.FileIO.TextFieldParser]$Parser = new-object Microsoft.VisualBasic.FileIO.TextFieldParser($memStream)
        $Parser.SetDelimiters($Delimiter)
        $Parser.HasFieldsEnclosedInQuotes = $true
        $writeStream.AutoFlush = $true
    }
    process {
        if ($InvalidCharRegex) {
            $writeStream.WriteLine($CsvRow)
            #flush here if not auto
            $memStream.Seek(0, [System.IO.SeekOrigin]::Begin) | out-null;
            write-output (($Parser.ReadFields() | %{$_ -replace $InvalidCharRegex,$ReplacementString }) -join $Delimiter)
        } else { #if we're not replacing anything, keep it simple
            $CsvRow
        }
    }
    end {
        "end {"
        try {$Parser.Close(); $Parser.Dispose()} catch{} 
        try {$writeStream.Close(); $writeStream.Dispose()} catch{} 
        try {$memStream.Close(); $memStream.Dispose()} catch{} 
        "} #end"
    }
}
$csv = @(
    (new-object -TypeName PSCustomObject -Property @{A="this is regular text";B="nothing to see here";C="all should be good"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text2";B="what the`nLine break!";C="all should be good2"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text3";B="ooh`r`nwindows line break!";C="all should be good3"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text4";B="I've got;a semi";C="all should be good4"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text5";B="""You're Joking!"" said the Developer`r`n""No honestly; it's all about the secret VB library"" responded the Google search result";C="all should be good5"})
) | convertto-csv -Delimiter ';' -NoTypeInformation
$csv | Clean-CsvStream -Delimiter ';' -InvalidCharRegex "[`r`n;]" -ReplacementString ':'

Answer 1

经过大量的游戏后，它似乎有效：

clear-host
[Reflection.Assembly]::LoadWithPartialName("System.IO") | out-null
[Reflection.Assembly]::LoadWithPartialName("Microsoft.VisualBasic") | out-null

function Clean-CsvStream {
    [CmdletBinding()]
    param (
        [Parameter(Mandatory = $true, ValueFromPipeline=$true)]
        [string]$CsvRow
        ,
        [Parameter(Mandatory = $false)]
        [char]$Delimiter = ','
        ,
        [Parameter(Mandatory = $false)]
        [regex]$InvalidCharRegex 
        ,
        [Parameter(Mandatory = $false)]
        [string]$ReplacementString 
    )
    begin {
        [bool]$IsSimple = [string]::IsNullOrEmpty($InvalidCharRegex) 
        if(-not $IsSimple) {
            [System.IO.MemoryStream]$memStream = New-Object System.IO.MemoryStream
            [System.IO.StreamWriter]$writeStream = New-Object System.IO.StreamWriter($memStream)
            [Microsoft.VisualBasic.FileIO.TextFieldParser]$Parser = new-object Microsoft.VisualBasic.FileIO.TextFieldParser($memStream)
            $Parser.SetDelimiters($Delimiter)
            $Parser.HasFieldsEnclosedInQuotes = $true
        }
    }
    process {
        if ($IsSimple) {
            $CsvRow
        } else { #if we're not replacing anything, keep it simple
            [long]$seekStart = $memStream.Seek(0, [System.IO.SeekOrigin]::Current) 
            $writeStream.WriteLine($CsvRow)
            $writeStream.Flush()
            $memStream.Seek($seekStart, [System.IO.SeekOrigin]::Begin) | out-null 
            write-output (($Parser.ReadFields() | %{$_ -replace $InvalidCharRegex,$ReplacementString }) -join $Delimiter)
        }
    }
    end {
        if(-not $IsSimple) {
            try {$Parser.Close(); $Parser.Dispose()} catch{} 
            try {$writeStream.Close(); $writeStream.Dispose()} catch{} 
            try {$memStream.Close(); $memStream.Dispose()} catch{} 
        }
    }
}
$csv = @(
    (new-object -TypeName PSCustomObject -Property @{A="this is regular text";B="nothing to see here";C="all should be good"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text2";B="what the`nLine break!";C="all should be good2"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text3";B="ooh`r`nwindows line break!";C="all should be good3"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text4";B="I've got;a semi";C="all should be good4"}) 
    ,(new-object -TypeName PSCustomObject -Property @{A="this is regular text5";B="""You're Joking!"" said the Developer`r`n""No honestly; it's all about the secret VB library"" responded the Google search result";C="all should be good5"})
) | convertto-csv -Delimiter ';' -NoTypeInformation
$csv | Clean-CsvStream -Delimiter ';' -InvalidCharRegex "[`r`n;]" -ReplacementString ':'

即。

在撰写之前寻求当前的位置
然后写
然后刷新（如果不是自动）
然后寻求数据的开始
然后阅读
重复

我不确定这是否正确;因为我找不到任何好的例子或文档解释，所以只是玩了一些模糊的有意义的事情。

如果有人知道如何从管道流直接读取，我仍然感兴趣;即删除奖金流的额外开销。

对于@ M.R.的评论

对不起，这太晚了;万一它对别人有用：

如果行结尾分隔符是CrLf（\r\n）而不仅仅是Cr（\r），那么很容易消除记录/行的结尾与字段内的换行符之间的歧义：

Get-Content -LiteralPath 'D:\test\file to clean.csv' -Delimiter "`r`n" | 
%{$_.ToString().TrimEnd("`r`n")} | #the delimiter is left on the end of the string; remove it
%{('"{0}"' -f $_) -replace '\|','"|"'} | #insert quotes at start and end of line, as well as around delimeters
ConvertFrom-Csv -Delimiter '|' #treat the pipeline content as a valid pipe delimitted csv

然而，如果不是，你将无法告诉哪个Cr是记录的结尾，哪个只是文本的中断。你可以通过计算管道的数量来稍微解决这个问题;即好像你有5列，第四个分隔符之前的任何CR都是换行符而不是记录的结尾。但是，如果有另一个换行符，则无法确定这是否是最后一列数据中的换行符，或者该行的结尾。如果您知道第一列或最后一列不包含换行符（或两者都有），您可以解决这个问题。对于所有这些更复杂的场景，我怀疑正则表达式是最好的选择;使用select-string之类的东西来应用它。如果需要;在这里发布一个问题，提出您的确切要求＆amp;有关您已经尝试过的信息以及其他人可以帮助您的信息。

从PowerShell中的Pipeline Stream读取

1 个答案: