我正在尝试使用PowerShell将非常大的VM映像（5-15 Gb大小）上传到HTTP服务器。
我尝试使用这几种方法（此处链接到script with net.WebClient.UploadFile和script with Invoke-webRequest）
适用于小于2GB的文件，但不适用于大于此文件的文件。
我正在尝试直接使用httpWebRequest但我无法将<div>
<input type="text" value="Text">
<input type="text" placeholder="Text">
<span>Text</span>
</div>放入其中。
所以我的问题是：如何将文件流放入webrequest？
或者更一般地说：如何使用PowerShell通过http上传大文件？
WebElement element = driver.findElement(By.id("the_id"));
element.sendKeys("necessary_text");
答案 0 :(得分：8)
默认情况下，HttpWebRequest正在缓冲内存中的数据。 只需将HttpWebRequest.AllowWriteStreamBuffering属性设置为false，您就可以上传几乎任何大小的文件。 请参阅msdn
了解更多详情答案 1 :(得分：8)
谢谢@Stoune，最后一件事有助于获得最终的解决方案。
另外，需要组织流文件读取和写入webrequest缓冲区。它可能与那段代码有关：
$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
{
$requestStream.write($chunk, 0, $bytesRead)
$requestStream.Flush()
}
最终脚本看起来像这样：
$user = "admin"
$pass = "admin123"
$dir = "C:\Virtual Hard Disks"
$fileName = "win2012r2std.vhdx"
$file = "$dir/$fileName"
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"
$Timeout=10000000
$bufSize=10000
$cred = New-Object System.Net.NetworkCredential($user, $pass)
$webRequest = [System.Net.HttpWebRequest]::Create($url)
$webRequest.Timeout = $timeout
$webRequest.Method = "POST"
$webRequest.ContentType = "application/data"
$webRequest.AllowWriteStreamBuffering=$false
$webRequest.SendChunked=$true # needed by previous line
$webRequest.Credentials = $cred
$requestStream = $webRequest.GetRequestStream()
$fileStream = [System.IO.File]::OpenRead($file)
$chunk = New-Object byte[] $bufSize
while( $bytesRead = $fileStream.Read($chunk,0,$bufsize) )
{
$requestStream.write($chunk, 0, $bytesRead)
$requestStream.Flush()
}
$responceStream = $webRequest.getresponse()
#$status = $webRequest.statuscode
$FileStream.Close()
$requestStream.Close()
$responceStream.Close()
$responceStream
$responceStream.GetResponseHeader("Content-Length")
$responceStream.StatusCode
#$status
答案 2 :(得分：1)
要上传到Sonatype Nexus3，我使用了以下代码。花了一些时间来弄明白，使用Nexus3的上传和响应以及上传和下载大文件（大于2GB）。我们在Nexus3前面安装Apache来处理https连接。 使用基本身份验证确保Nexus在Apache处于其前面时响应正确。 通过HEAD发送预身份验证并对大型文件使用分块上传，修复了上传大文件并未过早结束的问题。
通过Invoke-WebRequest下载大文件也会因为不同的错误而失败。现在我通过.Net使用WebRequest并且它可以正常工作，请参阅Download-File。
当代码通过自动化流程（来自System Center Orchestrator）运行时，https将失败。因此，当检测到方案https时，我们强制使用TLS 1.2。
function New-HttpWebRequest
{
<#
.SYNOPSIS
Creates a new [System.Net.HttpWebRequest] ready for file transmission.
.DESCRIPTION
Creates a new [System.Net.HttpWebRequest] ready for file transmission.
The method will be Put. If the filesize is larger than the buffersize,
the HttpWebRequest will be configured for chunked transfer.
.PARAMETER Url
Url to connect to.
.PARAMETER Credential
Credential for authentication at the Url resource.
.EXAMPLE
An example
#>
param(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$Url,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[pscredential]$Credential,
[Parameter(Mandatory=$true)]
[long]$FileSize,
[Parameter(Mandatory=$true)]
[long]$BufferSize
)
$webRequest = [System.Net.HttpWebRequest]::Create($Url)
$webRequest.Timeout = 600 * 1000;
$webRequest.ReadWriteTimeout = 600 * 1000;
$webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11;
$webRequest.Method = "PUT";
$webRequest.ContentType = "application/octet-stream";
$webRequest.KeepAlive = $true;
$webRequest.UserAgent = "<I use a specific UserAgent>";
#$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)';
$webRequest.PreAuthenticate = $true;
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
$webRequest.Headers["Authorization"] = "Basic $auth"
if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize)
{
Write-Verbose "FileSize is greater than BufferSize, using chunked transfer.";
$webRequest.AllowWriteStreamBuffering = $false;
$webRequest.SendChunked = $true;
}
else
{
# Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write.
# Chunked cannot be used in this case.
$webRequest.AllowWriteStreamBuffering = $true;
$webRequest.SendChunked = $false;
$webRequest.ContentLength = $FileSize;
}
return $webRequest;
}
function Get-BufferSize
{
<#
.SYNOPSIS
Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size.
.DESCRIPTION
Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default.
.PARAMETER ByteLength
Length of the bytes for which to calculate a valid buffer size.
.PARAMETER AtLeast
The minimum required buffer size, default 1MB.
.EXAMPLE
Get-BufferSize 4283304773
Returns 42991616 which is 41MB.
.EXAMPLE
Get-BufferSize 4283304
Returns 1048576 which is 1MB.
.EXAMPLE
Get-BufferSize 4283304 5MB
Returns 5242880 which is 5MB.
#>
param(
[Parameter(Mandatory=$true)]
[long]$ByteLength,
[long]$AtLeast = 1MB
)
[long]$size = $ByteLength / 100;
if ($size -lt $AtLeast)
{
$size = $AtLeast;
}
else
{
$size = [Math]::Round($size / 1MB) * 1MB;
}
return $size;
}
function Get-UseChunkedUpload
{
param(
[Parameter(Mandatory=$true)]
[long]$FileSize,
[Parameter(Mandatory=$true)]
[long]$BufferSize
)
return $FileSize -gt $BufferSize;
}
function Configure-Tls
{
param(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$Url
)
[System.Uri]$uri = $Url;
if ($uri.Scheme -eq "https")
{
Write-Verbose "Using TLS 1.2";
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12;
}
}
function Send-PreAuthenticate
{
param(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$Url,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[pscredential]$Credential
)
$response = $null;
try
{
[System.Uri]$uri = $Url;
$repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/');
Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority";
$wr = [System.Net.WebRequest]::Create($repositoryAuthority);
$wr.Method = "HEAD";
$wr.PreAuthenticate = $true;
$auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password));
$wr.Headers["Authorization"] = "Basic $auth"
$response = $wr.GetResponse();
}
finally
{
if ($response)
{
$response.Close();
$response.Dispose();
$response = $null;
}
}
}
function Upload-File
{
<#
.SYNOPSIS
Uploads a file to the Nexus repository.
.DESCRIPTION
Uploads a file to the Nexus repository.
If the file was uploaded successfully, the url via which the resource can be downloaded is returned.
.PARAMETER Url
The Url where the resource should be created.
Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload.
.PARAMETER File
The file that should be uploaded.
.PARAMETER Credential
Credential used for authentication at the Nexus repository.
.EXAMPLE
Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential)
.OUTPUTS
If the file was uploaded successfully, the url via which the resource can be downloaded.
#>
param(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$Url,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[System.IO.FileInfo]$File,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[pscredential]$Credential
)
Write-Verbose "Upload-File Url:$Url"
Configure-Tls -Url $Url;
$fileSizeBytes = $File.Length;
#$bufSize = Get-BufferSize $fileSizeBytes;
$bufSize = 4 * 1MB;
Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB));
if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize)
{
Write-Verbose "Using chunked upload. Send pre-auth first.";
Send-PreAuthenticate -Url $Url -Credential $Credential;
}
$progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length);
$webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize;
$chunk = New-Object byte[] $bufSize;
$bytesWritten = 0;
$fileStream = [System.IO.File]::OpenRead($File.FullName);
$requestStream = $WebRequest.GetRequestStream();
try
{
while($bytesRead = $fileStream.Read($chunk,0,$bufSize))
{
$requestStream.Write($chunk, 0, $bytesRead);
$requestStream.Flush();
$bytesWritten += $bytesRead;
$progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten / 1MB));
Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100);
}
}
catch
{
throw;
}
finally
{
if ($fileStream)
{
$fileStream.Close();
}
if ($requestStream)
{
$requestStream.Close();
$requestStream.Dispose();
$requestStream = $null;
}
Write-Progress -Activity $progressActivityMessage -Completed;
}
# Read the response.
$response = $null;
try
{
$response = $webRequest.GetResponse();
Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri);
return $response.ResponseUri;
}
catch
{
if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*"))
{
throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.")
}
throw;
}
finally
{
if ($response)
{
$response.Close();
$response.Dispose();
$response = $null;
}
if ($webRequest)
{
$webRequest = $null;
}
}
}
function Download-File
{
param(
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$Url,
[Parameter(Mandatory=$true)]
[ValidateNotNullOrEmpty()]
[string]$FileName
)
$SDXDownloadType = @"
using System.IO;
using System.Net;
public class SDXDownload
{
static public void DownloadFile(string Uri, string Filename)
{
HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri);
webRequest.Method = "GET";
using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse())
using (Stream fileStream = File.OpenWrite(Filename))
using (Stream streamResponse = myHttpWebResponse.GetResponseStream())
{
int bufSize = 64 * 1024;
byte[] readBuff = new byte[bufSize];
int bytesRead = streamResponse.Read(readBuff, 0, bufSize);
while (bytesRead > 0)
{
fileStream.Write(readBuff, 0, bytesRead);
bytesRead = streamResponse.Read(readBuff, 0, 256);
}
}
}
}
"@
Configure-Tls -Url $Url;
Add-Type -TypeDefinition $SDXDownloadType;
[SDXDownload]::DownloadFile($Url, $FileName);
}