Question

我尝试使用C＃解析此网页，但我不确定如何循环浏览不同的网页。我希望只使用每个页面的URL，但看起来他们正在使用一堆JavaScript调用来进入下一页，因此URL实际上并未在页面之间进行更改。有谁知道我怎么能这样做？

网页：http://www.roads.maryland.gov/pages/cic.aspx?PageId=857&Type=tab

提前致谢。

Answer 1

HTML Agility Pack应该可以做到。您可以使用它来模拟按钮点击，这将正确激活JavaScript。

这样的事情可能有用（假设你的表单上有webbrowser控件）：

var browser = new IE(webBrowser1.ActiveXInstance);
var page2Link = (Link)browser.Elements.Where(e => e.InnerHtml.Contains("Page$2")).First();
page2Link.Click();

Answer 2

您需要使用所有视图状态POST到相同的网址，然后将__EVENTARGUMENT帖子值更改为Page $ 1，Page $ 2，Page $ 3等

像这样的东西

    //Create request to URL.
    HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.roads.maryland.gov/pages/cic.aspx?PageId=857&Type=tab");

    //Set request headers.
    request.KeepAlive = true;
    request.Headers.Add("Origin", @"http://www.roads.maryland.gov");
    request.Headers.Add("X-MicrosoftAjax", "Delta=true");
    request.UserAgent = "Mozilla/1.0";
    request.ContentType = "application/x-www-form-urlencoded; charset=UTF-8";
    request.Accept = "*/*";
    request.Headers.Set(HttpRequestHeader.AcceptLanguage, "en-GB,en;q=0.8,en-US;q=0.6");

    //Set request method
    request.Method = "POST";

    // Disable 'Expect: 100-continue' behavior. More info: http://haacked.com/archive/2004/05/15/http-web-request-expect-100-continue.aspx
    request.ServicePoint.Expect100Continue = false;

    //Set request body.
    string body = "__EVENTARGUMENT=Page%24";
    body += pageNumber.ToString();
    body += @"&ctl00%24PlaceHolderMain%24ScriptManager1=ctl00%24PlaceHolderMain%24CICContract1%24displayPanel1%7Cctl00%24PlaceHolderMain%24CICContract1%24gvSSP&MSO_PageHashCode=687&MSOWebPartPage_PostbackSource=&MSOTlPn_SelectedWpId=&MSOTlPn_View=0&MSOTlPn_ShowSettings=False&MSOGallery_SelectedLibrary=&MSOGallery_FilterString=&MSOTlPn_Button=none&__REQUESTDIGEST=0x8DBC9CD602B61A910B12C83C03A47B6755DE07797515E34AEA8EE0BDC93813FC6A208251090430008C51E50BF4A3D0EBEAF4A9F2CC60072EC4B6B5FED3F48D31%2C20%20Nov%202014%2017%3A15%3A59%20-0000&MSOSPWebPartManager_DisplayModeName=Browse&MSOWebPartPage_Shared=&MSOLayout_LayoutChanges=&MSOLayout_InDesignMode=&MSOSPWebPartManager_OldDisplayModeName=Browse&MSOSPWebPartManager_StartWebPartEditingName=false&ctl00%24PlaceHolderSearchArea%24q=Search&ctl00%24PlaceHolderMain%24CICContract1%24ddlPulldown=tab&__EVENTTARGET=ctl00%24PlaceHolderMain%24CICContract1%24gvSSP&__LASTFOCUS=&__VIEWSTATE=%2F 

...massive viewstate...

    %3D&__VIEWSTATEGENERATOR=E6DD55AA&__EVENTVALIDATION=%2FwEWKwK7ur20CQKpuunwCgLbzK2CBQKBio7OBALDpsvcBQK6mIPlBQLa0ZnPCgLmidHRCQKu%2FbCaDQKvg%2BHxDwKl%2FYSaDQK87ov7BQKV8JfxAQKdo6DlBgLSppvcBQLFutGoAwK3suC5CgK3spS5CgK3sui5CgK3svC5CgK3suS5CgK3svi5CgK3ssy5CgK3ssC5CgLZuKSJCwKykYb0DgKrq%2FH8DAK3zM7sDALA%2BfaWBAKYyYezCwK3suC5CgK3spS5CgK3sui5CgK3svC5CgK3suS5CgK3svi5CgK3ssy5CgK3ssC5CgLZuKSJCwKykYb0DgKrq%2FH8DALl54b0BgK%2FjseDBA0f8JbX2FIE%2F7%2FP1ojwVriOGXif&__ASYNCPOST=true&";

    byte[] postBytes = System.Text.Encoding.UTF8.GetBytes(body);
    request.ContentLength = postBytes.Length;
    Stream stream = request.GetRequestStream();
    stream.Write(postBytes, 0, postBytes.Length);
    stream.Close();

    //Get response to request.
    response = (HttpWebResponse)request.GetResponse();

循环使用javascript的页面

2 个答案: