如何让cURL绕过这个严格的僵尸僵尸？

Question

我需要能够以编程方式读取或删除此URL的内容：

https://www.toysrus.com/product?productId=122872496

我尝试过设置用户代理，Cookie，各种各样的。我还试过Chrome＆＃39;复制为cURL＆＃39;来自Developer Tools-＆gt; Network的命令，它给出了cURL bash命令：

curl 'https://www.toysrus.com/product?productId=122872496' -H 'if-none-match: W/"2318d-c7pvMOCypyagcnOWwd+nRwWuY3M"' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-GB,en;q=0.8,en-US;q=0.6' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'cache-control: max-age=0' -H 'authority: www.toysrus.com' -H 'cookie: GUID=15F299082881F0F; AMCVS_B9241CFE53309B6A0A490D45%40AdobeOrg=1; _abck=CA9FF35985405E426CA113364E2843B95C7A361ECC7F0000CAC6E559CE126315~0~gKgTXIlJpZdnvAbG5IkQwQDNivNRDOZlT2fWOD3JtYs=~-1~-1; entry_domain=toysrus; UrCapture=32c9b0fc-a726-20df-9a6f-62e5b646ae80; Omni_ShopperID=59e5c6d2f0ada2826d8ffee0; invodoSession=qooDReJQgHIivnpF47JvZK; invodoVisitor=A2PNHsy8xb7HkbYFs0WmFv; RES_TRACKINGID=582138174429307; ResonanceSegment=1; ak_bmsc=07593E6331CACE711A54A233912585F35C7A361E8B6000009068E85921E49529~pl51CsTeE5lRXhxDt77+3bSwhUsz0NHEbHDZviNsf5zsoLDiMPES8naaZBbn66eCN80VGsaW+FKJpBRsZmnYSIdqgplNwdRljB7RHos/Mz880QrJOIiIMx8wkCaJ2s5NYFQ3idkdIeql4t4RztiH2oRraA8Bue0r0Ck9wsc8yXmtDZ1hDdGPn4+LqZtrMhlDxZ6cwu1TODBrQSBd5Ni5eGTRvJe1/oauUHr9ztu6ayRaw=; bm_sz=3234D15C4A9BAF5E11789569DCD4FD6C~QAAQHjZ6XO9Ayy5fAQAAt3TYMwFkUQkzB9j2O8d5YvvKQfRZdHumrTmjvdQ1iPQKYi+F0p7HcKT0cVl7JYImlqc9DVfB6C3At4C+9S4b+B4iLA2IAi9fU7gcH6GfDoQxi0iPBbQLk98gkY1cqAPfQbUsjwcUbJ0l+JYY7QjWSET3e054Al8z5jVSTGpFVqlx; AMCV_B9241CFE53309B6A0A490D45%40AdobeOrg=-1330315163%7CMCIDTS%7C17459%7CMCMID%7C52999333410090394193783160145267952168%7CMCAAMLH-1508835659%7C6%7CMCAAMB-1509008146%7CxfQZlSXQfPyHZHNXzhww8jeiMq5K2sHrXyNvM9qY7m_hTGU%7CMCOPTOUT-1508410546s%7CNONE%7CMCAID%7CNONE; TRUSESSIONID=0KYz7VGR4y7wsZpUXcvpdLFQfnRdJxJJf1w6M9JbLJVNqzu1xIPV!-1065198256; mbox=PC#7086875003bc497a93590bb613c34b50.17_21#1509615639|session#888c37d2b3124c40878eb0a1792e3265#1508407899|check#true#1508406099; s_ppvl=en_US%253A%2520TRU%253A%2520Product%2520Detail%253A%2520122872496%253A%2520LEGO%2520City%2520Jungle%2520Explorers%2520Jungle%2520Halftrack%2520Mission%2520%252860159%2529%2C21%2C21%2C759%2C1536%2C759%2C1536%2C864%2C1.25%2CL; s_cc=true; __utma=136881205.1239421491.1508230865.1508350392.1508403351.3; __utmc=136881205; __utmz=136881205.1508403351.3.3.utmcsr=lookingforagooddeal.com|utmccn=(referral)|utmcmd=referral|utmcct=/product/test-product/; __pr.7tp=rgrgcsjgwm; utag_main=v_id:015f29909284001f515df5e18dfc04073003306b00978$_sn:3$_ss:0$_st:1508407846067$ses_id:1508403348621%3Bexp-session$_pn:5%3Bexp-session; _cp_dtm_store_name=toysrus; OAS_SC1=1508406046406; IR_PI=1508230866697.bml86prs4m; _br_uid_2=uid%3D6871658334943%3Av%3D12.0%3Ats%3D1508230867182%3Ahc%3D10; PrefID=21-5485403154; s_nr=1508407773155-Repeat; s_previousPageName=en_US%3A%20TRU%3A%20Product%20Detail%3A%20122872496%3A%20LEGO%20City%20Jungle%20Explorers%20Jungle%20Halftrack%20Mission%20%2860159%29; s_cpm=%5B%5B%27ch%27%2C%271508407773158%27%5D%5D; TRUSESSIONID_SET_TIMESTAMP=1508407838524; bm_sv=08D1BE1D2408D1E12F94213DC168DABC~B3fBu73AESdKjl6l/qB2U3aCBKuZw4SCcjmyc+h+CWYxT1NJ0fXXP6GlFba3e+tKujhnRf47tnMJHAn83pK816cqr6bxeL5m0yn1yo9SPOiYNXSZsE35r/9sYgAQhRWDl7Gd4AljS43B80m36ONZMb6q9kL9aT3oJ3sooY69nAE=; s_ppv=en_US%253A%2520TRU%253A%2520Product%2520Detail%253A%2520122872496%253A%2520LEGO%2520City%2520Jungle%2520Explorers%2520Jungle%2520Halftrack%2520Mission%2520%252860159%2529%2C21%2C21%2C759%2C1536%2C419%2C1536%2C864%2C1.25%2CL; RT="sl=0&ss=1508406037346&tt=0&obo=0&sh=&dm=toysrus.com&si=d98b4ff0-ddd2-47e6-8a7c-761ce2d21e15&bcn=%2F%2F36e4f0e7.akstat.io%2F&ld=1508406046108&r=https%3A%2F%2Fwww.toysrus.com%2Fproduct%3F328e8990a18940b61934f007b9211205&ul=1508407960823"' -H 'referer: https://foundthru.com/product/lego-city-jungle-explorers/' --compressed

连接启动，但它只是超时。

我也尝试过：

wget http://www.toysrus.com

哪个给出了这个，然后永远挂起（除非我添加超时）：

我也尝试过使用node.js，phantomjs和horseman，使用这样的代码：

horseman
    .cookies(params.cookies_file_path)
    .open(params.url)
    .status()
    .then(function(status) {
        horseman.html()
        .then(function(html) {
                callback({status: status, data: html});
                return horseman.close();
        });
    })
    .catch(function (err) {
        console.log(err);
        horseman.close();
        callback(false);
    });

我错过了什么？如何获取cURL或phantomJS来获取此页面？

Answer 1

我在PHP中使用curl很多。您可以根据自己的环境调整我发布的内容。

没有僵尸阻滞剂。这是一个简单的https：// GET
您可能挂断了没有重定向到https：//

将网址更改为https：//
使用http：//导致重定向到https：//
那么你不想要SSL证书问题：
添加curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

此代码经过测试并有效：

<?php  
header('Content-Type: text/plain; charset=utf-8');
$request = array();
$request[] = 'host:www.toysrus.com';
$request[] = 'Connection: keep-alive';
$request[] = 'Pragma: no-cache';
$request[] = 'Cache-Control: no-cache';
$request[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8';
$request[] = 'User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.101 Safari/537.36';
$request[] = 'DNT: 1';
$request[] = 'Accept-Encoding: gzip, deflate';
$request[] = 'Accept-Language: en-US,en;q=0.8';


$url = 'https://www.toysrus.com/product?productId=122872496';
$ch = curl_init($url);

curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $request);
curl_setopt($ch, CURLOPT_ENCODING,"");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TIMEOUT,10);
curl_setopt($ch, CURLOPT_FAILONERROR,true);
$data = curl_exec($ch);
if (curl_errno($ch)){
   echo 'Retreive Base Page Error: ' . curl_error($ch);
}
else {
  $info = rawurldecode(var_export(curl_getinfo($ch),true));
  echo $data;

}
?>