Question

代码简单地插入到页面中，并从指定的表中获取所有表内容，将其插入到我的数据库中并回显它。

它做得非常慢，我需要想法来简化它以更快地工作

<?php

设置循环

$pagenumber = 1001;

while ($pagenumber <= 5000) {

获取内容

$url = "http://www.example.com/info.php?num=$pagenumber";
$raw = file_get_contents($url);

$newlines = array("\t","\n","\r","&nbsp;","\0","\x0B");
$content = str_replace($newlines, '', $raw);

$start = strpos($content,'>Details<');
$end = strpos($content,'</table>',$start);
$table1 = substr($content,$start,$end-$start);
// $table1 = strip_tags($table1);

获得名字

$start = strpos($table1,'<td');
$end = strpos($table1,'<br />',$start);
$fnames = substr($table1,$start,$end-$start);
$fnames = strip_tags($fnames);
$fnames = preg_replace('/\s\s+/', '', $fnames);

获得姓氏

$start = strpos($table1,'<br />');
$end = strpos($table1,'</td>',$start);
$lnames = substr($table1,$start,$end-$start);
$lnames = strip_tags($lnames);
$lnames = preg_replace('/\s\s+/', '', $lnames);

获取电话

$start = strpos($table1,'Phone:');
$end = strpos($table1,'</td>              </tr>              <tr>',$start);
$phone = substr($table1,$start,$end-$start);
$phone = strip_tags($phone);
$phone = str_replace("Phone:", "" ,$phone);
$phone = preg_replace('/\s\s+/', '', $phone);

获取地址

$start = strpos($table1,'Address:');
$end = strpos($table1,'</td>              </tr>              <tr>',$start);
$ad = substr($table1,$start,$end-$start);
$ad = strip_tags($ad);
$ad = str_replace("Address:", "" ,$ad);
$ad = preg_replace('/\s\s+/', '', $ad);

让公寓没有

$start = strpos($table1,'Apt:');
$end = strpos($table1,'</td>              </tr>              <tr>',$start);
$apt = substr($table1,$start,$end-$start);
$apt = strip_tags($apt);
$apt = str_replace("Apt:", "" ,$apt);
$apt = preg_replace('/\s\s+/', '', $apt);

获得国家

$start = strpos($table1,'Country:');
$end = strpos($table1,'</td>              </tr>              <tr>',$start);
$country = substr($table1,$start,$end-$start);
$country = strip_tags($country);
$country = str_replace("Country:", "" ,$country);
$country = preg_replace('/\s\s+/', '', $country);

获得城市

$start = strpos($table1,'City:<br />                 State/Province:');
$end = strpos($table1,'</td>              </tr>              <tr>',$start);
$city = substr($table1,$start,$end-$start);
$city = strip_tags($city);
$city = str_replace("City:                 State/Province:", "" ,$city);
$city = preg_replace('/\s\s+/', '', $city);

获取zip

$start = strpos($table1,'Zip:');
$end = strpos($table1,'</td>              </tr>              <tr>',$start);
$zip = substr($table1,$start,$end-$start);
$zip = strip_tags($zip);
$zip = str_replace("Zip:", "" ,$zip);
$zip = preg_replace('/\s\s+/', '', $zip);

收到电子邮件

$start = strpos($table1,'email:');
$end = strpos($table1,'</td>              </tr>',$start);
$email = substr($table1,$start,$end-$start);
$email = strip_tags($email);
$email = str_replace("email:", "" ,$email);
$email = preg_replace('/\s\s+/', '', $email);

回显

行

echo "<tr>
<td><a href='http://www.example.com/info.php?num=$pagenumber'>link</a></td>
<td>$fnames</td>
<td>$lnames</td>
<td>$phone</td>
<td>$ad</td>
<td>$apt</td>
<td>$country</td>
<td>$city</td>
<td>$zip</td>
<td>$email</td>
</tr>";

包括数据库信息

include("inf.php");
$tablename = 'list';

$fnames = mysql_real_escape_string($fnames);
$lnames = mysql_real_escape_string($lnames);
$phone = mysql_real_escape_string($phone);
$ad = mysql_real_escape_string($ad);
$apt = mysql_real_escape_string($apt);
$country = mysql_real_escape_string($country);
$city = mysql_real_escape_string($city);
$zip = mysql_real_escape_string($zip);
$email = mysql_real_escape_string($email);

将行插入db

$query = "INSERT INTO $tablename VALUES('', '$pagenumber', '$fnames', '$lnames', '$phone', '$ad', 

'$apt','$country','$city','$zip', '$email')";
mysql_query($query) or die(mysql_error());

重置循环

$pagenumber = $pagenumber + 1;
}

?>

Answer 1

不要将hge使用正则表达式。你应该使用xpath，特别是php，DOMXPath

Answer 2

你可以看一下curl

http://nl2.php.net/manual/en/book.curl.php

在抓取页面后，您可以使用单个模式来获取所有必填字段。匹配可以使用preg_match_all

完成

您所寻找的数据是否还没有可用的xml / rss源？看看您是否可以在示例站点上每页显示更多结果，这将减少您需要抓取的页面数。

编辑：请求一个简单的例子：

确保您的服务器已启用curl：

echo 'cURL is '.(function_exists('curl_init') ?: ' not').' enabled';

          $ch = curl_init();

    curl_setopt ($ch, CURLOPT_URL, 'http://example.com' );

    curl_setopt($ch, CURLOPT_REFERER, 'http://example.com');
    curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate');
    curl_setopt($ch, CURLOPT_AUTOREFERER, true);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
    curl_setopt($ch, CURLOPT_TIMEOUT, 5);
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);

           $page =curl_exec ($ch);

简化代码以加速php scraper

2 个答案: