Question

由于抓狂，我试图刮掉lacentrale.fr的网站，但是即使我轮换了用户代理和IP地址（感谢TOR），该网站也会检测到我的机器人并向我发送错误的值。请您检查我在中间件和设置中使用的代码，并告诉我是否出了问题。

中间件中的代码：

from tutorial.settings import * #USER_AGENT_LIST
import random

from stem.control import Controller
from toripchanger import TorIpChanger
from stem import Signal

class RandomUserAgentMiddleware(object):

def process_request(self, request, spider):
    ua = random.choice(USER_AGENT_LIST)
    if ua:
        request.headers.setdefault('User-Agent', ua)


def _set_new_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password='')
        controller.signal(Signal.NEWNYM)

ip_changer = TorIpChanger(reuse_threshold=10)
class ProxyMiddleware(object):
    _requests_count = 0

def process_request(self, request, spider):
    self._requests_count += 1
    if self._requests_count > 10:
        self._requests_count = 0
        ip_changer.get_new_ip()

        print("New Tor connection processed")

    request.meta['proxy'] = 'http://127.0.0.1:8118'
    spider.log('Proxy : %s' % request.meta['proxy'])

设置中使用的代码：

BOT_NAME = 'tutorial'

SPIDER_MODULES = ['tutorial.spiders']
NEWSPIDER_MODULE = 'tutorial.spiders'


ROBOTSTXT_OBEY = True


DOWNLOAD_DELAY = 1
RANDOMIZE_DOWNLOAD_DELAY = True
DOWNLOADER_MIDDLEWARES = {
    'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'tutorial.middlewares.RandomUserAgentMiddleware': 400,
    'scrapy.contrib.downloadermiddleware.httpcompression.HttpCompressionMiddleware':None, # to avoid the raise IOError, 'Not a gzipped file' exceptions.IOError: Not a gzipped file
    'scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware': 110,
    'tutorial.middlewares.ProxyMiddleware': 100
    }



USER_AGENT_LIST=[

    {'User-agent': 'Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/28.0.1469.0 Safari/537.36'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML like Gecko) Chrome/28.0.1469.0 Safari/537.36'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2049.0 Safari/537.36'},
    {
        'User-agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36'},
    {'User-agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:35.0) Gecko/20100101 Firefox/35.0'},
    {'User-agent': 'Mozilla/5.0 (Windows NT 6.3; rv:36.0) Gecko/20100101 Firefox/36.0'}
    ]

EDIT II

似乎tor每次都使用相同的ip，并且ip地址上没有循环。我不知道我可以在我的Middlwares文件中进行哪些更改来解决此问题！请知道吗？

Answer 1

可能会在多种因素下检测到您，包括您的抓取工具是否下载/运行javascript文件。如果是这样，您可能需要结合使用硒和Python / Scrapy之类的工具，以进一步伪装成普通的人类用户。

此stackoverflow帖子为入门提供了一些帮助：

https://stackoverflow.com/a/17979285/9693088

对于您的TOR设置可能出现的问题，我认为我无法提供太多指导

避免在报废期间被发现

1 个答案: