Question

我正在尝试使用Python中的Web刮刀来完成Aliexpress供应商的所有产品。我的问题是，当我没有记录它时，我最终被重定向到登录网页。我在我的代码中添加了登录部分，但它没有帮助。我将感谢所有的建议。

我的代码：

import requests
from bs4 import BeautifulSoup
import re
import sys
from lxml import html


def go_through_paginator(link):
    source_code = requests.get(link, data=payload,  headers = dict(referer = link))
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    print(soup)
    for page in soup.findAll ('div', {'class' : 'ui-pagination-navi util-left'}):
        for next_page in page.findAll ('a', {'class' : 'ui-pagination-next'}):
            next_page_link="https:" + next_page.get('href')
            print (next_page_link)
            gather_all_products (next_page_link)

def gather_all_products (url):
    source_code = requests.get(url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    for item in soup.findAll ('a', {'class' : 'pic-rind'}):
        product_link=item.get('href')
    go_through_paginator(url)


payload = {
    "loginId": "EMAIL", 
    "password": "LOGIN",
}

LOGIN_URL='https://login.aliexpress.com/buyer.htm?spm=2114.12010608.1000002.4.EihgQ5&return=https%3A%2F%2Fwww.aliexpress.com%2Fstore%2F1816376%3Fspm%3D2114.10010108.0.0.fs2frD&random=CAB39130D12E432D4F5D75ED04DC0A84'

session_requests = requests.session()
source_code = session_requests.get(LOGIN_URL)
source_code = session_requests.post(LOGIN_URL, data = payload)


URL='https://www.aliexpress.com/store/1816376?spm=2114.10010108.0.0.fs2frD'

source_code = requests.get(URL, data=payload,  headers = dict(referer = URL))
plain_text = source_code.text
soup = BeautifulSoup(plain_text)

for L1 in soup.findAll ('li', {'id' : 'product-nav'}):
    for L1_link in L1.findAll('a', {'class' : 'nav-link'}):
        link = "https:" + L1_link.get('href') 
        gather_all_products(link)

这是aliexpress登录网址： https://login.aliexpress.com/buyer.htm?spm=2114.12010608.1000002.4.EihgQ5&return=https%3A%2F%2Fwww.aliexpress.com%2Fstore%2F1816376%3Fspm%3D2114.10010108.0.0.fs2frD&random=CAB39130D12E432D4F5D75ED04DC0A84

Answer 1

尝试从响应Cookie中设置 xman_t 和 intl_common_forever 的Cookie值。

我是直接尝试抓取所有产品信息。在我设置xman_t和intl_common_forever之前，Aliexpress只允许我抓住7个产品。在设置xman_t和intl_common_forever之后，我成功获得了50个产品。

希望这可以帮助你刮擦他们的产品。

使用Python刮擦Aliexpress - 需要登录

1 个答案: