Question

我正在抓取具有以下结构的网站：

<td colspan="1">
    <font class="spy14">
        91.240.61.217
        <script type="text/javascript">document.write("<font class=spy2>:<\/font>"+(One2NineSix^Four3Eight)+(One2NineSix^Four3Eight)+(One2NineSix^Four3Eight)+(Zero9EightSeven^Nine6Six))</script>
</td>

当您在页面上看到它时，它看起来像这样：

91.240.61.217:6660

但是当我尝试抓取它时，我发现自己无法获得“：6660”部分，因为该部分是由上面的JS脚本生成的。

如何获得HTML的那一部分？这是我当前的代码：

val = Selector(response)
ip = val.xpath('//tr[@class="spy1xx"]/td[1]/font').extract_first()

JS代码看似荒谬，但页面中也包含以下脚本：

eval(function(p,r,o,x,y,s){y=function(c){return(c<r?'':y(parseInt(c/r)))+((c=c%r)>35?String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/,String)){while(o--){s[y(o)]=x[o]||y(o)}x=[function(y){return s[y]}];y=function(){return'\\w+'};o=1};while(o--){if(x[o]){p=p.replace(new RegExp('\\b'+y(o)+'\\b','g'),x[o])}}return p}('p=B^C;m=G^I;h=F^E;r=H^J;f=2;l=K^u;e=5;c=4;n=8;t=v^w;j=A^z;g=y^x;k=1;d=D^O;b=T^V;s=6;a=3;o=7;i=0;q=9;L=i^j;Q=k^g;P=f^b;M=a^m;N=c^d;R=e^l;S=s^r;X=o^p;W=n^t;U=q^h;',60,60,'^^^^^^^^^^Four^Seven4Seven^Five^Nine3Three^One^Zero^OneTwoZero^Seven9Four^Two^Nine6Six^Three^TwoOneNine^Eight5One^Seven^Eight^NineThreeTwo^Nine^Four3Eight^Six^Seven0Five^8090^5462^88^808^12035^3129^11286^10218^8909^6038^8888^7449^6689^3320^80^443^2546^Zero9EightSeven^Nine1SixNine^SixThreeFiveFive^8000^Nine7OneZero^Seven0FourOne^ZeroSixTwoEight^One2NineSix^9273^Six2ZeroFour^8118^Eight9SevenThree^FiveFourThreeTwo'.split('\u005e'),0,{}))

因此，它实际上确实打印出“：6660”。我测试过了。

有没有办法

a）通过document.write获得页面上显示的脚本输出

或

b）抓取JS代码并以某种方式通过Python在本地运行

Answer 1

Guillaume答案的另一种方法是在蜘蛛内部使用js2py执行JS代码：

>>> import js2py
>>> js = r"eval(function(p,r,o,x,y,s){y=function(c){return(c<r?'':y(parseInt(c/r)))+((c=c%r)>35?String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/,String)){while(o--){s[y(o)]=x[o]||y(o)}x=[function(y){return s[y]}];y=function(){return'\\w+'};o=1};while(o--){if(x[o]){p=p.replace(new RegExp('\\b'+y(o)+'\\b','g'),x[o])}}return p}('p=B^C;m=G^I;h=F^E;r=H^J;f=2;l=K^u;e=5;c=4;n=8;t=v^w;j=A^z;g=y^x;k=1;d=D^O;b=T^V;s=6;a=3;o=7;i=0;q=9;L=i^j;Q=k^g;P=f^b;M=a^m;N=c^d;R=e^l;S=s^r;X=o^p;W=n^t;U=q^h;',60,60,'^^^^^^^^^^Four^Seven4Seven^Five^Nine3Three^One^Zero^OneTwoZero^Seven9Four^Two^Nine6Six^Three^TwoOneNine^Eight5One^Seven^Eight^NineThreeTwo^Nine^Four3Eight^Six^Seven0Five^8090^5462^88^808^12035^3129^11286^10218^8909^6038^8888^7449^6689^3320^80^443^2546^Zero9EightSeven^Nine1SixNine^SixThreeFiveFive^8000^Nine7OneZero^Seven0FourOne^ZeroSixTwoEight^One2NineSix^9273^Six2ZeroFour^8118^Eight9SevenThree^FiveFourThreeTwo'.split('\u005e'),0,{}))"
>>> js2py.eval_js(js)
16296

Js2Py：https://github.com/PiotrDabkowski/Js2Py

关于从页面中提取JS代码，恐怕您需要分享有关页面本身的更多详细信息，以便我们帮助您弄清楚该怎么做。

Answer 2

您可以安装诸如NodeJS之类的Javascript解释器，然后从Python调用它。

java.lang.AssertionError: Verification failed: call 1 of 1: class com.example.FooBarTestKt.bar(eq(Foo(Hello, Spy!#1)))). Only one matching call to FooBarTestKt(static FooBarTestKt)/bar(Foo) happened, but arguments are not matching:
[0]: argument: com.example.Foo@476b0ae6, matcher: eq(Foo(Hello, Spy!#1)), result: -

此打印：

from subprocess import Popen, PIPE

js = r"eval(function(p,r,o,x,y,s){y=function(c){return(c<r?'':y(parseInt(c/r)))+((c=c%r)>35?String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/,String)){while(o--){s[y(o)]=x[o]||y(o)}x=[function(y){return s[y]}];y=function(){return'\\w+'};o=1};while(o--){if(x[o]){p=p.replace(new RegExp('\\b'+y(o)+'\\b','g'),x[o])}}return p}('p=B^C;m=G^I;h=F^E;r=H^J;f=2;l=K^u;e=5;c=4;n=8;t=v^w;j=A^z;g=y^x;k=1;d=D^O;b=T^V;s=6;a=3;o=7;i=0;q=9;L=i^j;Q=k^g;P=f^b;M=a^m;N=c^d;R=e^l;S=s^r;X=o^p;W=n^t;U=q^h;',60,60,'^^^^^^^^^^Four^Seven4Seven^Five^Nine3Three^One^Zero^OneTwoZero^Seven9Four^Two^Nine6Six^Three^TwoOneNine^Eight5One^Seven^Eight^NineThreeTwo^Nine^Four3Eight^Six^Seven0Five^8090^5462^88^808^12035^3129^11286^10218^8909^6038^8888^7449^6689^3320^80^443^2546^Zero9EightSeven^Nine1SixNine^SixThreeFiveFive^8000^Nine7OneZero^Seven0FourOne^ZeroSixTwoEight^One2NineSix^9273^Six2ZeroFour^8118^Eight9SevenThree^FiveFourThreeTwo'.split('\u005e'),0,{}))"

p = Popen(['node', '-p', js], stdout=PIPE)
output, _ = p.communicate()
print(output)

获取document.write的输出scrapy

2 个答案: