我正在抓取具有以下结构的网站:
<td colspan="1">
<font class="spy14">
91.240.61.217
<script type="text/javascript">document.write("<font class=spy2>:<\/font>"+(One2NineSix^Four3Eight)+(One2NineSix^Four3Eight)+(One2NineSix^Four3Eight)+(Zero9EightSeven^Nine6Six))</script>
</td>
当您在页面上看到它时,它看起来像这样:
91.240.61.217:6660
但是当我尝试抓取它时,我发现自己无法获得“:6660”部分,因为该部分是由上面的JS脚本生成的。
如何获得HTML的那一部分?这是我当前的代码:
val = Selector(response)
ip = val.xpath('//tr[@class="spy1xx"]/td[1]/font').extract_first()
JS代码看似荒谬,但页面中也包含以下脚本:
eval(function(p,r,o,x,y,s){y=function(c){return(c<r?'':y(parseInt(c/r)))+((c=c%r)>35?String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/,String)){while(o--){s[y(o)]=x[o]||y(o)}x=[function(y){return s[y]}];y=function(){return'\\w+'};o=1};while(o--){if(x[o]){p=p.replace(new RegExp('\\b'+y(o)+'\\b','g'),x[o])}}return p}('p=B^C;m=G^I;h=F^E;r=H^J;f=2;l=K^u;e=5;c=4;n=8;t=v^w;j=A^z;g=y^x;k=1;d=D^O;b=T^V;s=6;a=3;o=7;i=0;q=9;L=i^j;Q=k^g;P=f^b;M=a^m;N=c^d;R=e^l;S=s^r;X=o^p;W=n^t;U=q^h;',60,60,'^^^^^^^^^^Four^Seven4Seven^Five^Nine3Three^One^Zero^OneTwoZero^Seven9Four^Two^Nine6Six^Three^TwoOneNine^Eight5One^Seven^Eight^NineThreeTwo^Nine^Four3Eight^Six^Seven0Five^8090^5462^88^808^12035^3129^11286^10218^8909^6038^8888^7449^6689^3320^80^443^2546^Zero9EightSeven^Nine1SixNine^SixThreeFiveFive^8000^Nine7OneZero^Seven0FourOne^ZeroSixTwoEight^One2NineSix^9273^Six2ZeroFour^8118^Eight9SevenThree^FiveFourThreeTwo'.split('\u005e'),0,{}))
因此,它实际上确实打印出“:6660”。我测试过了。
有没有办法
a)通过document.write获得页面上显示的脚本输出
或
b)抓取JS代码并以某种方式通过Python在本地运行
答案 0 :(得分:1)
Guillaume答案的另一种方法是在蜘蛛内部使用js2py
执行JS代码:
>>> import js2py
>>> js = r"eval(function(p,r,o,x,y,s){y=function(c){return(c<r?'':y(parseInt(c/r)))+((c=c%r)>35?String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/,String)){while(o--){s[y(o)]=x[o]||y(o)}x=[function(y){return s[y]}];y=function(){return'\\w+'};o=1};while(o--){if(x[o]){p=p.replace(new RegExp('\\b'+y(o)+'\\b','g'),x[o])}}return p}('p=B^C;m=G^I;h=F^E;r=H^J;f=2;l=K^u;e=5;c=4;n=8;t=v^w;j=A^z;g=y^x;k=1;d=D^O;b=T^V;s=6;a=3;o=7;i=0;q=9;L=i^j;Q=k^g;P=f^b;M=a^m;N=c^d;R=e^l;S=s^r;X=o^p;W=n^t;U=q^h;',60,60,'^^^^^^^^^^Four^Seven4Seven^Five^Nine3Three^One^Zero^OneTwoZero^Seven9Four^Two^Nine6Six^Three^TwoOneNine^Eight5One^Seven^Eight^NineThreeTwo^Nine^Four3Eight^Six^Seven0Five^8090^5462^88^808^12035^3129^11286^10218^8909^6038^8888^7449^6689^3320^80^443^2546^Zero9EightSeven^Nine1SixNine^SixThreeFiveFive^8000^Nine7OneZero^Seven0FourOne^ZeroSixTwoEight^One2NineSix^9273^Six2ZeroFour^8118^Eight9SevenThree^FiveFourThreeTwo'.split('\u005e'),0,{}))"
>>> js2py.eval_js(js)
16296
Js2Py:https://github.com/PiotrDabkowski/Js2Py
关于从页面中提取JS代码,恐怕您需要分享有关页面本身的更多详细信息,以便我们帮助您弄清楚该怎么做。
答案 1 :(得分:0)
您可以安装诸如NodeJS之类的Javascript解释器,然后从Python调用它。
java.lang.AssertionError: Verification failed: call 1 of 1: class com.example.FooBarTestKt.bar(eq(Foo(Hello, Spy!#1)))). Only one matching call to FooBarTestKt(static FooBarTestKt)/bar(Foo) happened, but arguments are not matching:
[0]: argument: com.example.Foo@476b0ae6, matcher: eq(Foo(Hello, Spy!#1)), result: -
此打印:
from subprocess import Popen, PIPE
js = r"eval(function(p,r,o,x,y,s){y=function(c){return(c<r?'':y(parseInt(c/r)))+((c=c%r)>35?String.fromCharCode(c+29):c.toString(36))};if(!''.replace(/^/,String)){while(o--){s[y(o)]=x[o]||y(o)}x=[function(y){return s[y]}];y=function(){return'\\w+'};o=1};while(o--){if(x[o]){p=p.replace(new RegExp('\\b'+y(o)+'\\b','g'),x[o])}}return p}('p=B^C;m=G^I;h=F^E;r=H^J;f=2;l=K^u;e=5;c=4;n=8;t=v^w;j=A^z;g=y^x;k=1;d=D^O;b=T^V;s=6;a=3;o=7;i=0;q=9;L=i^j;Q=k^g;P=f^b;M=a^m;N=c^d;R=e^l;S=s^r;X=o^p;W=n^t;U=q^h;',60,60,'^^^^^^^^^^Four^Seven4Seven^Five^Nine3Three^One^Zero^OneTwoZero^Seven9Four^Two^Nine6Six^Three^TwoOneNine^Eight5One^Seven^Eight^NineThreeTwo^Nine^Four3Eight^Six^Seven0Five^8090^5462^88^808^12035^3129^11286^10218^8909^6038^8888^7449^6689^3320^80^443^2546^Zero9EightSeven^Nine1SixNine^SixThreeFiveFive^8000^Nine7OneZero^Seven0FourOne^ZeroSixTwoEight^One2NineSix^9273^Six2ZeroFour^8118^Eight9SevenThree^FiveFourThreeTwo'.split('\u005e'),0,{}))"
p = Popen(['node', '-p', js], stdout=PIPE)
output, _ = p.communicate()
print(output)