scrapy + selenium 爬取js生成的网页元素

最新推荐文章于 2025-05-15 18:05:55 发布

yun1221

最新推荐文章于 2025-05-15 18:05:55 发布

阅读量4.3k

点赞数

CC 4.0 BY-SA版权

文章标签： selenium python js crawl

本文链接：https://blog.csdn.net/lianliankan2/article/details/39085795

本文介绍了如何结合Scrapy和Selenium爬取JavaScript动态生成的网页内容。首先，安装Python 2.7，接着安装Scrapy，解决安装中可能出现的扩展问题。然后安装Selenium，下载并运行Selenium服务器。创建Scrapy工程，并参考相关教程实现动态网页抓取。在过程中可能会遇到如Socket错误和FirefoxProfile异常，可以参考相关StackOverflow解答和GitHub示例代码进行解决。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、安装python (我用的是2.7版本的)

2、安装scrapy: 详情请参考 http://blog.csdn.net/wukaibo1986/article/details/8167590 (提示，能下载源码安装的就避免用pip install **)

安装过程中遇到python扩展问题”unable to find vcvarsall.bat“的解决办法： http://blog.csdn.net/ren911/article/details/6448696

3、安装selenium, https://pypi.python.org/pypi/selenium，注意安装后Download the server separately, from: http://selenium-release.storage.googleapis.com/2.42/selenium-server-standalone-2.42.0.jar，然后运行(命令行运行，需要安装Java Runtime Environment (JRE)) java -jar selenium-server-standalone-2.42.0.jar

4、建scrapy工程，参考 http://www.searchtb.com/2011/01/an-introduction-to-crawler.html?spm=0.0.0.0.HrlxVl

5、scrapy和selenium结合抓取动态网页： https://github.com/tmslav/google_adwords_scrape/blob/master/adwords_bot.py

7、Python Selenium socket error - [Errno 61] Connection refused： http://stackoverflow.com/questions/12913991/python-selenium-socket-error-errno-61-connection-refused

8、Selenium: FirefoxProfile exception Can't load the profile： https://github.com/tmslav/google_adwords_scrape/blob/master/adwords_bot.py