出现此错误是因为ScrapyPlaywrightDownloadHandler没有正确安装或配置。建议按照以下步骤重新安装ScrapyPlaywrightDownloadHandler:
卸载ScrapyPlaywrightDownloadHandler: pip uninstall scrapy-playwright
安装Playwright: npm i playwright
安装ScrapyPlaywrightDownloadHandler: pip install scrapy-playwright
在settings.py文件中添加ScrapyPlaywrightDownloadHandler: DOWNLOADER_MIDDLEWARES = {'scrapy_playwright.PlaywrightMiddleware': 725}
在项目根目录下运行以下命令以启动Scrapy:
scrapy crawl
示例代码:
DOWNLOADER_MIDDLEWARES = {'scrapy_playwright.PlaywrightMiddleware': 725}
class MySpider(scrapy.Spider): custom_settings = {'PLAYWRIGHT_LAUNCH_OPTIONS': {'headless': False}}
def start_requests(self): yield scrapy.Request(url='http://example.com', callback=self.parse)
def parse(self, response): print(response.body)