Scrapy with ProxyPanel

Scrapy with ProxyPanel

Scrapy is a powerful web scraping framework for Python.

Setting Up the Proxy

Add to your settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 1,
}

HTTP_PROXY = "http://api-key:api-key@proxy-host:http-port"

In Your Spider

import scrapy

class MySpider(scrapy.Spider):
    name = "myspider"

    def start_requests(self):
        yield scrapy.Request(
            url="https://httpbin.org/ip",
            meta={"proxy": "http://api-key:api-key@proxy-host:http-port"}
        )

    def parse(self, response):
        self.logger.info(response.text)

Using SOCKS5

For SOCKS5 support in Scrapy, install scrapy-proxy-pool or use a custom middleware with PySocks.