Rotating Proxies for Scrapy
Point your Scrapy spiders at one Proxy Rotator gateway and get a fresh IP on every request from a 100M+ pool of residential, datacenter, mobile and IPv6 proxies. No proxy-list package, no rotation middleware to build. Set one proxy line and scrape.
Plans from $24.95/mo New IP per request HTTPS & SOCKS5
Why Proxy Rotator for Scrapy
The hard parts of proxy management are handled at the gateway, so your spider code stays simple.
New IP per request
The gateway hands every Scrapy request a different exit IP. No proxy lists, no dead-IP retries, no custom rotation middleware.
Rotating or sticky
Use rotating IPs for high-volume crawls, or sticky sessions when a flow needs the same IP across several requests.
All proxy types
Residential, datacenter, mobile and IPv6 in one pool. Switch types from your dashboard without touching the spider.
HTTPS & SOCKS5
Standard protocols work with Scrapy out of the box and with the SOCKS handling in Twisted when you need it.
City & country targeting
Target by country, and by city on residential, to crawl localized pages, SERPs and pricing.
Built to scale
Thread-based plans handle Scrapy's concurrent requests, from small crawls to heavy production pipelines.
Prerequisites
- An active Proxy Rotator plan (from $24.95/mo).
- Your gateway host and port:
gateway.proxyrotator.com:8080. - Your gateway username and password, or a whitelisted server IP, from your dashboard. See the rotating proxy API for endpoint details.
- Scrapy installed:
pip install scrapy.
Throughout this guide, replace USER:PASS with your real gateway credentials.
Add Proxy Rotator to Scrapy
Our gateway already rotates, so you only need one proxy line. No scrapy-rotating-proxies package required.
Set the proxy in your spider (user:pass)
The simplest method is to attach the proxy on each request in start_requests. Scrapy's built-in HttpProxyMiddleware reads request.meta['proxy'] automatically.
import scrapy
PROXY = "http://USER:PASS@gateway.proxyrotator.com:8080"
class IpSpider(scrapy.Spider):
name = "ipcheck"
def start_requests(self):
urls = ["https://api.ipify.org"] * 5
for url in urls:
# gateway rotates, so each request exits a new IP
yield scrapy.Request(
url,
meta={"proxy": PROXY},
callback=self.parse,
dont_filter=True,
)
def parse(self, response):
self.logger.info("Exit IP: %s", response.text)
Run it with scrapy runspider spider.py. Each of the five requests prints a different IP.
Or apply the proxy to every request via middleware
To avoid repeating the proxy on each request, set it once in a tiny downloader middleware. Drop this in middlewares.py and enable it in settings.py.
class ProxyRotatorMiddleware:
PROXY = "http://USER:PASS@gateway.proxyrotator.com:8080"
def process_request(self, request, spider):
request.meta["proxy"] = self.PROXY
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.ProxyRotatorMiddleware": 350,
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}
IP whitelist auth (fallback)
Prefer not to embed credentials? Whitelist your server IP in the dashboard, then drop the USER:PASS@ part. No username or password is needed.
PROXY = "http://gateway.proxyrotator.com:8080" # in start_requests or your middleware request.meta["proxy"] = PROXY
Verify it works
Before running a full crawl, confirm the gateway is reachable and rotating from the command line.
curl -x http://USER:PASS@gateway.proxyrotator.com:8080 https://api.ipify.org # run it again, you get a different IP
A different IP on each run confirms your credentials work and the gateway is rotating.
Rotating vs sticky for Scrapy
Use rotating proxies for most crawls: every Scrapy request exits a fresh IP, which is ideal for high-volume data collection where you do not need to keep a session. Choose sticky proxies when a target requires the same IP across several requests, such as a login followed by paginated pages. You can switch between them from your dashboard without changing your spider.
Scrapy proxy FAQ
Do I need scrapy-rotating-proxies or a proxy-list package?
Which plan do I need for Scrapy?
Do you support SOCKS5 with Scrapy?
How do I authenticate without putting credentials in code?
Can I target a specific country or city?
Will rotating proxies fix Scrapy 403 Forbidden errors?
Do Scrapy's concurrent requests each get a different IP?
Do free or shared proxies work with Scrapy?
Run your Scrapy spiders on rotating proxies
Create an account, grab your gateway credentials, and get a new IP per request from a 100M+ pool. Plans from $24.95/mo.