CONCURRENT_REQUESTS_PER_IP #6562

It is not meant for your use case. It should not affect proxy IPs in any way.

CONCURRENT_REQUESTS_PER_IP is a barely used and barely supported alternative to CONCURRENT_REQUESTS_PER_DOMAIN.

It is meant for a scenario where you might target 2 or more domains that actually have the same IP address, and you don’t want to send too many requests to them.

Let’s say you have a spider targeting these domains @ IPs:

a.example @ 203.0.113.1
b.example @ 203.0.113.1
c.example @ 203.0.113.2

a.example and b.example are different domains but point to the same IP address, which usually means the same server (well, it always means the same server, but it could be a lightweight reverse proxy for multiple servers or for a single server, you don’t really know that). You do not want to hit that server too hard, so you want to set the request concurrency limit per IP, not per domain. So CONCURRENT_REQUESTS_PER_IP=5 means that 203.0.113.1 (a.example and b.example) will only get 5 parallel requests at most at any given time between the two, while CONCURRENT_REQUESTS_PER_DOMAIN=5 would allow 10 parallel request to 203.0.113.1 (5 to a.example, 5 to b.example).

In practice, however, this scenario where different domains you are targeting point to the same IP address is uncommon. It is in fact more common to see a domain that points to multiple IP addresses instead. So there is rarely a use case for this setting.

Moreover, some features of Scrapy are not compatible with the use of CONCURRENT_REQUESTS_PER_IP. For example, the scrapy.pqueues.DownloaderAwarePriorityQueue value of the SCHEDULER_PRIORITY_QUEUE setting, which is ideal for broad crawls (crawls where you target many different domains), does not support CONCURRENT_REQUESTS_PER_IP.

So I would personally discourage the use of this setting in general.