A scraper that waits for each page before asking for the next is slow. One that keeps fifty requests open at once is fifty times faster, as long as the target and the proxy both keep up. Some proxy services cap this number, sometimes calling it threads.
With a rotating proxy, each of those connections gets its own address. If the parts of the job must stay separate visitors for longer, give each worker its own session ID. Raise the number slowly: too many at once against one site is what triggers rate limiting.
Questions about concurrent connections
How many connections can I open at once?
Gigabyte plans on BooProxies are billed by traffic, not per connection. In practice the limit is your target: raise the number until it starts answering with errors or captchas, then step back.
Do more connections use more traffic?
No. The same pages weigh the same however many you fetch at once. More connections finish the job sooner; they do not make it bigger.
Related terms
- Session ID
- A session ID is a short label you add to a proxy username. Requests that carry the same label leave from the same IP address.
- Rate limiting
- Rate limiting is a cap on how many requests a website accepts from one visitor in a given time. Going over it brings errors or a block.
- Proxy latency
- Proxy latency is the extra time a request takes because it travels through a proxy before it reaches the website and comes back.