A site counts requests per address, per account or per session. Past the limit it answers with an error, most often 429 Too Many Requests, or quietly serves a captcha. The limit protects the site from being overloaded.
There are two honest ways to stay under it: slow down, or spread the same work across more addresses with a rotating proxy. Slowing down costs time; rotating costs traffic. For small sites, slowing down is also the considerate choice.
Limits are rarely published, and they differ between parts of the same site: a search page is usually guarded more tightly than a product page, and a login form most of all. A well-behaved scraper backs off when it sees a 429, waits longer after each one, and returns to full speed slowly.
Questions about rate limiting
What does a 429 error mean?
Too Many Requests: you went over the site's limit. Many sites add a Retry-After header that says how long to wait before trying again.
How many requests per second are safe?
There is no single number. Start slowly, watch for 429 errors and captchas, and raise the pace step by step. Small sites deserve a much gentler pace than large platforms.
Related terms
- IP rotation
- IP rotation means changing the IP address your requests come from, either on every connection or after a set time, to spread them out.
- IP ban
- An IP ban is when a website refuses requests from a particular IP address, usually after seeing too many requests or behaviour it disliked.
- Concurrent connections
- Concurrent connections are the requests a program has open through a proxy at the same moment. More of them finish a job sooner.
Read next: Proxies for web scraping