A scraper requests pages the way a browser does, picks out the parts it needs and saves them. Businesses use it to follow competitors' prices, track their search rankings, gather listings and check where their adverts appear.
Sites limit how many pages one address may load, so larger jobs spread their requests over many addresses with a rotating proxy. Collect public pages only, follow the law where you and the site are, and keep the pace low enough that the site stays fast for its real visitors.
Related plans on BooProxies
Questions about web scraping
Is web scraping legal?
It depends on what you collect, how, and where you are. Public facts are treated differently from personal data, copyrighted work or pages behind a login. Check the law that applies to you and the site's terms before you start.
How much proxy traffic does scraping use?
A plain request for a page is tens or hundreds of kilobytes; a full load in a browser is megabytes. Blocking images and media in a headless browser is the single biggest saving.
Related terms
- Rotating proxy
- A rotating proxy gives each new connection a different IP address from a pool, so no single address makes enough requests to be noticed.
- Rate limiting
- Rate limiting is a cap on how many requests a website accepts from one visitor in a given time. Going over it brings errors or a block.
- Headless browser
- A headless browser is a real browser that runs without a window and is driven by a program, used to load pages that need JavaScript.
- Bot detection
- Bot detection is the set of checks a website runs to tell people from programs: the IP address, the browser fingerprint and the behaviour.
Read next: Proxies for web scraping