Websites read it to decide which version of a page to send, and bot filters read it to see whether the visitor claims to be a browser. A library's default, such as the one Python sends, announces a script immediately.
Setting a realistic user agent is a basic step in any scraper, but it has to match everything else. A request that says Chrome and behaves nothing like Chrome is easier to spot than one that is honest about being a script. A proxy does not change the user agent.
A typical browser user agent names the browser, its version and the operating system. Sites compare it with what the browser reveals in other ways, so a string that claims an iPhone while the screen size says desktop does more harm than good.
Questions about user agent
Should I rotate user agents?
Only together with everything else that should match. A changing user agent on an unchanged browser fingerprint looks stranger than one honest, current browser string.
Does a proxy change my user agent?
No. The proxy passes your request on as it is. The user agent is set by your browser or by your code.
Related terms
- Browser fingerprint
- A browser fingerprint is the mix of details a browser reveals, such as screen size, fonts and graphics card, that together single it out.
- Bot detection
- Bot detection is the set of checks a website runs to tell people from programs: the IP address, the browser fingerprint and the behaviour.
- Headless browser
- A headless browser is a real browser that runs without a window and is driven by a program, used to load pages that need JavaScript.
Read next: Web scraping without getting blocked