How to Use Proxies with Python Requests

Which proxy works best with Python Requests?

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

Proxy settings for Python Requests: host, port and login

A BooProxies plan gives you four things: a host, a port, a username and a password. These are the values for Residential; the username and password are on your plan page.

Host	`resi.proxies.boo`
HTTP port	`8080`
SOCKS5 port	`1080`
Username	`USERNAME`
Password	from your plan page

As one address:

http://USERNAME:PASSWORD@resi.proxies.boo:8080

The country and the session are chosen in the username, not in a separate setting. The generator on your plan page writes the username for you, and Targeting and sessions explains each part.

Python Requests proxy setup, step by step

Build the proxy address with your username and password in it.
Pass it for both http and https. The key names the kind of site you are visiting, not the kind of proxy.
Set a timeout, so a slow exit does not hang the script.

Python Requests proxy example

Python

import requests

proxy = "http://USERNAME:PASSWORD@resi.proxies.boo:8080"
r = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, timeout=30)
print(r.text)

Replace USERNAME and PASSWORD with the login from your plan page. The example asks an IP-check address, so the output is the address your request left from.

Rotating or sticky sessions for Python Requests?

For crawling many pages with Python Requests, leave the username without a session so every request leaves from a new IP address. For anything with steps that belong together, such as a login, a cart or a paginated search that sets cookies, add a session to the username so those requests share one address.

A sticky username looks like this:

USERNAME-country-us-time-600-session-a1b2c3d4

Python Requests proxy not working?

Passing only the http key leaves https sites going out directly, from your own IP address. Always set both.
For SOCKS5, install requests[socks] and use socks5h:// with port 1080. The h makes the proxy look up the site's name.
A requests.Session reuses connections. With a rotating username, the IP address then changes less often than you expect; close the session or send Connection: close to force a new one.
A 407 error means the username or password was not accepted. Copy both again from your plan page, and check that nothing added a space or cut the username short.
If the IP address is in the wrong country, look at the username: the country code is part of it. More fixes.

Questions about Python Requests and proxies

Does Python Requests work with SOCKS5 proxies?

Yes. Use port 1080 and the same username and password.

Which BooProxies proxy should I use with Python Requests?

Terms explained: web scraping, rotating proxy, sticky session, rate limiting, headless browser.

Python Requests is a product of its own makers. BooProxies is not affiliated with it; the name is used here only to say what this guide is about. Use it within the rules of the sites you work with and our acceptable use policy.