How to use proxies with curl_cffi

Which proxy works best with curl_cffi?

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

• Residentialfrom $3.52 per GB
• Residential Litefrom $0.96 per GB
• Unlimited Datacenterfrom $0.56 per hour

Proxy settings for curl_cffi: host, port and login

A BooProxies plan gives you four things: a host, a port, a username and a password. These are the values for Residential; the username and password are on your plan page.

As one address:

http://USERNAME:PASSWORD@resi.proxies.boo:8080

The country and the session are chosen in the username, not in a separate setting. The generator on your plan page writes the username for you, and Targeting and sessions explains each part.

curl_cffi proxy setup, step by step

1. Pass the proxy for both http and https.
2. Choose a browser to impersonate, so the connection looks like that browser as well as coming from a home IP address.

curl_cffi proxy example

Python

from curl_cffi import requests

proxy = "http://USERNAME:PASSWORD@resi.proxies.boo:8080"
r = requests.get("https://api.ipify.org", impersonate="chrome", proxies={"http": proxy, "https": proxy})
print(r.text)

Replace USERNAME and PASSWORD with the login from your plan page. The example asks an IP-check address, so the output is the address your request left from.

Rotating or sticky sessions for curl_cffi?

For crawling many pages with curl_cffi, leave the username without a session so every request leaves from a new IP address. For anything with steps that belong together, such as a login, a cart or a paginated search that sets cookies, add a session to the username so those requests share one address.

A sticky username looks like this:

USERNAME-country-us-time-600-session-a1b2c3d4

curl_cffi proxy not working?

• A browser-like fingerprint on a data centre IP address is still a data centre IP address. Sites that block by address need a residential or mobile proxy.
• A 407 error means the username or password was not accepted. Copy both again from your plan page, and check that nothing added a space or cut the username short.
• If the IP address is in the wrong country, look at the username: the country code is part of it. More fixes.

Questions about curl_cffi and proxies

Terms explained: web scraping, rotating proxy, sticky session, rate limiting, headless browser.

curl_cffi is a product of its own makers. BooProxies is not affiliated with it; the name is used here only to say what this guide is about. Use it within the rules of the sites you work with and our acceptable use policy.