How to use proxies with Go net/http

Which proxy works best with Go net/http?

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

• Residentialfrom $3.52 per GB
• Residential Litefrom $0.96 per GB
• Unlimited Datacenterfrom $0.56 per hour

Proxy settings for Go net/http: host, port and login

A BooProxies plan gives you four things: a host, a port, a username and a password. These are the values for Residential; the username and password are on your plan page.

As one address:

http://USERNAME:PASSWORD@resi.proxies.boo:8080

The country and the session are chosen in the username, not in a separate setting. The generator on your plan page writes the username for you, and Targeting and sessions explains each part.

Go net/http proxy setup, step by step

1. Parse the proxy address with url.Parse.
2. Set Proxy: http.ProxyURL(...) on the transport and use that client.

Go net/http proxy example

Go

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
)

func main() {
	proxy, _ := url.Parse("http://USERNAME:PASSWORD@resi.proxies.boo:8080")
	client := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxy)}}
	res, err := client.Get("https://api.ipify.org")
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()
	body, _ := io.ReadAll(res.Body)
	fmt.Println(string(body))
}

Replace USERNAME and PASSWORD with the login from your plan page. The example asks an IP-check address, so the output is the address your request left from.

Rotating or sticky sessions for Go net/http?

For crawling many pages with Go net/http, leave the username without a session so every request leaves from a new IP address. For anything with steps that belong together, such as a login, a cart or a paginated search that sets cookies, add a session to the username so those requests share one address.

A sticky username looks like this:

USERNAME-country-us-time-600-session-a1b2c3d4

Go net/http proxy not working?

• The transport keeps connections alive, so repeated requests can share one IP address. Set DisableKeepAlives: true when you want a new address for every request.
• A socks5:// address works in the same place, with port 1080.
• A 407 error means the username or password was not accepted. Copy both again from your plan page, and check that nothing added a space or cut the username short.
• If the IP address is in the wrong country, look at the username: the country code is part of it. More fixes.

Questions about Go net/http and proxies

Terms explained: web scraping, rotating proxy, sticky session, rate limiting, headless browser.

Go net/http is a product of its own makers. BooProxies is not affiliated with it; the name is used here only to say what this guide is about. Use it within the rules of the sites you work with and our acceptable use policy.