Skip to content

feat: add SetResolver and SetHosts for custom DNS (#169) - #516

Open
ManuelReschke wants to merge 1 commit into
imroc:masterfrom
ManuelReschke:feat_169
Open

feat: add SetResolver and SetHosts for custom DNS (#169)#516
ManuelReschke wants to merge 1 commit into
imroc:masterfrom
ManuelReschke:feat_169

Conversation

@ManuelReschke

Copy link
Copy Markdown
Contributor

Add SetResolver and SetHosts for custom DNS on HTTP/1 and HTTP/2

Fixes #169.
Closes #168 (discussion).

Summary

Adds a simple way to customize DNS resolution for HTTP/1 and HTTP/2, addressing long DNS timeouts when crawling with a known host list.

  • Client.SetResolver(*net.Resolver) — use a custom net.Resolver (e.g. a specific DNS server) via SetDial + net.Dialer.
  • Client.SetHosts(map[string]string) — static hostname→IP mapping (hosts-file style). Unlisted hostnames fail immediately with "no such host" and do not hit the system resolver.
  • Global wrappers: SetResolver, SetHosts.
  • HTTP/2 fix: when DialContext is set, the HTTP/2 transport uses it for the underlying TCP dial (including EnableForceHTTP2), with TLS handshake timeout/trace handling aligned to HTTP/1 addTLS.

Motivation

From discussion #168: crawlers that hit many domains pay for system DNS timeouts on dead hosts. A custom resolver or hosts map that can return NOHOST quickly covers the common case.

SetDial already allowed this, but a dedicated API is clearer and documents the intended pattern.

API

// Custom DNS server
r := &net.Resolver{
	PreferGo: true,
	Dial: func(ctx context.Context, network, address string) (net.Conn, error) {
		var d net.Dialer
		return d.DialContext(ctx, "udp", "1.1.1.1:53")
	},
}
client := req.C().SetResolver(r)

// Static hosts map (fail closed for unknown hostnames)
client := req.C().SetHosts(map[string]string{
	"api.internal": "10.0.0.5",
	"db.internal":  "10.0.0.6",
	"v6.internal":  "::1", // brackets like "[::1]" also accepted
})

Notes:

  • Only applies to HTTP/1 and HTTP/2 (same scope as SetDial). HTTP/3 is unchanged.
  • Host matching is case-insensitive and IDNA-normalized.
  • Map values must be literal IPs; non-IPs never fall through to system DNS (clear error instead). IPv6 brackets are normalized.
  • IP-literal request URLs (e.g. https://1.2.3.4/) skip the map and dial directly.
  • Empty/nil map → every non-literal hostname fails closed.
  • SetDialTLS still bypasses this dialer for HTTPS when set.
  • SetDial, SetResolver, SetHosts, and SetUnixSocket replace each other (last call wins).
  • The map is copied.

Tests

  • SetResolver: IP URL works with a broken custom resolver; hostname fails with that resolver’s error.
  • SetHosts: successful HTTP/1 and forced HTTP/2 via mapped hostname.
  • Unknown host / empty map fail fast with "no such host".
  • IP-literal passthrough, invalid IP (no DNS), IPv6 bracket normalization, IDNA key normalization.
  • Caller map mutation ignored; SetDial replaces hosts dialer.

Validation:

go test ./...
go vet ./...
git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support customize the dns resolver

1 participant