net: go cannot resolve localhost on darwin when net.DefaultResolver.PreferGo = true

[cee-dub]

What did you do?

# main.go

package main

import (
	"fmt"
	"io/ioutil"
	"net"
	"runtime"
)

func main() {
	fmt.Println("GOOS:", runtime.GOOS)
	p, err := ioutil.ReadFile("/etc/hosts")
	fmt.Println("# /etc/hosts:")
	fmt.Println(string(p))

	addrs, err := net.LookupHost("localhost")
	fmt.Println("net.LookupHost addrs:", addrs, "err:", err)

	net.DefaultResolver.PreferGo = true

	addrs, err = net.LookupHost("localhost")
	fmt.Println("net.LookupHost addrs:", addrs, "err:", err)
}

$ GODEBUG=netdns=2 go run main.go
GOOS: darwin
# /etc/hosts:
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost

go package net: using cgo DNS resolver
go package net: hostLookupOrder(localhost) = cgo
net.LookupHost addrs: [::1 127.0.0.1] err: <nil>
go package net: hostLookupOrder(localhost) = cgo
net.LookupHost addrs: [] err: lookup localhost on 8.8.8.8:53: no such host

$ GODEBUG=netdns=go+2 go run main.go
GOOS: darwin
# /etc/hosts:
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost

go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(localhost) = files,dns
net.LookupHost addrs: [127.0.0.1 ::1] err: <nil>
go package net: hostLookupOrder(localhost) = files,dns
net.LookupHost addrs: [127.0.0.1 ::1] err: <nil>

What did you expect to see?

Go is able to resolve localhost when net.DefaultResolver.PreferGo = true.

What did you see instead?

Go is unable to resolve localhost. In the debug output go package net: hostLookupOrder(localhost) = cgo shows that it is not looking at /etc/hosts.

System details

go version go1.10 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/cameron/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/cameron/src/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.10/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.10/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/r3/kh_tqb1j47ldzdbqfrtts6q40000gn/T/go-build873748070=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.10 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.10
uname -v: Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.13.3
BuildVersion:	17D102
lldb --version: lldb-900.0.64
  Swift-4.0

[philpennock]

With resolvers in /etc/resolv.conf on the local network, I can't reproduce. When I switch to Google's Public DNS service, I can reproduce. When I use OpenDNS, I can't reproduce. Nor Level 3's open recursor. Nor Norton DNS, nor censurfridns.

This appears to be specific to an interaction with the two Google Public DNS IPs.

[philpennock]

Oh of course: most recursors implement local data for localhost, to avoid hitting public roots. Google's Public DNS is almost unique in not doing so. Thus the problem manifests when the DNS servers don't have an entry for localhost, and so on Darwin the PreferGo logic is not hitting /etc/hosts (because no /etc/nsswitch.conf?) and this particular test-case only exposes the issue when using Google's DNS servers.

If something more unique is put into /etc/hosts, then this issue becomes reproducible. I have 127.0.0.5 fred-test laying around and that works reliably for exposing the issue.

[philpennock]

With .PreferGo set, lookup_unix.go:(*Resolver).lookupHost() leaves order == hostLookupCgo before falling through to call r.goLookupHostOrder and dnsclient_unix.go:(*Resolver).goLookupHostOrder only considers if order == hostLookupFilesDNS || order == hostLookupFiles to decide if lookupStaticHost should be used.

Thus GODEBUG=netdns=go changes the order and .goLookupHostOrder() handles things one way, but using .PreferGo does not make the same change to the order and the .goLookupHostOrder() function uses different logic, thus the documentation on PreferGo asserting:

It is equivalent to setting GODEBUG=netdns=go, but scoped to just this resolver.

is incorrect.

[andybons]

@bradfitz

[ydnar]

@philpennock is right—the documentation stating It is equivalent to setting GODEBUG=netdns=go, but scoped to just this resolver isn’t quite right.

When the example program is called with GODEBUG=netdns=go it reads from files (/etc/hosts) before network.

[bradfitz]

I can reproduce on a Mac, adding 127.0.0.5 bradtest to my /etc/hosts:

bradfitzlaptop go$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost

127.0.0.5       bradtest
bradfitzlaptop go$ uname -a
Darwin bradfitz-macbookpro.roam.corp.google.com 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1\
/RELEASE_X86_64 x86_64

With my just-sent https://golang.org/cl/100875, it now works as expected:

bradfitzlaptop go$ GODEBUG=netdns=go+2 go run ~/localhost.go
go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(bradtest) = files,dns
net.LookupHost addrs: [127.0.0.5] err: <nil>

PreferGo = true:
go package net: hostLookupOrder(bradtest) = files,dns
net.LookupHost addrs: [127.0.0.5] err: <nil>

vs

bradfitzlaptop go$ GODEBUG=netdns=cgo+2 go run ~/localhost.go
go package net: using cgo DNS resolver
go package net: hostLookupOrder(bradtest) = cgo
net.LookupHost addrs: [127.0.0.5] err: <nil>

PreferGo = true:
go package net: hostLookupOrder(bradtest) = files,dns
net.LookupHost addrs: [127.0.0.5] err: <nil>

[gopherbot]

Change https://golang.org/cl/100875 mentions this issue: net: make Resolver.PreferGo work more as documented