Implement check for resolving localhost

[aelkugia]

Describe the error

Customer reported the watcher container in the influxdb pod is constantly failing because it couldn't connect to influxdb as expected. After upgrading from 1.4.5 to 1.4.54, started to experience issues with InfluxDB. The pod kept restarting with status CrashLoopBackOff.

Cause of the error

The error was caused by a bad localhost resolution to an invalid ip which is not 127.0.0.1.

Expected behavior

A precheck should exist for localhost to resolve correctly in the cluster. This will ensure user fixes dns before continuing with installation.

[aelkugia]

It was found that resolving from the dns server instead of /etc/hosts/is the Go behavior if cannot find /etc/nsswitch file in the container.

There is a fix expected in Go 1.15 release. Discussion can be found here: golang/go#35305

[terrywang]

Just to correct, it's the /etc/nsswitch.conf file provided by glibc package on Fedora / EL and libc-bin on Debian/Ubuntu.

NOTE: On Arch Linux and its variants (e.g. Manjaro) that I use as workstation, the file is provided by filesystem.

[knisbet]

It might be possible to just hardcode localhost in the coredns configuration. Depends a bit on whether the query is for localhost or localhost..

[knisbet]

This is fixed in 5.5.x by #1214 / gravitational/monitoring-app#145 by fixing the cause of leaking localhost queries, as opposed to implementing a check for a capturing DNS resolver which is a valid customer deployment option.