Intermittent test failure

[brikis98]

We sometimes get a test failure like this:

TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:10Z retry.go:72: Check Nomad cluster has expected number of servers and clients
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:10Z nomad_helpers.go:214: Making an HTTP GET to URL http://54.200.89.51:4646/v1/nodes
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:15Z nomad_helpers.go:227: Response from Nomad for URL http://54.200.89.51:4646/v1/nodes: No cluster leader
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:15Z retry.go:84: Check Nomad cluster has expected number of servers and clients returned an error: invalid character 'N' looking for beginning of value. Sleeping for 10s and will try again.
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:25Z retry.go:72: Check Nomad cluster has expected number of servers and clients
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:25Z nomad_helpers.go:214: Making an HTTP GET to URL http://54.200.89.51:4646/v1/nodes
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:30Z nomad_helpers.go:227: Response from Nomad for URL http://54.200.89.51:4646/v1/nodes: No cluster leader
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:30Z retry.go:84: Check Nomad cluster has expected number of servers and clients returned an error: invalid character 'N' looking for beginning of value. Sleeping for 10s and will try again.
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:40Z retry.go:72: Check Nomad cluster has expected number of servers and clients
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:40Z nomad_helpers.go:214: Making an HTTP GET to URL http://54.200.89.51:4646/v1/nodes
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:46Z nomad_helpers.go:227: Response from Nomad for URL http://54.200.89.51:4646/v1/nodes: No cluster leader
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:46Z retry.go:84: Check Nomad cluster has expected number of servers and clients returned an error: invalid character 'N' looking for beginning of value. Sleeping for 10s and will try again.
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:56Z retry.go:72: Check Nomad cluster has expected number of servers and clients
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:17:56Z nomad_helpers.go:214: Making an HTTP GET to URL http://54.200.89.51:4646/v1/nodes
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:18:01Z nomad_helpers.go:227: Response from Nomad for URL http://54.200.89.51:4646/v1/nodes: No cluster leader
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:18:01Z retry.go:84: Check Nomad cluster has expected number of servers and clients returned an error: invalid character 'N' looking for beginning of value. Sleeping for 10s and will try again.
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:18:11Z retry.go:72: Check Nomad cluster has expected number of servers and clients
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:18:11Z nomad_helpers.go:214: Making an HTTP GET to URL http://54.200.89.51:4646/v1/nodes
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:18:16Z nomad_helpers.go:227: Response from Nomad for URL http://54.200.89.51:4646/v1/nodes: No cluster leader
TestNomadConsulClusterColocatedAmazonLinux2Ami 2021-01-26T12:18:16Z retry.go:84: Check Nomad cluster has expected number of servers and clients returned an error: invalid character 'N' looking for beginning of value. Sleeping for 10s and will try again.

After a bunch more retries, it times out and fails the test. I wonder what response we're getting there?