Request to catalog/datacenters hangs

[tpham-here]

We have a Consul server (1 of 5 in a cluster) that is not responding to GET v1/catalog/datacenters requests. The request hangs on that one Consul server. Normally, this GET request responds within seconds. Other requests to the catalog on that Consul server work fine, e.g. v1/catalog/nodes and v1/catalog/services. The Consul server has been online for the past 5 weeks. The VM that the Consul server is on is functioning fine, i.e. tcpdump, ping, nmap, iperf shows the correct ports are opened and traffic is flowing smoothly.

There was an issue back in August 2015 having similar symptoms - #1154. There is not much in the logs that detail what could have gone wrong. The only workaround we have was to restart the Consul server. The Consul server we are running is:

# consul version
Consul v0.6.3
Consul Protocol: 3 (Understands back to: 1)

The patch for issue #1154 should have already been included in our version. Please advise.

[slackpad]

Hi @tpham-here if you run consul members -wan on that server does it show any other servers on the WAN? I'm wondering if that server isn't properly joined to any others in the WAN, or it has some bogus entries for failed servers. If it doesn't look right you should be able to consul join -wan <another server> to get it introduced to the other WAN members.

[tpham-here]

I recently encountered the same issue and got a chance to issue 'consul members -wan'. The command hangs.

[tetsusoh]

vote. I had similar problem. consul members -wan returned as normal and other api also responsive except the /v1/catalog/datacenters

[slackpad]

We completely reworked the guts of this in Consul 0.8.0 - the RPC interface is gone and this uses the regular HTTP API, and we updated the WAN internals. I'll close this out for now since we haven't seen any more reports of this. Please let us know if you can reproduce this with 0.8.0 or later.