Support multiple health checks per service

[ryanuber]

Allows multiple health checks to be registered for a single service. Preserves backwards compatibility of configs by using a new checks field. Registering multiple health checks can be done in a few different ways:

Embedded in service definitions in the checks key. If there is only a single check, the check ID will be service:<service name>. If there are multiple checks, the service IDs will be service:<service name>:<n>.

{
  "service": {
    ...
    "checks": [
      {
        "script": "/usr/bin/true",
        "interval": "10s"
      },
      ...
    ]
  }
}

In check configuration files using the service_id key. This allows additive configuration of service checks in a consul.d-style directory:

{
  "checks": [
    {
      "name": "redis check",
      "script": "/usr/bin/true",
      "interval": "10s",
      "service_id": "redis"
    },
    ...
  ]
}

From the HTTP API by specifying service_id in the check definition. This allows adding additional health checks at runtime, and utilizes the new service/check persistence logic to restore the checks on later agent starts:

PUT /v1/agent/check/register
{
  "name": "redis check",
  "script": "/usr/bin/true",
  "interval": "10s",
  "service_id": "redis"
}

Fixes #230.

[ryanuber]

One thing that has also changed is that check registrations will be rejected if the service ID they are associated with does not exist. Shouldn't be a big deal and prevents a weird case where the agent tries to anti-entropy the check in to consul and the fsm rejects it.

[armon]

Awesome! This LGTM! I think this also fixes the issue of registering a check after the service is already created too?

[armon]

I don't think the FSM / State Store handle this, so I think you need to make changes to the RegisterRequest path too

[ryanuber]

Oh you are right. I added this in 97c5d53, is that what you meant?

[armon]

LGTM! The only thing I wonder about is a newer client (0.5.0) with an older server (0.4.1) will have issues potentially. I think we document in the upgrade notes that servers must be upgraded first (typically the case with our updates anyways).

[ryanuber]

@armon Hm, yeah I think that is the case. We could modify the code path to continue passing the single .Check field through to preserve compatibility, but there would still be this weird gap if a client restarted using the newer .Checks field instead. I think advising in the upgrade documentation is probably the most reasonable thing to do. Should we do this in the release notes when 0.5 is cut?

[ryanuber]

I've made a few adjustments in here after some experimentation. What we have now should be compatible with older versions of Consul by using Check instead of Checks when there is only a single check for a service (which would be the case on older Consuls).

We also solve the issues @armon mentioned earlier by batching service sync operations with check syncs where appropriate. Basically, if a check is to be synced, and is associated with a service, we attach the service to the RegisterRequest if it is out of sync or doesn't exist. This requires scoping the RegisterRequest to a single service and its associated checks, which is handled in syncChecks().

One final change is that we batch all checks which are not associated with any service together to reduce the number of RPC requests required to perform anti-entropy.

[armon]

I think we should warn about the service registration (once) not the checks. Since Checks cannot be blocked currently.

[ryanuber]

@armon I ended up reverting syncChecks back to its original syncCheck, and instead modifying syncService so that it searches for any out-of-sync checks and piggybacks them along with it. Reduced complexity quite a lot. I think this is ready for a once-over.

[armon]

LGTM! Killing it. 👍

[ryanuber]

I couldn't get Travis to pass with this PR due to some Makefile changes recently, even after brute-forcing the rebuild button. They do pass locally. I'm going to address this outside of this PR.

[kdlan]

This feature seems not documented on https://www.consul.io/docs/agent/http/agent.html#agent_check_register

But request is worked

PUT /v1/agent/check/register
{
  "Name": "redis check",
  "Script": "/usr/bin/true",
  "Interval": "10s",
  "ServiceID": "redis"
}