Allow multiple health checks per service

[tgross]

We've had a few folks request the ability to have multiple health checks for a single service. In order to accomplish this, we'd need to separate the service definition from the health check. We'll make the following changes:

• ContainerPilot maintains state for all services, which can be either HEALTHY or UNHEALTHY
• a service in a HEALTHY will heartbeat to discovery every heartbeat seconds with a TTL of ttl.
• a health check will poll every poll seconds, with a timeout of timeout. If any health check fails, the associated service is marked UNHEALTHY and a Fail message is sent to the discovery service
• If any subsequent health check fails, the associated service is marked HEALTHY and a Pass message is sent to the discovery service.

The proposed configuration would be as follows. (Note this uses the new HCL syntax from #236 (comment)).

service "app" {
    port = 80
    tll = 10
    heartbeat = 3
}

health {
    service = "app"
    check = "/usr/bin/curl --fail -s -o /dev/null http://localhost/app"
    poll = 5
    timeout = "5s"
}

[fitz123]

So you finally succumb to it)
Ok, will brake down big health scripts into small checks right into CP config file. At least it requires less files to be opened in an editor)

[tgross]

Partially released in https://github.com/joyent/containerpilot/releases/tag/3.0.0-dev.1. The configuration syntax changes to support it are coming next. Refer to RFD86 and the v3 roadmap in #283 for details.

[tgross]

Implemented in #316

[tgross]

This feature has undergone some internal review and we've decided to cut it, as the semantics of multiple health checks for a service don't really make sense given that we can only report to Consul once.