Gateway downstream health checks

[trevor-scheer]

Purpose

This PR introduces a new configuration option to the gateway: performServiceHealthChecks. This option adds an additional precautionary step to the gateway load and schema update code paths.

A gateway with this setting turned on will send a simple graphql query to all downstream services ({ __typename }) to ensure the service is life and responsive. It performs these queries on initial schema load as well as during a schema update. The interesting failure behaviors are:

• On load: throw an error, failure to load.
• On schema update: "rollback" (never roll forward). Log the event, but continue serving requests with the old schema.

TODO:

• Docs
• Changelog
• PR Description

[abernix]

This looks wonderful! Really a nice addition to the reliability of Gateway and I think addresses the largest concern raised in https://github.com/apollographql/apollo-server/issues/3540.

In that regard, we may also want to provide documentation for how to use it with Apollo Server's onHealthCheck to meet the requirements of most health-check probes, which might be as simple as:

const gateway = new ApolloGateway({ performServiceHealthChecks: true });
const server = new ApolloServer({
  gateway,
  onHealthCheck() {
    /* Assuming no existing functionality */
    return gateway.performServiceHealthChecks();
  }
});

... and then a recommendation to leverage https://server:port/.well-known/apollo/server-health as the endpoint to ping.

Nothing blocking here, and you should feel free to move this forward however you like, but I did leave a few comments within.

[abernix]

565d96e looks good to me, even if it's a complicated beast.

Let's not do it now, but in the future, we might consider exposing a __testing property if it makes things particularly more simple. Hiding that from TypeScript might take some slight but worthwhile crafting.