Serve client API from exported GCS bucket

[jonasfj]

We can point GCLB at a prefix within a GCS bucket, so ideally we should be able to export responses for everything matching:

• GET /api/packages/<package>
• GET /api/packages/<package>/advisories
• GET /api/archives/<package>-<version>.tar.gz

Export them into a GCS bucket, under prefixes:

• /latest/ (all deployments will write to this concurrently)
• /<runtimeVersion>/ (each deployment will have it's own prefix)

Then GCLB can be pointed to /latest/ for URLs matching the above routes.

If a new deployment of pub.dev starts producing garbled output in /latest/, we can update GCLB to point to /<runtimeVersion>/ from the previous deployment. That way we still have a fallback mechanism.

To do this we'll need the following:

• An abstraction around the bucket containing exported API responses.
  • Must be able to GC old /<runtimeVersion>/.
  • Must be able to GC deleted packages.
    (Should probably wait until they are 1 day old, to avoid race conditions between creation/deletion).
• A updater function that given a package name ensures all affected responses are updated.
  • Ideally, this can compute a hash of the contents and avoid unnecessary updates.
• A backend that runs continuously (similar to how taskBackend works) and scans for packages that have been updated, calling the updater function.
• Calls to updater function when a new version is published, retracted, moderated, deleted, etc.
• Tests for all components.
• Provisioning of new bucket.
• Mapping of traffic on staging to bucket using GCLB.
• Mapping of traffic in production to bucket using GCLB.

[sigurdm]

Does this mean that api responses can only be updated reliably after all old deployments are gone?