integration with pushing/pulling images from Google Cloud Registry

[mattnworb]

This is a bit of a meta-issue, but we are interested at Spotify in better integrating docker-client with Google Container Registry and making it possible to push/pull images from there. There are some existing issues that cover some of the roadblocks today that prevent this from working but I wanted to create a new issue to track the overall work.

• see the comments in Error pulling from gcloud private registry #447 (comment) that outline a portion of the problem with GCR around short-lived access tokens

- #629 there is a new-ish auth API in docker API > 1.23 that docker-client doesn't have support for

- (possibly covered by the above issue) - docker-client in general seems unaware of how to use tokens when sending authenticated requests

[mattnworb]

After looking further #629 is not necessary for pushing/pulling images to GCR, as the user is likely not authenticating with GCR via the docker login method.

[mattnworb]

I also crossed out the last bullet point in my original issue summary as docker-client is able to send the contents of the auths[].auth field from the docker config file since 8e0b055.

[mattnworb]

I think the largest with pushing/pulling images to GCR is that the tokens that gcloud docker -a or gcloud docker -- push .. write to the docker config file are short-lived and need to be refreshed periodically. I am not sure how long they are valid for, but as mentioned in #447 gcloud docker will attempt to update the token on each operation.

Since they are short-lived, constructing a DockerClient instance with something like

final DockerClient client DefaultDockerClient.fromEnv()
        .registryAuth(RegistryAuth.fromDockerConfig().build())
        .build();

will cause pulls/pushes to GCR to fail after a period of time if you intend to use the dockerClient instance for a long period of time, since the DefaultDockerClient is holding the RegistryAuth data in memory. The RegistryAuth config is only read when this object is constructed.

(There is another issue here in that RegistryAuth.fromDockerConfig() has bad behavior in that it reads the first entry in the auths JSON object, which may or may not be what the user wants. I'd like to deprecate/remove this method altogether to be able to support reading all of the RegistryAuth objects from the config file, but I will create a new issue for that)

One idea I have for solving this is to refactor the "registry auth" support within docker-client to better handle looking up the RegistryAuth data at operation-time rather than at construction-time.

I am thinking of adding an interface like:

interface RegistryAuthSupplier {
  /** 
   * return a single RegistryAuth struct for use with a specific repository 
   */
  RegistryAuth authFor(String registryName);

  /** 
   * When building an image (and only for this operation), the Docker API 
   * wants you to send information for all of the registries that the client is 
   * configured for. See 
   * https://docs.docker.com/engine/api/v1.28/#operation/ImageBuild for more info. 
   */
  RegistryAuths allAuths();

A method would be added to DefaultDockerClient.Builder to allow the user to pass this in, and the DefaultDockerClient would store this as a field, and remove the field for the (single) RegistryAuth. Internally the Builder can convert what it used to pass as the RegistryAuth field to an instance of this interface.

The RegistryAuthSupplier would then be invoked for each operation (push, pull, build, etc) that needs authentication. For operations other than build, DefaultDockerClient will inspect the image name to figure out the registry name (i.e. gcr.io) in order to pass this to the Supplier.

Using the pull() method as an example, it would change like:

+    final String registryName = parseRegistryName(image);
+    final RegistryAuth registryAuth = this.authSupplier.authFor(registryName);
+
     // authHeader(..) method does the base64 encoding
-    final String authHeader = authHeader(this.registryAuth);
+    final String authHeader = authHeader(registryAuth);
     try (ProgressStream pull =
              request(POST, ProgressStream.class, resource,
                      resource
                          .request(APPLICATION_JSON_TYPE)
                          .header("X-Registry-Auth", authHeader))) {
       pull.tail(handler, POST, resource.getUri());

It would then be possible for the user to pass in various implementations of RegistryAuthSupplier, e.g.:

• return a static RegistryAuth instance for all operations
• parse the ~/.docker/config.json file freshly for each operation
• use the google-cloud-sdk library to manually/explicitly get a new token
• etc etc

I think one nice benefit of this would be that it decouples the "how to get a token" logic from needing to existing within DefaultDockerClient or even within docker-client itself - it will be easy to customize and extend by users for various registries and authentication schemes.

[mattnworb]

more work is going on with this in #762

[mattnworb]

done in v8.6.0