Cannot create image stream from private hub.docker.com registry

[feedm3]

I want to deploy a docker image which I have pushed to hub.docker.com but whenever I import the private registry as image stream I get an unauthenticated error although I have created a docker secret within openshift (related docs).

Version

oc v3.7.1+ab0f056
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth

Server https://api.starter-us-west-2.openshift.com:443
openshift v3.7.23
kubernetes v1.7.6+a08f5eeb62

Steps To Reproduce

1. Create a secret to be authorized for the private docker registry (not sure if the docker-server parameter is correct, official openshift docs are very unclear about that)

oc secrets new-dockercfg pull-secret \
    --docker-server=docker.io \
    --docker-username=MY_USER \
    --docker-password=MY_PASSWORD \
    --docker-email=unused

2. Use secret for pull

oc secrets link default pull-secret --for=pull

3. Try to import from the private docker registry

oc import-image my-private-registry --from=docker.io/feedme/learning-openshift --confirm

Current Result

The import completed with errors.

Name:                   my-private-registry
Namespace:              test-namespace
Created:                Less than a second ago
Labels:                 <none>
Annotations:            openshift.io/image.dockerRepositoryCheck=2018-02-05T19:21:46Z
Docker Pull Spec:       172.30.254.23:5000/test-namespace/my-private-registry
Image Lookup:           local=false
Unique Images:          0
Tags:                   1

latest
  tagged from docker.io/feedme/learning-openshift

  ! error: Import failed (Unauthorized): you may not have access to the Docker image "docker.io/feedme/learning-openshift:latest"
      Less than a second ago

error: tag latest failed: you may not have access to the Docker image "docker.io/feedme/learning-openshift:latest"

Expected Result

I expect the registry to be imported so that I can later on create an app.

[0xmichalis]

@openshift/sig-developer-experience

[bparees]

@juanvallejo is this the issue w/ docker secret being created in the wrong format? (what level did that issue exist in, what level is it fixed in, and assuming it is that issue, is there a workaround we can give @feedm3 ?)

[jim-minter]

Hi @feedm3 :)

I'm assuming that with the username and password that you're using, a docker login / docker pull will work? Please make sure.

I think the issue is probably down to the --docker-server argument: it must match the remote server otherwise OpenShift won't select the secret when it's trying to pull the image down. Please could you try each of the following arguments (one at a time) and see if either works? --docker-server=auth.docker.io/token, --docker-server=index.docker.io/v1/. Be sure to delete any existing dockercfg secrets and imagestreams that were created between each attempt.

The oc secrets link default pull-secret --for=pull command is not necessary as when you run oc import-image or oc tag, the OpenShift backend selects the secret "magically" by iterating all the secrets of the relevant type in the same project to find the first that matches.

oc import-image should work for this use case (you should also be able to omit --from and just do oc import-image docker.io/feedme/learning-openshift:latest --confirm), but the recommendation is to use oc tag to set up the imagestream in the first place, then use oc import-image to kick it any time you want it to refresh.

[juanvallejo]

@bparees

is this the issue w/ docker secret being created in the wrong format? (what level did that issue exist in, what level is it fixed in, and assuming it is that issue, is there a workaround we can give @feedm3 ?)

This looks likely to be the same issue. I believe only oc secrets new-dockercfg was affected on the Origin side. oc create secret docker-registry should be a workaround - I will test this locally to make sure.

We did address part of the problem in 3.7: #18062
But missed the second half of this fix: #18226
as it was only backported to 3.8.

Will go ahead and cherry pick #18226 into 3.7
@soltysh fyi

[feedm3]

I'm assuming that with the username and password that you're using, a docker login / docker pull will work?

Yep, works.

I tried both auth.docker.io/token and index.docker.io/v1/ as server but neither didn't work. I also tried oc secrets new-dockercfg pull-secret and oc create secret docker-registry pull-secret but also there neither didn't work.

oc create secret docker-registry pull-secret \
    --docker-server=auth.docker.io/token \
    --docker-username=MY_USER \
    --docker-password=MY_PASSWORD \
    --docker-email=unused
oc import-image docker.io/feedme/learning-openshift --confirm

I'm always getting the same unauthorized error :(

@jim-minter

...but the recommendation is to use oc tag to set up the imagestream in the first place, then use oc import-image to kick it any time you want it to refresh.

Can you go a little bit more in detail what you mean?

[bparees]

I can confirm the docker-server value should be "docker.io", i was able to make the oc create secret docker-registry syntax work, however i'm on a level that includes all the fixes, so i don't know if it's a valid workaround on v3.7 or not. @feedm3 if you didn't set the docker-server to "docker.io" try that. If you did try that, i guess we'll have to wait for @juanvallejo to get back.

The other thing you can do is manually create the secret. First do a docker login docker.io locally and ensure you have a ~/.docker/config.json w/ an entry for docker.io. Then run:

oc create secret generic --from-file=.dockerconfigjson=/path/to/your/.docker/config.json --type=kubernetes.io/dockerconfigjson pullsecret

That should create the secret in the right format. You may also need to delete any other secrets you've created for docker.io.

[feedm3]

Thanks @bparees, it works! What is important is that by default docker stores it's secrets in the credentials store and not in the config. Therefore you need to put an empty value for the credsStore key in the ~/.docker/config.json and redo a docker login docker.io.

So, short working conclusion:

1. Check for empty value in credsStore variable in ~/.docker/config.json
2. docker login docker.io
3. oc create secret generic --from-file=.dockerconfigjson=.docker/config.json --type=kubernetes.io/dockerconfigjson pullsecret
4. oc import-image docker.io/feedme/learning-openshift --confirm

[juanvallejo]

Per the current state of the release-3.7 branch, oc create secret docker-registry behaves correctly, but creates secrets in the old dockercfg format, rather than the new config.json format [1][2].

The same is true of oc secrets new-dockercfg [3][4].

It could be that the docker server expects the new format. If that is the case, newer clients updated both of these commands to create secrets in the new config.json format. [5][6].

1. https://github.com/openshift/origin/blob/release-3.7/vendor/k8s.io/kubernetes/pkg/kubectl/secret_for_docker_registry.go#L125

2. https://github.com/openshift/origin/blob/release-3.7/vendor/k8s.io/kubernetes/pkg/kubectl/secret_for_docker_registry.go#L83

3. https://github.com/openshift/origin/blob/release-3.7/pkg/oc/cli/secrets/dockercfg.go#L128

4. https://github.com/openshift/origin/blob/release-3.7/pkg/oc/cli/secrets/dockercfg.go#L137

5. https://github.com/openshift/origin/blob/master/pkg/oc/cli/secrets/dockercfg.go#L140

6. https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/kubectl/secret_for_docker_registry.go#L94

[bparees]

closing this since it seems like the issue is fixed in newer code and we've got a workaround for @feedm3's env.

[feedm3]

@bparees I would like to reopen this issue. Unfortunately, it's currently not possible to deploy an image from a private repository form hub.docker.com.

After creating the image pull secret, I was able to create an image stream. The image stream also seemed fine as the configuration and information about the image were correct. But deploying from this image stream doesn't work. I tried all commands in all combinations mentioned in the comments above but every single time the deployment failed with "Failed to pull image "docker.io/feedme/learning-openshift": rpc error: code = Unknown desc = unauthorized: authentication required".

[mxxk]

@feedm3—I just confirmed the same thing you're seeing, and it seems specific to docker.io. Something about how docker.io is doing authentication is not playing nicely with OpenShift feeding the right secrets.

Seeing as the new issue is not with image stream creation but rather with pod creation (and failed image pull), I created a new wall of text to document it. Head on over: #18932

[membrive]

I can confirm the same issue. I can get the image metadata from docker.io but when I create the application, OpenShift can't pull the image: Failed to pull image "membrive/test-image:latest": rpc error: code = Unknown desc = repository docker.io/membrive/test-image not found: does not exist or no pull access.

[bparees]

@feedm3 @mxxk @membrive if your pod can't pull the image you have not setup your service account secrets properly. that is unrelated to imagestreams.

if you'd like help w/ that issue, please open a new issue and describe how you've created your secret and pod, preferably w/o imagestreams just to keep the configuration simple (just a pod referencing the docker hub image directly).