pushing the same blob twice with a different name fails

[yuval-k]

Pushing the same blob, with different names to a repo that requires authorization only for pushing fails.
The reason is, that since the blob exists, only the manifest needs to be pushed.
The manifest is pushed in a flow that doesn't allow for retries. The request fails as 401, because it is the first request that needs auth, and not retried.

I wasn't sure what's the best approch to fix this in ORAS, will be happy to contribute a PR once we agree on a solution.

The error message received:

{"errors:" [{"code":"DENIED","message": "not authorized","detail": "not authorized"}]}
Error: failed commit on ref "manifest-sha256:cfdd09ec9b5584b26b9d21caf8abff7122fe29ea74f5bd92b48d927775876e3e": unexpected status: 401 Unauthorized

Workaround:

Manually calling the root of the api and adding it to the authorizer before pusing. we implemented it here:
https://github.com/solo-io/wasme/blob/839c7818e16d715d4139393179004899a9d619c3/pkg/push/push.go#L114

to reproduce:

docker run -dp 5001:5000 --restart=always --name registry registry

# proxy to require auth on put\post
envoy -c silly.yaml &

# use foo\foo as user password
docker login localhost:5000
echo hi > foo
oras push localhost:5000/hello:v1 foo
oras push localhost:5000/hello:v2 foo # <- this will fail with a 401

silly.yaml:

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address: { address: 0.0.0.0, port_value: 5000 }
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: namespace.local_service
              domains: ["*"]
              routes:
              # if it is POST or PUT and Authorizaiton header doesn't exist deny
              - match:
                  prefix: "/"
                  headers:
                  - name: "authorization"
                    exact_match: "Basic Zm9vOmJhcg=="
                route: { cluster: dockerregistry }
                # require auth for /v2/
              - match:
                  path: "/v2/"
                  headers:
                  - name: ":method"
                    exact_match: "GET"
                  - name: "authorization"
                    present_match: true
                    invert_match: true
                response_headers_to_add:
                - header:
                    key: www-authenticate
                    value: Basic realm="foo"
                direct_response:
                  status: 401
                  body: 
                    inline_string: not-authorized
                # allow get and head
              - match:
                  prefix: "/"
                  headers:
                  - name: ":method"
                    exact_match: "GET"
                route: { cluster: dockerregistry }
              - match:
                  prefix: "/"
                  headers:
                  - name: ":method"
                    exact_match: "HEAD"
                route: { cluster: dockerregistry }
              - match: { prefix: "/" }
                response_headers_to_add:
                - header:
                    key: www-authenticate
                    value: Basic realm="foo"
                direct_response:
                  status: 401
                  body: 
                    inline_string: not-authorized
                name: catchall
          http_filters:
          - name: envoy.router
  clusters:
  - name: dockerregistry
    connect_timeout: 0.25s
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    hosts: [{ socket_address: { address: 127.0.0.1, port_value: 5001 }}]

For extra context, stack trace of the failing request (just before it is sent):

github.com/containerd/containerd/remotes/docker.dockerPusher.Push (/home/yuval/go/pkg/mod/github.com/containerd/containerd@v1.3.0/remotes/docker/pusher.go:225)
github.com/containerd/containerd/remotes/docker.(*dockerPusher).Push (Unknown Source:1)
github.com/containerd/containerd/remotes.push (/home/yuval/go/pkg/mod/github.com/containerd/containerd@v1.3.0/remotes/handlers.go:154)
github.com/containerd/containerd/remotes.PushHandler.func1 (/home/yuval/go/pkg/mod/github.com/containerd/containerd@v1.3.0/remotes/handlers.go:146)
github.com/containerd/containerd/remotes.PushContent (/home/yuval/go/pkg/mod/github.com/containerd/containerd@v1.3.0/remotes/handlers.go:216)
github.com/deislabs/oras/pkg/oras.Push (/home/yuval/go/pkg/mod/github.com/deislabs/oras@v0.7.0/pkg/oras/push.go:54)
github.com/solo-io/wasme/pkg/push.(*PusherImpl).Push (/home/yuval/Projects/solo/extend-envoy/pkg/push/push.go:115)
github.com/solo-io/wasme/pkg/cmd/push.runPush (/home/yuval/Projects/solo/extend-envoy/pkg/cmd/push/push.go:61)
github.com/solo-io/wasme/pkg/cmd/push.PushCmd.func1 (/home/yuval/Projects/solo/extend-envoy/pkg/cmd/push/push.go:45)
github.com/spf13/cobra.(*Command).execute (/home/yuval/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:826)
github.com/spf13/cobra.(*Command).ExecuteC (/home/yuval/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914)
github.com/spf13/cobra.(*Command).Execute (/home/yuval/go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864)
github.com/solo-io/wasme/pkg/cmd.Run (/home/yuval/Projects/solo/extend-envoy/pkg/cmd/cmd.go:49)
main.main (/home/yuval/Projects/solo/extend-envoy/main.go:16)
runtime.main (/home/yuval/bin/go/src/runtime/proc.go:203)
runtime.goexit (/home/yuval/bin/go/src/runtime/asm_amd64.s:1357)

[SteveLasker]

Thanks for posting Yuval,
I'm just wondering if this is an issue with docker distribution, as I can push the same artifact to ACR without issues.

[yuval-k]

I need to refresh my memory, but I don't think so. I think it happened when only a new manifest is needed to be pushed. the code flow where it is pushed doesn't retry in case of 401.

[sjackman]

I just ran into this same issue myself.

[jdolitsky]

@yuval-k @sjackman - can you share which registry you are using? is this distribution (docker registry)?

[sjackman]

I am using GitHub Packages Docker Registry https://ghcr.io

[sjackman]

Since that failure, I have pushed many tags that re-use blobs (layers), and haven't encountered an error.

[yuval-k]

@jdolitsky i provided a reproduction script that works with docker registry. see my original description

[sjackman]

Since that failure, I have pushed many tags that re-use blobs (layers), and haven't encountered an error.

My failure re-used blobs (layers), but not the actual manifest itself. Perhaps reusing the manifest is the issue.

[shizhMSFT]

This should be auto resolved once we move to oras-go v2. Let's validate it after the oras-go v2 migration.