-
Notifications
You must be signed in to change notification settings - Fork 180
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error: PUT request body is not rewindable #456
Comments
@jobcespedes Is this recurring or intermittent? Looks like blob a39831967c0c(all_state_image_artifact.tar.gz) got uploaded but got 401 when querying the status. |
I got it consistently with a file, let's call it file A. Then I tried to push a smaller file B. The smaller one got pushed. I also tried a different file C, bigger than B, it through the error. After that, I downgraded to Oras version 0.8.0, pushed file A and it worked without issues. |
Just to confirm, with oras 0.13.0,
With oras 0.8.0, ABC can all be uploaded |
I mislead you with my suggestion bigger file size had something to do. Sorry. This are correct the upload results acording to file sizes in my case: file A.tar.gz: 3122 -> consistent error However, with other oras version (it is actually v0.8.1) there are no errors with any of those three files. |
Can you kindly share the debugging logs for pushing |
Also how did you provide the credential? Via running |
/cc @nima for helping investigation. |
Via
Command Preparing /tmp/testA.tar.gz
DEBU[0000] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/manifests/sha256:0be5fae8c8cbfdce92f692f51656a125ee62a53f9504cc75ac233c495eaf1253"
DEBU[0000] Request method: "HEAD"
DEBU[0000] Request headers:
DEBU[0000] "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.cncf.oras.artifact.manifest.v1+json"
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0000] Response Status: "401 Unauthorized"
DEBU[0000] Response headers:
DEBU[0000] "Content-Type": "application/json; charset=utf-8"
DEBU[0000] "Content-Length": "58"
DEBU[0000] "Connection": "keep-alive"
DEBU[0000] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0000] "Www-Authenticate": "Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\""
DEBU[0000] "Date": "Tue, 26 Jul 2022 04:09:50 GMT"
DEBU[0000] Request URL: "https://public.ecr.aws/token/?scope=aws&scope=repository%3Ae3i0l4d7%2Fall_state_image_artifact%3Apull&service=public.ecr.aws"
DEBU[0000] Request method: "GET"
DEBU[0000] Request headers:
DEBU[0000] "Authorization": "*****"
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0000] Response Status: "200 OK"
DEBU[0000] Response headers:
DEBU[0000] "Date": "Tue, 26 Jul 2022 04:09:50 GMT"
DEBU[0000] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0000] "Content-Type": "application/json; charset=utf-8"
DEBU[0000] "Connection": "keep-alive"
DEBU[0000] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/manifests/sha256:0be5fae8c8cbfdce92f692f51656a125ee62a53f9504cc75ac233c495eaf1253"
DEBU[0000] Request method: "HEAD"
DEBU[0000] Request headers:
DEBU[0000] "Accept": "application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.cncf.oras.artifact.manifest.v1+json"
DEBU[0000] "Authorization": "*****"
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0000] Response Status: "404 Not Found"
DEBU[0000] Response headers:
DEBU[0000] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0000] "Content-Type": "application/json; charset=utf-8"
DEBU[0000] "Content-Length": "79"
DEBU[0000] "Connection": "keep-alive"
DEBU[0000] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0000] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/sha256:39c608eccf630ae7d883f36bd3878b3bb52e6cb79e24947569b9a34ec1cbb8cf"
DEBU[0000] Request method: "HEAD"
DEBU[0000] Request headers:
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0000] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
DEBU[0000] Request method: "HEAD"
DEBU[0000] Request headers:
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0000] Response Status: "401 Unauthorized"
DEBU[0000] Response headers:
DEBU[0000] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0000] "Content-Type": "application/json; charset=utf-8"
DEBU[0000] "Content-Length": "58"
DEBU[0000] "Connection": "keep-alive"
DEBU[0000] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0000] "Www-Authenticate": "Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\""
DEBU[0000] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/sha256:39c608eccf630ae7d883f36bd3878b3bb52e6cb79e24947569b9a34ec1cbb8cf"
DEBU[0000] Request method: "HEAD"
DEBU[0000] Request headers:
DEBU[0000] "Authorization": "*****"
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0000] Response Status: "401 Unauthorized"
DEBU[0000] Response headers:
DEBU[0000] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0000] "Www-Authenticate": "Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\""
DEBU[0000] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0000] "Content-Type": "application/json; charset=utf-8"
DEBU[0000] "Content-Length": "58"
DEBU[0000] "Connection": "keep-alive"
DEBU[0000] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
DEBU[0000] Request method: "HEAD"
DEBU[0000] Request headers:
DEBU[0000] "Authorization": "*****"
DEBU[0000] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "404 Not Found"
DEBU[0001] Response headers:
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
Uploading 39c608eccf63 /tmp/testA.tar.gz
DEBU[0001] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/"
DEBU[0001] Request method: "POST"
DEBU[0001] Request headers:
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "404 Not Found"
DEBU[0001] Response headers:
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
Uploading 44136fa355b3 application/vnd.unknown.config.v1+json
DEBU[0001] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/"
DEBU[0001] Request method: "POST"
DEBU[0001] Request headers:
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "401 Unauthorized"
DEBU[0001] Response headers:
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0001] "Content-Type": "application/json; charset=utf-8"
DEBU[0001] "Content-Length": "58"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001] "Www-Authenticate": "Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\""
DEBU[0001] Request URL: "https://public.ecr.aws/token/?scope=aws&scope=repository%3Ae3i0l4d7%2Fall_state_image_artifact%3Apull%2Cpush&service=public.ecr.aws"
DEBU[0001] Request method: "GET"
DEBU[0001] Request headers:
DEBU[0001] "Authorization": "*****"
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "200 OK"
DEBU[0001] Response headers:
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0001] "Content-Type": "application/json; charset=utf-8"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001] Response Status: "401 Unauthorized"
DEBU[0001] Response headers:
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001] "Www-Authenticate": "Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\""
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0001] "Content-Type": "application/json; charset=utf-8"
DEBU[0001] "Content-Length": "58"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/"
DEBU[0001] Request method: "POST"
DEBU[0001] Request headers:
DEBU[0001] "Authorization": "*****"
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/"
DEBU[0001] Request method: "POST"
DEBU[0001] Request headers:
DEBU[0001] "Authorization": "*****"
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "202 Accepted"
DEBU[0001] Response headers:
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001] "Location": "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/0a5cb83a-5039-441b-bca4-df9ed55c5ef8"
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:51 GMT"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] "Range": "bytes=0-10485760"
DEBU[0001] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/0a5cb83a-5039-441b-bca4-df9ed55c5ef8?digest=sha256%3A39c608eccf630ae7d883f36bd3878b3bb52e6cb79e24947569b9a34ec1cbb8cf"
DEBU[0001] Request method: "PUT"
DEBU[0001] Request headers:
DEBU[0001] "Content-Type": "application/octet-stream"
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "202 Accepted"
DEBU[0001] Response headers:
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:52 GMT"
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001] "Location": "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/b8764924-1d05-4bbb-a9b6-a81089b766de"
DEBU[0001] "Range": "bytes=0-10485760"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] Request URL: "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/b8764924-1d05-4bbb-a9b6-a81089b766de?digest=sha256%3A44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a"
DEBU[0001] Request method: "PUT"
DEBU[0001] Request headers:
DEBU[0001] "Content-Type": "application/octet-stream"
DEBU[0001] "User-Agent": "oras/0.13.0"
DEBU[0001] Response Status: "401 Unauthorized"
DEBU[0001] Response headers:
DEBU[0001] "Date": "Tue, 26 Jul 2022 04:09:52 GMT"
DEBU[0001] "Content-Type": "application/json; charset=utf-8"
DEBU[0001] "Content-Length": "58"
DEBU[0001] "Connection": "keep-alive"
DEBU[0001] "Docker-Distribution-Api-Version": "registry/2.0"
DEBU[0001] "Www-Authenticate": "Bearer realm=\"https://public.ecr.aws/token/\",service=\"public.ecr.aws\",scope=\"aws\""
ERRO[0001] Error in getting response: %!w(*errors.errorString=&{context canceled})
Error: PUT "https://public.ecr.aws/v2/e3i0l4d7/all_state_image_artifact/blobs/uploads/0a5cb83a-5039-441b-bca4-df9ed55c5ef8?digest=sha256%3A39c608eccf630ae7d883f36bd3878b3bb52e6cb79e24947569b9a34ec1cbb8cf": request body is not rewindable |
@jobcespedes Thanks for reporting this error. It was caused by the PUT failure during pushing an artifact. As a registry client, ORAS should do the layer pushing in two step:
During PUT(step 2), ORAS fails to reuse the token obtained by POST(step 1) and ends in rewinding the request body. Why it only fails for ECR: We had a fix in oras-go and will apply the change to CLI. This issue will be fixed in upcoming ORAS release. cc @nima |
Hi @jobcespedes , Could you pls help to verify if this issue has been fixed? I think we can build the latest binary from the main branch. |
Sure. Would you share the latest binary with me? Or, you want me to built it? |
@jobcespedes We haven't done a release yet. You can build via following https://oras.land/cli/5_developer_guide/ |
Confirm: no error oras version
|
Closing as fix has been confirmed. |
This upgrades our oras cli to 0.14.0 which fixes a bug in the oras push command. This is a second attempt, as we previously tried 0.13.0 (to get the --password-stdin flag, but were blocked by the aforementioned push bug). The 0.14.0 release should include fixes that make this possible to use once again. The push bug: oras-project/oras#456 The previous attempt: aws#398
This upgrades our oras cli to 0.14.0 which fixes a bug in the oras push command. This is a second attempt, as we previously tried 0.13.0 (to get the --password-stdin flag, but were blocked by the aforementioned push bug). The 0.14.0 release should include fixes that make this possible to use once again. The push bug: oras-project/oras#456 The previous attempt: aws#398
This upgrades our oras cli to 0.14.1 which fixes bugs in the oras push command. This is third attempt, as we previously tried 0.13.0 (to get the --password-stdin flag, but were blocked by the aforementioned push bugs). Then 0.14.0 (which introduced a different push bug). The 0.14.1 release includes fixes that make this possible to use once again. The push bugs: - oras-project/oras#456 - oras-project/oras-go#252 The previous attempt: aws#398
Oras version 0.13.0
The text was updated successfully, but these errors were encountered: