Intermittent Hangs on Registry Push #1568

dgershman · 2023-04-06T20:36:35Z

Environment

Device and OS: AMD64 / Ubuntu 22.04
App version: 0.25.2
Kubernetes distro being used: K3D K3S v1.25.7+k3s1
Other: Big Bang 1.57.1

Steps to reproduce

zarf package deploy zarf-package-big-bang-example-amd64-1.57.1.tar.zst --confirm --log-level=trace
Occasionally the process gets hung up "Updating image ..."

Expected result

That things wouldn't get hung up, and continue along.

Actual Result

Things get hung up and/or sometimes fail / timeout.

Visual Proof (screenshots, videos, text, etc)

  DEBUG   2023-04-06T20:20:34Z  -  crane.Push() /tmp/zarf-1329914535/images:registry1.dso.mil/ironbank/fluxcd/kustomize-controller:v0.34.0 -> 127.0.0.1:34857/ironbank/fluxcd/kustomize-controller:v0.34.0)
└ (/home/runner/work/zarf/zarf/src/pkg/packager/deploy.go:403)
  ⠦  Updating image registry1.dso.mil/ironbank/fluxcd/kustomize-controller:v0.34.0 (2m5s))
  ⠧  Updating image registry1.dso.mil/ironbank/fluxcd/kustomize-controller:v0.34.0 (2m5s)
  DEBUG   2023-04-06T20:22:44Z  -  "msg"="error creating error stream for port 34857 -> 5000: Timeout occurred\n" "error"=null
└ (/home/runner/go/pkg/mod/github.com/go-logr/logr@v1.2.4/funcr/funcr.go:183)
  DEBUG   2023-04-06T20:24:01Z  -  "msg"="error creating error stream for port 34857 -> 5000: Timeout occurred\n" "error"=null
└ (/home/runner/go/pkg/mod/github.com/go-logr/logr@v1.2.4/funcr/funcr.go:183)
  DEBUG   2023-04-06T20:24:03Z  -  "msg"="error creating error stream for port 34857 -> 5000: Timeout occurred\n" "error"=null
└ (/home/runner/go/pkg/mod/github.com/go-logr/logr@v1.2.4/funcr/funcr.go:183)
  DEBUG   2023-04-06T20:33:33Z  -  "msg"="lost connection to pod\n" "error"=null.0 (13m13s)
└ (/home/runner/go/pkg/mod/github.com/go-logr/logr@v1.2.4/funcr/funcr.go:183)

Full logs:

full-logs.txt

Severity/Priority

There is a workaround, by keeping retrying until the process succeeds.

Additional Context

I don't believe this is specifically related to the bigbang component.
I have seen this happen on zarf init as well, specifically when pushes are happening to the registry.
I tested to see if the tunnel was still active and it was during the time of the error.

The text was updated successfully, but these errors were encountered:

dgershman · 2023-04-11T17:37:11Z

Interesting piece of information I think, when connected to the console (using either SSM or SSH) of an EC2 instance is when we see this issue. When running this locally and pointed to the same K8S cluster, we have no issues.

dgershman · 2023-04-15T02:35:35Z

I think this may have something to do with using k3d. I'm not seeing the intermittent issues with k3s.

…ushes (#1590) ## Description This PR creates a tunnel per image push (making it easier to implement concurrency - may do that in this PR if we can confirm that issues are mitigated) moves the CRC from the image name to the tag and changes the UI to use a progressbar instead of a spinner for better user feedback. ## Related Issue Relates to #1568 , #1433, #1218, #1364 This also will make #1594 slightly easier. (See aws/containers-roadmap#853) Fixes: #1541 ## Type of change - [X] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Other (security config, docs update, etc) ## Checklist before merging - [X] Test, docs, adr added or updated as needed - [X] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow) followed

Racer159 · 2023-04-25T18:45:40Z

@dgershman have you gotten a chance to try v0.26.0 and are you seeing any issues with that and k3d?

dgershman · 2023-04-26T15:54:04Z

Still having issues with K3D and tried with v0.26.1. It times out on the registry push step. This is the latest response:

     ERROR:  Failed to deploy package: unable to deploy all components in this Zarf Package: unable to deploy
             component bigbang: unable to push images to the registry: Head
             "http://127.0.0.1:46657/v2/ironbank/opensource/kubernetes/kubectl/blobs/sha256:b4f2e7ad23124dea1bfd0a239be05aaa5c7ddad088a5039b7cf799eddcfa278a":
             dial tcp 127.0.0.1:46657: connect: connection refused
     ERROR:  Failed to deploy package: unable to deploy all components in this Zarf Package: unable to deploy
             component bigbang: unable to push images to the registry: Head
             "http://127.0.0.1:46657/v2/ironbank/opensource/kubernetes/kubectl/blobs/sha256:b4f2e7ad23124dea1bfd0a239be05aaa5c7ddad088a5039b7cf799eddcfa278a":
             dial tcp 127.0.0.1:46657: connect: connection refused
     ERROR:  Failed to deploy package: unable to deploy all components in this Zarf Package: unable to deploy
             component bigbang: unable to push images to the registry: Head
             "http://127.0.0.1:46657/v2/ironbank/opensource/kubernetes/kubectl/blobs/sha256:b4f2e7ad23124dea1bfd0a239be05aaa5c7ddad088a5039b7cf799eddcfa278a":
             dial tcp 127.0.0.1:46657: connect: connection refused
     ERROR:  Failed to deploy package: unable to deploy all components in this Zarf Package: unable to deploy
             component bigbang: unable to push images to the registry: Head
             "http://127.0.0.1:46657/v2/ironbank/opensource/kubernetes/kubectl/blobs/sha256:b4f2e7ad23124dea1bfd0a239be05aaa5c7ddad088a5039b7cf799eddcfa278a":
             dial tcp 127.0.0.1:46657: connect: connection refused

Racer159 · 2023-04-27T14:51:36Z

Thanks for the update @dgershman the SSH note might be an interesting / good lead to look into as well - some but not all had reported that this was happening on EC2 VMs over SSH - I had thought it was bad networking and tested with that locally but potentially there is something more going on with driving image uploads from an SSH session. I'll see if I can keep digging

Racer159 · 2023-05-17T14:31:40Z

@dgershman doing some more testing on this, but if you get a chance does the code in #1721 solve the hanging for you? (note that parts of the push/pull of OCI Zarf packages are broken on that PR - not sure if that is in your flow though - won't affect local Zarf packages)

dgershman · 2023-05-17T15:53:03Z

@dgershman doing some more testing on this, but if you get a chance does the code in #1721 solve the hanging for you? (note that parts of the push/pull of OCI Zarf packages are broken on that PR - not sure if that is in your flow though - won't affect local Zarf packages)

Putting this on my todo list for today.

dgershman · 2023-05-18T02:54:49Z

@dgershman doing some more testing on this, but if you get a chance does the code in #1721 solve the hanging for you? (note that parts of the push/pull of OCI Zarf packages are broken on that PR - not sure if that is in your flow though - won't affect local Zarf packages)

Putting this on my todo list for today.

I ran one test so far that looked good. Will do two more tomorrow.

dgershman · 2023-05-18T20:46:33Z

@dgershman doing some more testing on this, but if you get a chance does the code in #1721 solve the hanging for you? (note that parts of the push/pull of OCI Zarf packages are broken on that PR - not sure if that is in your flow though - won't affect local Zarf packages)

Putting this on my todo list for today.

I ran one test so far that looked good. Will do two more tomorrow.

I think we are good!

Racer159 · 2023-05-19T17:23:37Z

Ok going to clean up the PR and then rollback one change we made that didn't fix anything (and at this point only serves to slow down image pushes). Will leave this open though in case this doesn't fully solve the issue - the main thought in the PR is that the multithreaded jobs in crane are either overwhelming the resource limits of the registry pod and/or they are stepping on each other and resulting in a stuck state.

## Description Creating this PR to test the performance impact of setting crane jobs to `1` (which has shown initially effective in resolving the issue of registry push hanging) ## Related Issue Relates to #1568 Fixes #1656 Fixes #1734 ## Type of change - [X] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Other (security config, docs update, etc) ## Checklist before merging - [X] Test, docs, adr added or updated as needed - [X] [Contributor Guide Steps](https://github.com/defenseunicorns/zarf/blob/main/CONTRIBUTING.md#developer-workflow) followed --------- Co-authored-by: Cole Winberry <86802655+mike-winberry@users.noreply.github.com>

Racer159 · 2023-05-23T21:06:10Z

Moving this back a milestone to give the fixes in v0.27.0 some time to bake with the community to see if this is truly fixed.

Racer159 · 2023-06-05T18:10:25Z

Closing as fixed for now - we can reopen if it presents itself again

ranimbal · 2023-10-24T17:58:54Z

We are still seeing this issue in zarf version 0.29.2. We have a multi-node cluster on AWS EC2 running RKE2 v1.26.9+rke2r1. Our package size is about 2.9G, the "zarf package deploy..." command stalls and hangs about 80% of the time or so. A retry usually works fine.

Here are a few things that we noticed after some extensive testing:

this issue is not seen on a single EC2 node RKE2 cluster, it seems to only occur on multi-node clusters.
our zarf docker registry is backed by S3. The issue is always seen in this case, but only if a multi-node cluster.
if we back the registry with the default PVC (instead of S3), the issue is not seen at all. Since data transfer to S3 is slower than to the EBS backed PVC, maybe this extra time causes the problem to appear?
disabling or enabling the zarf docker registry HPA doesn't seem to matter either ways.

Can you please re-open this issue, else let me know if I should create a new issue? Thanks.

dgershman added the possible-bug 🐛 label Apr 6, 2023

YrrepNoj added this to Zarf Project Board Apr 6, 2023

github-project-automation bot moved this to New Requests in Zarf Project Board Apr 6, 2023

dgershman changed the title ~~Intermittent Hangs on Registry PUsh~~ Intermittent Hangs on Registry Push Apr 6, 2023

dgershman mentioned this issue Apr 6, 2023

Adding additional details for debugging about registry pushes #1558

Merged

5 tasks

Racer159 mentioned this issue Apr 11, 2023

Switch to per-image tunnels, CRC on tags, and progressbar for image pushes #1590

Merged

5 tasks

Racer159 self-assigned this Apr 18, 2023

Racer159 added this to the v0.26.1 milestone Apr 18, 2023

Racer159 modified the milestones: v0.26 (m2), v0.26 (m3) Apr 20, 2023

Racer159 modified the milestones: v0.26 (m3-5.2), v0.27 (m1-5.9) May 2, 2023

This was referenced May 2, 2023

Intermitent Zarf (crane) image push failure (RKE2 + Longhorn) #1433

Closed

Pushing Images to zarf registry hangs forever #1218

Closed

Racer159 modified the milestones: v0.27 (m1-5.9), v0.27 (m2-5.16) May 9, 2023

Racer159 modified the milestones: v0.27 (m1-5.16), v0.27 (m2-5.23) May 16, 2023

Racer159 mentioned this issue May 17, 2023

1568 test upgrade crane #1721

Merged

5 tasks

Racer159 modified the milestones: v0.27 (m2-5.23), v0.27 (m3-5.30) May 23, 2023

Racer159 closed this as completed Jun 5, 2023

ranimbal mentioned this issue Oct 26, 2023

Intermittent Hangs at crane.Push() on Registry Push #2104

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intermittent Hangs on Registry Push #1568

Intermittent Hangs on Registry Push #1568

dgershman commented Apr 6, 2023 •

edited

Loading

dgershman commented Apr 11, 2023 •

edited

Loading

dgershman commented Apr 15, 2023

Racer159 commented Apr 25, 2023

dgershman commented Apr 26, 2023

Racer159 commented Apr 27, 2023

Racer159 commented May 17, 2023

dgershman commented May 17, 2023

dgershman commented May 18, 2023

dgershman commented May 18, 2023

Racer159 commented May 19, 2023 •

edited

Loading

Racer159 commented May 23, 2023

Racer159 commented Jun 5, 2023

ranimbal commented Oct 24, 2023 •

edited

Loading

Intermittent Hangs on Registry Push #1568

Intermittent Hangs on Registry Push #1568

Comments

dgershman commented Apr 6, 2023 • edited Loading

Environment

Steps to reproduce

Expected result

Actual Result

Visual Proof (screenshots, videos, text, etc)

Severity/Priority

Additional Context

dgershman commented Apr 11, 2023 • edited Loading

dgershman commented Apr 15, 2023

Racer159 commented Apr 25, 2023

dgershman commented Apr 26, 2023

Racer159 commented Apr 27, 2023

Racer159 commented May 17, 2023

dgershman commented May 17, 2023

dgershman commented May 18, 2023

dgershman commented May 18, 2023

Racer159 commented May 19, 2023 • edited Loading

Racer159 commented May 23, 2023

Racer159 commented Jun 5, 2023

ranimbal commented Oct 24, 2023 • edited Loading

dgershman commented Apr 6, 2023 •

edited

Loading

dgershman commented Apr 11, 2023 •

edited

Loading

Racer159 commented May 19, 2023 •

edited

Loading

ranimbal commented Oct 24, 2023 •

edited

Loading