fix: transient errors for s3 artifacts: Fixes #7349

[RoelvandenBerg]

When connection problems occur while loading artifacts, the workflow init step exits and the workflow fails. This is a regression.
We propose a fix in this PR for issue: #7349.
Fixes #7349

In this PR we add an extra check for Transient errors every time the S3 Transient errors are checked.

Don't bother creating a PR until you've done this:

• Run make pre-commit -B to fix codegen, lint, and commit message problems.

Create your PR as a draft.

• Your PR needs to pass the required checks before it can be approved. If the check is not required (e.g. E2E tests) it
  does not need to pass.
• Once required tests have passed, you can make it "Ready for review".
• Say how how you tested your changes. If you changed the UI, attach screenshots.

Tips:

• If changes were requested, and you've made them, then dismiss the review to get it looked at again.
• Add you organization to USERS.md if you like.
• You can ask for help!

[simster7]

@terrytangyuan Can you take a look here?

[alexec]

lint build is failing where running ./hack/check-logging.sh. Can you run this script and check output? It's meant to make sure you do not add Infof/Errorf logging statements.

[RoelvandenBerg]

I found a problem in my local test setup. That is fixed now. Sorry for the many fix: codestyle commits. I'm going to squash the previous (codestyle) commits run the tests again and force push them again.

[alexec]

@terrytangyuan would you be willing to own reviewing this?

passed to terry

[RoelvandenBerg]

In case you want me to rebase onto master and force-push this to deal with the lint error, just let me know.

[terrytangyuan]

Yes please fix the lint errors

[ado120]

Hey all,

I have an issue currently with a s3 backend for workflow artifacts. If I run a workflow I'm able to view the pod logs for a few minutes before I get a 500 Internal Server Error. Is this related? Using Argo Workflows 3.2.4