Unable to download Artifacts from the Archived Workflows UI (v3.4.3)

[pulungw]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I ran a simple artifact-passing workflow that created a text file artifact. When I click on the artifact in the Argo UI on the "Workflow" page, I can see the contents of the artifact come up in the UI. But, after the workflow is archived and deleted, the UI on the "Archived Workflow" page displays an "Internal Server Error", with error="workflows.argoproj.io \"artifact-passing-9rzv7\" not found" message on the argo-server logs.

Note that this happens right after the workflow is deleted from the "Workflows" UI. Before it was deleted, I can see the Artifacts normally even in the "Archived Workflows" UI (When the same workflow appears at the same time on "Workflows" and "Archived Workflows" UI).

Furthermore, on the same "Archived Workflows" page, when I click on the container node -> INPUTS/OUTPUTS -> artifact, Argo is able to successfully retrieve the artifact (with the URL pointing to: https://<argo-server-url>/artifact-files/<username>/archived-workflows/a093e078-097a-4ab6-b51f-3e0b62123265/artifact-passing-9rzv7-2621575856/outputs/hello-art).

I think this might be related with the previous issue -- Argo UI - Cannot View Artifact in Archived Workflows. (ref: #9858)

• Here is a screenshot of the error in the "Archived Workflows" UI:
  

• Here is a screenshot of normal Artifacts viewer in the "Workflows" UI before the workflow was archived and deleted:
  

What you expected to happen?

I expected to be able to view and download the artifact in the Archived Workflows UI, just as I was able to view it before the workflow was archived and deleted. I was able to download Artifacts normally in Argo v3.3.7.

Version

v3.4.3

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          # bind message to the hello-art artifact
          # generated by the generate-artifact step
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["cowsay hello world | tee /tmp/hello_world.txt"]
      resources:
        limits:
          cpu: "100m"
          memory: "100Mi"
    outputs:
      artifacts:
      # generate hello-art artifact from /tmp/hello_world.txt
      # artifacts can be directories as well as files
      - name: hello-art
        path: /tmp/hello_world.txt
        archive:
          none: {}

  - name: print-message
    inputs:
      artifacts:
      # unpack the message input artifact
      # and put it at /tmp/message
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]
      resources:
        limits:
          cpu: "100m"
          memory: "100Mi"

Logs from the workflow controller

workflow-controller logs doesn't show error, here is the error shown in the argo-server logs.

kubectl logs -n argo argo-server-76ddbdd84c-cv5x8

time="2022-11-04T01:59:21.589Z" level=info msg="selected SSO RBAC service account for user" email=<username>@<company> loginServiceAccount=<username> serviceAccount=<username> ssoDelegated=false ssoDelegationAllowed=false subject=2018876
[mysql] 2022/11/04 01:59:21 packets.go:123: closing bad idle connection: EOF
time="2022-11-04T01:59:21.697Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=ListArchivedWorkflowLabelKeys grpc.service=workflowarchive.ArchivedWorkflowService grpc.start_time="2022-11-04T01:59:21Z" grpc.time_ms=204.176 span.kind=server system=grpc
time="2022-11-04T01:59:21.698Z" level=info duration=206.82373ms method=GET path=/api/v1/archived-workflows-label-keys size=632 status=0
time="2022-11-04T01:59:23.793Z" level=info msg="selected SSO RBAC service account for user" email=<username>@<company> loginServiceAccount=<username> serviceAccount=<username> ssoDelegated=false ssoDelegationAllowed=false subject=2018876
time="2022-11-04T01:59:23.793Z" level=info msg="tracking UI usage️️" email=<username>@<company> name=openedArchivedWorkflowDetails subject=2018876
time="2022-11-04T01:59:23.793Z" level=info msg="finished unary call with code OK" grpc.code=OK grpc.method=CollectEvent grpc.service=info.InfoService grpc.start_time="2022-11-04T01:59:23Z" grpc.time_ms=41.052 span.kind=server system=grpc
time="2022-11-04T01:59:23.794Z" level=info duration=42.334383ms method=POST path=/api/v1/tracking/event size=2 status=0
time="2022-11-04T01:59:25.672Z" level=info msg="selected SSO RBAC service account for user" email=<username>@<company> loginServiceAccount=<username> serviceAccount=<username> ssoDelegated=false ssoDelegationAllowed=false subject=2018876
time="2022-11-04T01:59:25.673Z" level=info msg="Get artifact file" artifactName=hello-art namespace=<username> nodeId=artifact-passing-9rzv7-2621575856 workflowName=artifact-passing-9rzv7
time="2022-11-04T01:59:26.072Z" level=error msg="Artifact Server returned internal error" error="workflows.argoproj.io \"artifact-passing-9rzv7\" not found"
time="2022-11-04T01:59:26.072Z" level=info duration=450.346224ms method=GET path=/artifact-files/<username>/workflows/artifact-passing-9rzv7/artifact-passing-9rzv7-2621575856/outputs/hello-art size=22 status=500
time="2022-11-04T01:59:36.444Z" level=info duration="170.626µs" method=GET path=index.html size=473 status=0
time="2022-11-04T01:59:56.444Z" level=info duration="104.362µs" method=GET path=index.html size=473 status=0
time="2022-11-04T02:00:16.445Z" level=info duration="79.225µs" method=GET path=index.html size=473 status=0
time="2022-11-04T02:00:36.445Z" level=info duration="81.39µs" method=GET path=index.html size=473 status=0
time="2022-11-04T02:00:56.443Z" level=info duration="93.15µs" method=GET path=index.html size=473 status=0

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[terrytangyuan]

@tczhao Would you like to take a look?

[tczhao]

Sure.
Give me a few days:
seeing 2 issues so far:

• artifact still undergoes GC with artifactGCStrategy Never
• after workflow get updated with artifactGC, the info should propagate to archivedWf as well

[sarabala1979]

@tczhao Are you going to work on these issues?

[talebzeghmi]

I'm seeing this issue reproduce in v3.5.4 with the same kind of error

time="2022-11-04T01:59:26.072Z" level=error msg="Artifact Server returned internal error" error="workflows.argoproj.io \"artifact-passing-9rzv7\" not found"

how do I reopen this?

[agilgur5]

how do I reopen this?

You don't; it's a new bug in a different version (the previous one was fixed over a year ago), so you'd open a new bug report.
The general practice in OSS repos is that it's probably better to open a new issue than comment on a very old, dated, closed, fixed issue.

I'm seeing this issue reproduce in v3.5.4 with the same kind of error

There was a similar bug in 3.5 reported in #12331, but that was fixed in #12397. It seems like that didn't make it into 3.5.4 -- it is slated for the next patch release: #11997 (comment)