ci.jenkins.io fails to delete stashed artifacts with Access Denied

[basil]

Service(s)

ci.jenkins.io

Summary

I see a lot of log spam on ci.jenkins.io of this form:

2023-06-29 18:54:48.222+0000 [id=10897] WARNING j.m.BackgroundGlobalBuildDiscarder#lambda$processJob$0: An exception occurred when executing Project Build Discarder
Also:   org.jclouds.aws.AWSResponseException: request DELETE https://ci-jenkins-io-artifacts.s3-us-east-2.amazonaws.com/ci.jenkins.io/Core/jenkins/master/5062/stashes/launchable-session-linux-jdk11.txt.tgz HTTP/1.1 failed with code 403, error: AWSError{requestId='<redacted>', requestToken='<redacted>', code='AccessDenied', message='Access Denied', context='{HostId=<redacted>}'}
                at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:76)
        Caused: org.jclouds.rest.AuthorizationException: Access Denied
                at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:123)
                at org.jclouds.s3.handlers.ParseS3ErrorFromXmlContent.refineException(ParseS3ErrorFromXmlContent.java:97)
                at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:90)
                at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
                at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
                at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
                at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
                at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
                at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
                at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
                at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
                at com.sun.proxy.$Proxy832.deleteObject(Unknown Source)
                at org.jclouds.s3.blobstore.S3BlobStore.removeBlob(S3BlobStore.java:335)
                at io.jenkins.plugins.artifact_manager_jclouds.JCloudsArtifactManager.clearAllStashes(JCloudsArtifactManager.java:349)
        Caused: java.io.IOException
                at io.jenkins.plugins.artifact_manager_jclouds.JCloudsArtifactManager.clearAllStashes(JCloudsArtifactManager.java:353)
                at org.jenkinsci.plugins.workflow.flow.StashManager.clearAll(StashManager.java:183)
                at org.jenkinsci.plugins.workflow.job.WorkflowRun.deleteArtifacts(WorkflowRun.java:687)
                at hudson.tasks.LogRotator.perform(LogRotator.java:196)

This seems worrisome to me, because if artifacts are not being deleted, then we are probably leaking them but still paying for them.

Reproduction steps

Run a jenkinsci/bom build and watch the Jenkins service log on ci.jenkins.io.

[dduportal]

Thanks for reporting!

It was already in the task list of "things to fix on ci.jenkins.io caught while working on #3535": better with an issue ❤️

---

• I remember that @jglick recommended to set up a cleanup process outside the artifact-manager-s3 plugin. This error messages is an example: I guess the permissions are not set up properly. I have mixed feelings:

  • Letting the plugin manage the deletion ensures that the builds rotations results always map to the stored artifacts (if a build is kept above the rotation threshold for instance, then its artifact are kept as well).
  • But it means ci.jenkins.io as permission to delete and it adds more steps that can be read as failure, additionnaly it adds pressure on the thread/HTTP client in charge of adding/removing artifacts.

• I'll work on this as soon as javac vanished from docker-highmem machines #3641 is finished

[dduportal]

Update:

• This deletion error prevent the Build Discarder. Caught while working in [ci.jenkins.io] Use a new VM instance type #3535
• There should be no problem to only disable the DELETE and then set up an external garbage collector

[dduportal]

S3 Storage cost for the month of June is steadily growing: 0.22$ for the 1st of June, to 0.37$ the 30th of June.

It's less than 1$ daily: not a damage to our credits, but a garbage collector is needed to avoid bad surprises in 1 or 2 years.

[lemeurherve]

Stashes deletion by the artifact-manager-s3 plugin disabled, after ci.jenkins.io controller restart buil are correctly discarded, we saw an immediate reduction of 6GB on its storage.

Keeping this issue open so we can either allow the plugin to delete items on S3, or either put in place a cleanup mechanism outside ci.jenkins.io

[jglick]

Not sure if I mentioned this elsewhere, but it is deliberate that stashes and artifacts are in distinct top-level folders, so that you can more easily apply distinct collection policies in S3.

[lemeurherve]

An issue has been created for the garbage collector: #3663

Closing this issue as ci.jenkins.io error is resolved.