Remote repository for repo.karuslabs.com

[basil]

Service(s)

Artifactory

Summary

Context

The Stapler and lib-annotation-indexer test suites originally used the Hickory library. That library has not been maintained in approximately a decade and no longer supports recent versions of Java, so in jenkinsci/stapler#394 and jenkinsci/lib-annotation-indexer#50 we migrated from Hickory to Elementary, a recently-developed annotation processor testing framework based on JUnit 5 and Java 11.

Problem

In order to consume Elementary, the Stapler and lib-annotation-indexer builds have the following repository (per the README):

<repository>
  <id>elementary-releases</id>
  <url>https://repo.karuslabs.com/repository/elementary-releases/</url>
</repository>

This repository is less likely to have the redundancy and reliability of Maven Central. If, for some reason, it were to go down either temporarily or permanently (and my experience is that this is highly likely for smaller repositories), then we would no longer be able to build and test Stapler and lib-annotation-indexer, two Jenkins core components. Of course, we could remove Elementary and the corresponding tests, but this would decrease test coverage.

Proposed Solution

Use Artifactory's "remote repository" feature to create a local mirror of this repository. That way, we insulate ourselves from potential failures on (or disappearance of) karuslabs.com by maintaining a local cache of the artifacts.

Note

I do not feel strongly about this request, and I am not sure if there are other considerations from the infrastructure team's perspective that would make this request undesirable or burdensome. Feel free to close this ticket if you do not feel that it is a good idea.

Reproduction steps

Block connections to repo.karuslabs.com in your firewall, then try to build Stapler without the benefit of a Maven cache. The Stapler build should fail to fetch the Elementary library.

[daniel-beck]

The entire repo is really tiny (fits on a handful of floppy disks), so

recent topic of traffic and storage

I mentioned in #3108 (comment) doesn't matter.

+1 from me.

[jglick]

Seems fine, though have you asked the maintainers to just publish to Central? (I did not see any issue in their repo to that effect.)

[basil]

have you asked the maintainers to just publish to Central?

I have not.

[basil]

Why is this in the "not directly actionable by infra team" milestone? There is a clear action, described above in the Proposed Solution section of the ticket.

[timja]

Possibly because no one in the infra team has management access to artifactory as far as I'm aware, (although should be clarified when adding to that milestone on why)

[basil]

If the infrastructure team does not have management access to Artifactory, please move this ticket to the party that does have management access.

[timja]

@daniel-beck and @Wadeck I believe

[daniel-beck]

I made @dduportal an Artifactory admin several weeks ago.

[jglick]

IIUC the proposal here is that https://repo.jenkins-ci.org/public/ would mirror these artifacts, so that we could drop the explicit <repository> from Stapler? Still means that anyone building Jenkins stuff using their own mirror based on Central + https://repo.jenkins-ci.org/releases/ would need to take manual action to add a new delegate. If the original developers of this library refuse to use Central for some reason, and assuming that our dep on Elementary rarely changes (there seem to have only ever been four releases), we could alternately just mvn deploy:deploy-file the lib to releases without any dep on the infra team and without introducing new requirements for anyone with their own mirror. Seems to be a different case than https://shibboleth.net/downloads/java-opensaml/ which has regular releases and is part of (security-critical!) production code so we want to be sure to get updates.

[dduportal]

The infra-team missed this issue during the last meeting, hence why it was "rotting" in the not-directly-actionable-by-infra-team milestone, sorry for that.

I confirm that @daniel-beck gave me admin access to repo.jenkins-ci.org so we can work on this.

The issue has been moved in the infra-team-sync-next mileston which mean we'll evaluate it during our next weekly team meeting (priority and team bandwidth).

[basil]

Still means that anyone building Jenkins stuff using their own mirror based on Central + https://repo.jenkins-ci.org/releases/ would need to take manual action to add a new delegate.

Gee, I wonder who might be doing that. But yes, this would be a breaking change that "they" would need to adapt to.

Using mvn deploy:deploy-file to deploy the library to releases introduces long-term technical debt that I would rather not take on. Whether security-critical or not, we want to get updates for all dependencies, so this seems like a similar case to java-opensaml(or any dependency of ours that is not on Maven Central).

[basil]

IIUC the proposal here is that https://repo.jenkins-ci.org/public/ would mirror these artifacts, so that we could drop the explicit <repository> from Stapler?

Not necessarily. I was intentionally vague about which remote repository to allow for multiple solutions to be discussed. Indeed, a solution that uses the public remote repository and drops the explicit <repository> from Stapler would be simplest, but it would break "anyone building Jenkins stuff using their own mirror based on Central + https://repo.jenkins-ci.org/releases/", though anyone doing that would appear to have made a faulty assumption that public is equivalent to Central (doesn't the name merely imply that it is to be used for publicly available things?). An alternative would be to create a new remote repository for non-Central artifacts and switch the <repository> in Stapler from repo.karuslabs.com to the new remote repository. This would accommodate "them" at the cost of additional complexity.

[jglick]

👀 Pante/elementary#153

[dduportal]

A new "remote repository" (e.g. a maven mirror) has been created: https://repo.jenkins-ci.org/artifactory/elementary-releases/ .

This repository is available through the public repository: you should be able to build the projects without the additional repository section.

Please note that I did not have the time to test it myself: I'll probably do it in the upcoming days. It means that I'm not sure if it works but I wanted to let you know that it might.

The issue will be updated once we'll have tested and confirmed it works.

[lemeurherve]

I've tried a stapler build locally without elementary-releases in a Maven settings reproducing the one used by ACP but it failed:

[ERROR] Failed to execute goal on project stapler: Could not resolve dependencies for project org.kohsuke.stapler:stapler:jar:999999-SNAPSHOT: The following artifacts could not be resolved: com.karuslabs:elementary:jar:1.1.3 (absent): Could not find artifact com.karuslabs:elementary:jar:1.1.3 in do-proxy (https://repo.do.jenkins.io/public/) -> [Help 1]

[dduportal]

I've tried a stapler build locally without elementary-releases in a Maven settings reproducing the one used by ACP but it failed:

[ERROR] Failed to execute goal on project stapler: Could not resolve dependencies for project org.kohsuke.stapler:stapler:jar:999999-SNAPSHOT: The following artifacts could not be resolved: com.karuslabs:elementary:jar:1.1.3 (absent): Could not find artifact com.karuslabs:elementary:jar:1.1.3 in do-proxy (https://repo.do.jenkins.io/public/) -> [Help 1]

I can reproduce locally. Checking the status

[dduportal]

Current status: https://repo.jenkins-ci.org/artifactory/elementary-releases/com/karuslabs/elementary/ shows folders, which are empty, and no maven metadata.

The synchronisation most probably failed. Checking in JFrog UI.

[dduportal]

I don't understand why Artifactor is unable to mirror the remote repository:

• If I use the https://repo.karuslabs.com/repository/elementary-releases/ URL (same as the pom.xml) then the "Test Repository" button shows an HTTP/404 failure, and the mirror is empty
• If I use the "HTML" browsing URL (https://repo.karuslabs.com/service/rest/repository/browse/elementary-releases/) then the directories are empty (I guess because this endpoint does not show all the maven meta files

I guess this sonatype instance is refusing requests from artifactory or there might be a specific setting that I'm not aware of.

I'm out of options and knowledge: the available logs in our artifactory instance only shows either the 404 of that the syncronisation of element-release is complete (but with no artefacts)

[jglick]

Pante/elementary#153 (comment)

[timja]

@dduportal can you remove the remote registry please? it's no longer needed

[dduportal]

@dduportal can you remove the remote registry please? it's no longer needed

Done, remote repo removed from repo.jenkins-ci.org!

I see that https://repo.jenkins-ci.org/artifactory/public/com/karuslabs/elementary/ is now available in /public but it only shows the 2.0.0: I guess stapler need an update in its dependencies?

[dduportal]

Pante/elementary#153 (comment)

Thanks @Vlatombe @jglick

[timja]

I updated stapler and removed the custom repository in jenkinsci/stapler#468

to use the new maven central version

[lemeurherve]

Corresponding exception removal from artifact caching proxy settings: jenkins-infra/jenkins-infra#2916

[lemeurherve]

I updated stapler and removed the custom repository in jenkinsci/stapler#468

to use the new maven central version

FTR, @basil did the same for lib-annotation-indexer (the only other repo with a dependency to Elementary) in jenkinsci/lib-annotation-indexer#70 (thanks!)