Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve test dependencies in parallel #454

Merged
merged 2 commits into from
Mar 27, 2023

Conversation

basil
Copy link
Member

@basil basil commented Mar 26, 2023

I have long been annoyed at the fact that resolve-test-dependencies downloads one dependency at a time. This is made worse by my slow Internet connection. This PR makes it work in parallel, the same way regular Maven dependency resolution works.

Unfortunately maven-artifact-transfer (the nice high-level API we were using) does not support this use case. It was a very thin layer over Aether, anyway. Here we use Aether directly, which can do what we want. There is precedent for invoking Aether directly in other Maven plugins, such as maven-enforcer-plugin.

We do at most two "passes": one regular pass, and one (possibly skipped) fallback pass to retain the functionality of JENKINS-62658. Each pass resolves multiple artifacts in parallel for maximum performance.

This entails a bit of code duplication, but I think the performance improvement is worth it.

basil added a commit to basil/bom that referenced this pull request Mar 26, 2023
@basil
Copy link
Member Author

basil commented Mar 26, 2023

In jenkinsci/bom, before:

[2023-03-26T04:40:45.726Z] [INFO] --- hpi:3.39:resolve-test-dependencies (default-resolve-test-dependencies) @ sample ---
[2023-03-26T04:41:31.413Z] [INFO] 

After:

[2023-03-26T19:00:54.353Z] [INFO] --- hpi:3.41-rc1397.a_3fb_b_d24b_e6b_:resolve-test-dependencies (default-resolve-test-dependencies) @ sample ---
[2023-03-26T19:00:57.456Z] [INFO] 

46 seconds to 3 seconds (> 1000% performance improvement)

I also tested this manually in text-finder by adding the latest version of all the AWS SDK plugins to my POM in test scope. Before this PR they were painfully downloading one-by-one. After this PR they were downloading in parallel.

Copy link
Contributor

@MarkEWaite MarkEWaite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Comment for the debug logging is optional

…jo.java

Co-authored-by: Mark Waite <mark.earl.waite@gmail.com>
@basil basil merged commit 2b23516 into jenkinsci:master Mar 27, 2023
@basil basil deleted the test-dependencies branch March 27, 2023 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants