Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Flatten update lookup tasks #27369

Merged
merged 40 commits into from
Aug 19, 2024

Conversation

zharinov
Copy link
Collaborator

@zharinov zharinov commented Feb 16, 2024

Changes

  • Collect all dep fetches for all package files for all managers into the single array
  • Process it concurrently (with the limit of simulataneous runs)
  • Adjust error handling to behave similarly to current functionality

Context

Documentation (please check one with an [x])

  • I have updated the documentation, or
  • No documentation update is required

How I've tested my work (please select one)

I have verified these changes via:

  • Code inspection only, or
  • Newly added/modified unit tests, or
  • No unit tests but ran on a real repository, or
  • Both unit tests + ran on a real repository

@zharinov zharinov marked this pull request as draft February 16, 2024 20:32
@zharinov
Copy link
Collaborator Author

This one should be tested well

@zharinov zharinov requested review from rarkins and viceice February 16, 2024 20:32
@zharinov
Copy link
Collaborator Author

I haven't formalized my thoughts yet, but here we may want the single queue to be cancelable. Because, even we get ExternalHostError, fetching continues to some degree for both current and this new implementations.

@zharinov
Copy link
Collaborator Author

@rarkins Could you provide a hint of how lookup error handling should work? Given we have flat list of lookup tasks performed concurrently, it's not clear how to deal with errors:

  • Should some errors stop processing completely? (i.e., for all hosts and for all managers)
  • Should some errors stop further requests to particular host, while others are processed as usual?
  • Should we log/collect other types of errors?

@zharinov zharinov marked this pull request as ready for review February 19, 2024 15:12
@zharinov zharinov marked this pull request as draft February 19, 2024 15:16
@zharinov zharinov marked this pull request as ready for review February 19, 2024 15:40
@zharinov zharinov changed the title refactor: Flatten update lookup tasks feat: Flatten update lookup tasks Feb 19, 2024
@zharinov zharinov changed the title feat: Flatten update lookup tasks refactor: Flatten update lookup tasks Feb 19, 2024
@zharinov zharinov requested a review from viceice March 6, 2024 13:49
Copy link
Collaborator

@rarkins rarkins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs rate limiting first

viceice
viceice previously approved these changes Apr 12, 2024
Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️ needs extensive testing

@rarkins
Copy link
Collaborator

rarkins commented May 7, 2024

@zharinov needs deconflicting

Copy link
Member

@viceice viceice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤷‍♂️

@viceice viceice added this pull request to the merge queue Aug 19, 2024
Merged via the queue into renovatebot:main with commit dc43ad4 Aug 19, 2024
38 checks passed
@viceice viceice deleted the refactor/flatten-lookups branch August 19, 2024 14:24
@renovate-release
Copy link
Collaborator

🎉 This issue has been resolved in version 38.41.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

viceice added a commit that referenced this pull request Aug 21, 2024
@viceice viceice added the regression Issue about a regression bug, or the PR caused it label Aug 21, 2024
@viceice
Copy link
Member

viceice commented Aug 21, 2024

This causes OOM with repo cache and regex managers, so it's getting reverted.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
regression Issue about a regression bug, or the PR caused it
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants