Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Remote Segments] Segment merges creates high backlog of segments to upload to remote store #7474

Open
ashking94 opened this issue May 9, 2023 · 0 comments
Open

[Remote Segments] Segment merges creates high backlog of segments to upload to remote store #7474

ashking94 opened this issue May 9, 2023 · 0 comments
Labels
enhancement Enhancement or improvement to existing feature or request Performance This is for any performance related enhancements or bugs Storage:Durability Issues and PRs related to the durability framework v2.8.0 'Issues and PRs related to version v2.8.0'

Comments

@ashking94
Copy link
Member

ashking94 commented May 9, 2023 •
edited
Loading

Is your feature request related to a problem? Please describe.
After after every refresh today, we upload the newer segment files to the remote store. In case a segment merge was to happen, it will trigger a refresh and a sudden surge of backlog gets created for upload to remote store. As of now, the segments upload from RemoteStoreRefreshListener is serial and takes as much as 300s to finish. What this means is that there would not be any refreshes that can happen until the upload finishes - leading to freshness delay on remote store and subsequently replicas.

Describe the solution you'd like
Parallelise segment file uploads in RemoteStoreRefreshListener.
@ashking94 ashking94 added enhancement Enhancement or improvement to existing feature or request Storage:Durability Issues and PRs related to the durability framework Performance This is for any performance related enhancements or bugs v2.8.0 'Issues and PRs related to version v2.8.0' labels May 9, 2023
@sachinpkale sachinpkale mentioned this issue May 9, 2023
Closed
13 tasks
@ashking94 ashking94 mentioned this issue May 9, 2023
Closed
@sachinpkale sachinpkale mentioned this issue May 9, 2023
Closed
@sachinpkale sachinpkale mentioned this issue Sep 6, 2023
Closed
31 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Enhancement or improvement to existing feature or request Performance This is for any performance related enhancements or bugs Storage:Durability Issues and PRs related to the durability framework v2.8.0 'Issues and PRs related to version v2.8.0'
Projects
OpenSearch Lucene & Core Performance Tracking
Status: Open
Storage Project Board
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ashking94