Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Searchable Snapshots] Downloads should block only on downloads to the exact same key #7031

Closed
andrross opened this issue Apr 6, 2023 · 0 comments · Fixed by #7129
Closed

[Searchable Snapshots] Downloads should block only on downloads to the exact same key #7031

andrross opened this issue Apr 6, 2023 · 0 comments · Fixed by #7129
Labels
bug Something isn't working distributed framework

Comments

@andrross
Copy link
Member

andrross commented Apr 6, 2023

I opened a PR (#7015) to change this behavior, but the approach in that initial revision isn't quite right. After discussion, the idea is to do the following:

...the cached object itself should not be an IndexInput at all, but rather something capable of creating (or returning a previously created) IndexInput. It would need to know the length of the file (for the cache weigher function to work) and then provide a thread-safe mechanism for getting/creating the IndexInput (perhaps using a CompletableFuture). The signature changes might have a bit of a ripple effect but the overall change shouldn't be too complex.

I'm opening this issue for tracking purposes.
@andrross andrross added bug Something isn't working untriaged labels Apr 6, 2023
@andrross andrross mentioned this issue Apr 6, 2023
Closed
6 tasks
andrross added a commit to andrross/OpenSearch that referenced this issue Apr 11, 2023
@andrross
Download block after adding cache entry
60da291 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
andrross added a commit to andrross/OpenSearch that referenced this issue Apr 11, 2023
@andrross
Download block after adding cache entry
8d55b39 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
andrross added a commit to andrross/OpenSearch that referenced this issue Apr 12, 2023
@andrross
Download block after adding cache entry
4cbe65e 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
@andrross andrross mentioned this issue Apr 12, 2023
Merged
6 tasks
andrross added a commit to andrross/OpenSearch that referenced this issue Apr 12, 2023
@andrross
Download block after adding cache entry
c6b655c 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
andrross added a commit to andrross/OpenSearch that referenced this issue Apr 12, 2023
@andrross
Download block after adding cache entry
a2e31e2 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
andrross added a commit to andrross/OpenSearch that referenced this issue Apr 13, 2023
@andrross
Download block after adding cache entry
816afdd 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
andrross added a commit that referenced this issue Apr 13, 2023
@andrross
Download block after adding cache entry (#7129)
a9b4c45 
As detailed in #7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves #7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
@github-project-automation github-project-automation bot moved this from Todo to Done in Searchable Snapshots Apr 13, 2023
opensearch-trigger-bot bot pushed a commit that referenced this issue Apr 13, 2023
@github-actions
Download block after adding cache entry (#7129)
f1c4a44 
As detailed in #7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves #7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
(cherry picked from commit a9b4c45)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kotwanikunal pushed a commit that referenced this issue Apr 13, 2023
@opensearch-trigger-bot @github-actions
Download block after adding cache entry (#7129) (#7148)
dfc4258 
As detailed in #7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves #7031


(cherry picked from commit a9b4c45)

Signed-off-by: Andrew Ross <andrross@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
austintlee pushed a commit to austintlee/OpenSearch that referenced this issue Apr 28, 2023
@andrross @austintlee
Download block after adding cache entry (opensearch-project#7129)
e1a2aad 
As detailed in opensearch-project#7031, we should not guard the download operations with
the cache lock as it is too coarse-grained. This refactoring moves the
long running operation outside of the cache operations.

Resolves opensearch-project#7031

Signed-off-by: Andrew Ross <andrross@amazon.com>
@andrross andrross mentioned this issue Apr 10, 2024
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working distributed framework
Projects
Searchable Snapshots
Status: Done
Milestone
No milestone
2 participants
@andrross @anasalkouz