Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Test RemoteFSTranslogTests.testMetadataFileDeletion is flaky #9605

Closed
mch2 opened this issue Aug 29, 2023 · 3 comments · Fixed by #11899
Closed

[BUG] Test RemoteFSTranslogTests.testMetadataFileDeletion is flaky #9605

mch2 opened this issue Aug 29, 2023 · 3 comments · Fixed by #11899
Assignees
@ashking94
Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Storage:Remote

Comments

@mch2
Copy link
Member

mch2 commented Aug 29, 2023

#9595 (comment)
@mch2 mch2 added bug Something isn't working untriaged flaky-test Random test failure that succeeds on second run labels Aug 29, 2023
@kotwanikunal kotwanikunal mentioned this issue Sep 23, 2023
Merged
@BhumikaSaini-Amazon BhumikaSaini-Amazon mentioned this issue Oct 4, 2023
Merged
6 tasks
@mch2 mch2 mentioned this issue Oct 25, 2023
Merged
8 tasks
@mch2
Copy link
Member Author

mch2 commented Oct 25, 2023

@ashking94 Can you pls take a look at this one.

@ashking94
Copy link
Member

Running this test on loop to reproduce this issue and accordingly decide the next step.

@kotwanikunal kotwanikunal mentioned this issue Jan 8, 2024
Closed
@reta reta mentioned this issue Jan 10, 2024
Closed
ashking94 added a commit to ashking94/OpenSearch that referenced this issue Jan 16, 2024
@ashking94
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test opensea…
a63fa6a 
…rch-project#9605

Signed-off-by: Ashish Singh <ssashish@amazon.com>
@ashking94
Copy link
Member

ashking94 commented Jan 16, 2024 •
edited
Loading

I have been able to root cause the flakiness in this test on account of translog.trimUnreferencedReaders() being no-op when called due to previous deletion still ongoing and holding the permits. The current run of translog.trimUnreferencedReaders() on checking for permits sees that the required permits are not available and skips the deletion. To elaborate more on generation deletion, the approach is to acquire the permits, trigger async deletion and post successful deletion, release the permits.

@ashking94 ashking94 mentioned this issue Jan 16, 2024
Merged
8 tasks
@bowenlan-amzn bowenlan-amzn mentioned this issue Jan 17, 2024
Merged
8 tasks
sachinpkale pushed a commit that referenced this issue Jan 18, 2024
@ashking94
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test #9605 (#…
904c9a9 
…11899)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
opensearch-trigger-bot bot pushed a commit that referenced this issue Jan 18, 2024
@github-actions
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test #9605 (#…
0de6f99 
…11899)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
(cherry picked from commit 904c9a9)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kotwanikunal pushed a commit that referenced this issue Jan 24, 2024
@opensearch-trigger-bot @github-actions
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test #9605 (#…
fab16d7 
…11899) (#11915)

(cherry picked from commit 904c9a9)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
peteralfonsi pushed a commit to peteralfonsi/OpenSearch that referenced this issue Mar 1, 2024
@ashking94
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test opensea…
604886f 
…rch-project#9605 (opensearch-project#11899)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
rayshrey pushed a commit to rayshrey/OpenSearch that referenced this issue Mar 18, 2024
@ashking94 @rayshrey
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test opensea…
a81a5de 
…rch-project#9605 (opensearch-project#11899)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this issue Apr 25, 2024
@ashking94 @shiv0408
Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test opensea…
96d7a1a 
…rch-project#9605 (opensearch-project#11899)

Signed-off-by: Ashish Singh <ssashish@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ashking94 ashking94

Labels
bug Something isn't working flaky-test Random test failure that succeeds on second run Storage:Remote
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix flaky RemoteFSTranslogTests testMetadataFileDeletion test #9605 ashking94/OpenSearch
4 participants
@mch2 @peternied @ashking94 @anasalkouz