Fix flaky test BulkIntegrationIT.testDeleteIndexWhileIndexing

[Poojita-Raj]

Signed-off-by: Poojita Raj poojiraj@amazon.com

Description

The test BulkIntegrationIT.testDeleteIndexWhileIndexing is failing intermittently whenever a thread that's indexing documents does not terminate in the timeout interval provided by the join method and is still alive failing an assertion (seen in the stacktrace of the linked issue).

9 failures were seen in the last 2 months: (3607,3757,3789,3839,4952,6624,6635,6723,6979)

Issues Resolved

Resolves #5162

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Poojita-Raj]

There were a couple of approaches that could have been taken with this failing test -

1. Increasing the timeout period for the thread join call to provide enough time for the thread to terminate. -> The number of failures did not decrease on increasing this value - roughly the same 1 or 2 failures for 100 runs took place.
2. Setting the timeout period for the join call to infinity just made the test hang indefinitely.
3. Catching the thread that was still alive after the join call and interrupting it. -> Even after catching the IllegalStateException on the thread, the test would still fail due to the failure of AssertRequestsFinished method (this is in OpenSearchIntegTestCase's afterTest methods called on cleanup) due to the leftover in flight requests to the node.

The purpose of the test was to determine that the index deletion took place even while a simultaneous thread was actively indexing documents into it. Therefore, the change to limit it to 1 indexing thread (instead of a range of 1 to 4) solved the flakiness while preserving the functionality of the test.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/7774/
• CommitID: b6f841e

[codecov-commenter]

Codecov Report

Merging #5491 (b6f841e) into main (2416d37) will decrease coverage by 0.01%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main    #5491      +/-   ##
============================================
- Coverage     71.05%   71.04%   -0.02%     
+ Complexity    58285    58243      -42     
============================================
  Files          4717     4717              
  Lines        277669   277669              
  Branches      40194    40194              
============================================
- Hits         197300   197259      -41     
- Misses        64245    64269      +24     
- Partials      16124    16141      +17     

Impacted Files	Coverage Δ
.../opensearch/search/profile/ProfileShardResult.java	28.57% <0.00%> (-60.72%)	⬇️
...ch/transport/ReceiveTimeoutTransportException.java	50.00% <0.00%> (-50.00%)	⬇️
...ile/aggregation/AggregationProfileShardResult.java	55.55% <0.00%> (-40.75%)	⬇️
...h/action/ingest/SimulateDocumentVerboseResult.java	60.71% <0.00%> (-39.29%)	⬇️
...ava/org/opensearch/search/profile/NetworkTime.java	31.57% <0.00%> (-36.85%)	⬇️
...n/admin/indices/readonly/AddIndexBlockRequest.java	35.71% <0.00%> (-35.72%)	⬇️
...ion/admin/cluster/node/info/PluginsAndModules.java	53.12% <0.00%> (-34.38%)	⬇️
.../search/aggregations/pipeline/HoltLinearModel.java	20.33% <0.00%> (-32.21%)	⬇️
.../search/profile/query/QueryProfileShardResult.java	68.33% <0.00%> (-28.34%)	⬇️
...rc/main/java/org/opensearch/ingest/IngestInfo.java	51.72% <0.00%> (-27.59%)	⬇️
... and 468 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[dreamer-89]

@Poojita-Raj : Thanks for this change. 👍

Do you mind explaining why failure was happening and making this change fixes that ?

[Poojita-Raj]

@Poojita-Raj : Thanks for this change. 👍

Do you mind explaining why failure was happening and making this change fixes that ?

Sure @dreamer-89,

In the original code, we can see that multiple threads are created to run the block of code that indexes documents into the index. While this indexing is actively taking place, the main thread is deleting the index. All the threads indexing documents have some shared objects, namely, the AtomicInteger docID and the AtomicBoolean stopped.

In the case that we have multiple threads indexing, very occasionally we run into the situation where one thread is left in a WAITING state after it does an Object.wait() on a shared object. It is now waiting for a notify() or a notifyAll() call from the other threads that never takes place since they are now terminated and the waiting thread is left in a waiting state indefinitely, thus failing the assertFalse(thread.isAlive()) assertion.

However, when we only spin up a single thread to index while deleting documents, there is no contention for the shared objects so the thread always terminates, fixing the flakiness of the test. Since, all we want to test is whether we can delete the index while actively indexing documents - one thread suffices for this test.

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-1.x 1.x
# Navigate to the new working tree
pushd ../.worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-5491-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 7108ee5f703a1566efaa55b21d4d10397c38de91
# Push it to GitHub
git push --set-upstream origin backport/backport-5491-to-1.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-5491-to-1.x.