Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

peternied · 2023-11-02T21:33:10Z

Description

During the testSkipDuplicates its possible that not all documents were fully indexed by the time the search with suggestions was created, updating the indexing operations to refresh the index before returning.

As its possible that did not fix the issue, I've added logging around the test case to capture the state when the error occurred that can assist in future troubleshooting.

Related Issues

Resolves [BUG] org.opensearch.search.suggest.CompletionSuggestSearchIT.testSkipDuplicates is flaky #10798

Check List

New functionality includes testing.
- All tests pass
~~New functionality has been documented.~~
- ~~New functionality has javadoc added~~
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

During the testSkipDuplicates its possible that not all documents were fully indexed by the time the search with suggestions was created, updating the indexing operations to refresh the index before returning. As its possible that did not fix the issue, I've added logging around the test case to capture the state when the error occurred that can assist in future troubleshooting.

Signed-off-by: Peter Nied <petern@amazon.com>

github-actions · 2023-11-02T21:42:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/29487/
CommitID: 9f18436
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-11-02T22:09:43Z

Compatibility status:

Checks if related components are compatible with change 5d128ee

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/k-nn.git]

github-actions · 2023-11-02T22:30:55Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteLargeBlob
      1 org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew

URL: https://build.ci.opensearch.org/job/gradle-check/29489/
CommitID: 540b540
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-11-02T22:33:26Z

Codecov Report

Merging #11068 (540b540) into main (63aff16) will decrease coverage by 0.07%.
Report is 5 commits behind head on main.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main   #11068      +/-   ##
============================================
- Coverage     71.29%   71.22%   -0.07%     
- Complexity    58742    58781      +39     
============================================
  Files          4872     4872              
  Lines        276777   276780       +3     
  Branches      40240    40241       +1     
============================================
- Hits         197316   197138     -178     
- Misses        62943    63236     +293     
+ Partials      16518    16406     -112

Files	Coverage Δ
...s/replication/SegmentReplicationTargetService.java	`56.17% <100.00%> (+0.56%)`	⬆️

... and 484 files with indirect coverage changes

peternied · 2023-11-02T23:24:55Z

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

#10303
#6090

...er/src/internalClusterTest/java/org/opensearch/search/suggest/CompletionSuggestSearchIT.java

sohami · 2023-11-03T03:41:56Z

@peternied I guess we don't need to merge the logging change and can be done locally to debug the test issue by running with Repeat annotation. Or is the refresh fixing the test here and logs are added to debug future failure just in case if the issue happens again ?

peternied · 2023-11-03T16:24:10Z

guess we don't need to merge the logging change and can be done locally to debug the test issue by running with Repeat annotation.

@sohami I haven't been able to reproduce the issue locally. I've run ~2000 iterations on this test with and without difference seed / locales. I've tweaked the 'random' elements to be hardcode to attempt to find an issue - no luck. The only theory I have is around the index not being fully current when the search query is issued. I think its of minimal risk to force refreshing on index.

By leaving telemetry in, if this comes back up I should have more precise information about what wasn't working to better root cause it. How does that sound to you?

peternied · 2023-11-03T22:06:50Z

@Poojita-Raj @mch2 @reta @dreamer-89 could I get a review on this change? Added you folks since you reported this test as a failure in the past 2 days.

mch2

just nits so approved

CHANGELOG.md

...er/src/internalClusterTest/java/org/opensearch/search/suggest/CompletionSuggestSearchIT.java

Signed-off-by: Peter Nied <petern@amazon.com>

opensearch-trigger-bot · 2023-11-07T17:28:00Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-11068-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0ba5d5807860a799c76f0fb6e063e8c5bd458492
# Push it to GitHub
git push --set-upstream origin backport/backport-11068-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-11068-to-2.x.

github-actions · 2023-11-07T17:31:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/29676/
CommitID: 5d128ee
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

…sh when indexing its test documents (opensearch-project#11068) During the testSkipDuplicates its possible that not all documents were fully indexed by the time the search with suggestions was created, updating the indexing operations to refresh the index before returning. As its possible that did not fix the issue, I've added logging around the test case to capture the state when the error occurred that can assist in future troubleshooting. Signed-off-by: Peter Nied <petern@amazon.com> (cherry picked from commit 0ba5d58)

…sh when indexing its test documents (#11068) (#11141) During the testSkipDuplicates its possible that not all documents were fully indexed by the time the search with suggestions was created, updating the indexing operations to refresh the index before returning. As its possible that did not fix the issue, I've added logging around the test case to capture the state when the error occurred that can assist in future troubleshooting. Signed-off-by: Peter Nied <petern@amazon.com> (cherry picked from commit 0ba5d58) Signed-off-by: Peter Nied <petern@amazon.com>

…g its test documents (opensearch-project#11068) During the testSkipDuplicates its possible that not all documents were fully indexed by the time the search with suggestions was created, updating the indexing operations to refresh the index before returning. As its possible that did not fix the issue, I've added logging around the test case to capture the state when the error occurred that can assist in future troubleshooting. Signed-off-by: Peter Nied <petern@amazon.com>

…g its test documents (opensearch-project#11068) During the testSkipDuplicates its possible that not all documents were fully indexed by the time the search with suggestions was created, updating the indexing operations to refresh the index before returning. As its possible that did not fix the issue, I've added logging around the test case to capture the state when the error occurred that can assist in future troubleshooting. Signed-off-by: Peter Nied <petern@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

github-actions bot added >test-failure Test failure from CI, local build, etc. bug Something isn't working flaky-test Random test failure that succeeds on second run labels Nov 2, 2023

peternied changed the title ~~Fix SuggestSearch skipDuplicates flaky test~~ SuggestSearch skipDuplicates forces refresh when indexing its test documents Nov 2, 2023

peternied changed the title ~~SuggestSearch skipDuplicates forces refresh when indexing its test documents~~ Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents Nov 2, 2023

Spotless + changelog entry

540b540

Signed-off-by: Peter Nied <petern@amazon.com>

This was referenced Nov 2, 2023

[BUG] AzureBlobContainerRetriesTests.testReadRangeBlobWithRetries flaky test failure #6090

Closed

[BUG] o.o.i.s.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew is flaky #10303

Open

peternied added the backport 2.x Backport to 2.x branch label Nov 2, 2023

peternied mentioned this pull request Nov 2, 2023

[Backport 2.x] Mute flaky concurrent segment search tests pending fixes (#10437) #11050

Merged

8 tasks

jed326 approved these changes Nov 2, 2023

View reviewed changes

...er/src/internalClusterTest/java/org/opensearch/search/suggest/CompletionSuggestSearchIT.java Show resolved Hide resolved

mch2 approved these changes Nov 5, 2023

View reviewed changes

CHANGELOG.md Show resolved Hide resolved

...er/src/internalClusterTest/java/org/opensearch/search/suggest/CompletionSuggestSearchIT.java Show resolved Hide resolved

reta approved these changes Nov 6, 2023

View reviewed changes

Merge branch 'main' into duplication-suggestions-flak

5d128ee

Signed-off-by: Peter Nied <petern@amazon.com>

peternied requested review from adnapibar and tlfeng as code owners November 7, 2023 17:26

peternied merged commit 0ba5d58 into opensearch-project:main Nov 7, 2023
31 of 32 checks passed

peternied deleted the duplication-suggestions-flak branch November 7, 2023 17:27

opensearch-trigger-bot bot added the backport-failed label Nov 7, 2023

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

peternied commented Nov 2, 2023 •

edited

Loading

github-actions bot commented Nov 2, 2023

github-actions bot commented Nov 2, 2023 •

edited

Loading

github-actions bot commented Nov 2, 2023

codecov bot commented Nov 2, 2023

peternied commented Nov 2, 2023

sohami commented Nov 3, 2023

peternied commented Nov 3, 2023

peternied commented Nov 3, 2023 •

edited

Loading

mch2 left a comment

opensearch-trigger-bot bot commented Nov 7, 2023

github-actions bot commented Nov 7, 2023

Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

Fix SuggestSearch.testSkipDuplicates by forceing refresh when indexing its test documents #11068

Conversation

peternied commented Nov 2, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Nov 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Nov 2, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Nov 2, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Nov 2, 2023

Codecov Report

peternied commented Nov 2, 2023

sohami commented Nov 3, 2023

peternied commented Nov 3, 2023

peternied commented Nov 3, 2023 • edited Loading

mch2 left a comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Nov 7, 2023

github-actions bot commented Nov 7, 2023

Gradle Check (Jenkins) Run Completed with:

peternied commented Nov 2, 2023 •

edited

Loading

github-actions bot commented Nov 2, 2023 •

edited

Loading

peternied commented Nov 3, 2023 •

edited

Loading