Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test by preventing duplicate keys in random mapping fields #9184

Merged
merged 2 commits into from
Aug 17, 2023

Conversation

dbwiddis
Copy link
Member

@dbwiddis dbwiddis commented Aug 9, 2023

Description

Although rare, if you throw random alpha strings together often enough, you'll get a collision.

Related Issues

Fixes #6739

Check List

  • New functionality includes testing.
    • All tests pass
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 31m 10s

@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

@dbwiddis
Copy link
Member Author

dbwiddis commented Aug 9, 2023

Annoying when one flaky test fix has a failure from another flaky test. :) #9178

@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 32m 33s

@dbwiddis
Copy link
Member Author

dbwiddis commented Aug 9, 2023

Latest flaky test failure tracked in #1703...

Copy link
Member Author

@dbwiddis dbwiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Poojita-Raj I don't think the suggested changes are needed in this case. The collisions are extremely rare and the occasional decrement is still well within (and barely impacts) the random ranges being generated.

If you really think this needs to be engineered more robustly, I'd suggest adding another utility method in OpenSearchTestCase similar to generateRandomStringArray() that creates unique strings, and then audit all the tests to see if there are other cases like this one.

For flaky tests, could we also mention how many times the test was run to ensure it's no longer flaky for confidence before we merge it in?

I originally closed the flaky test issue because I could calculate the extremely rare probability of a failure, and asked a similar question here: #6739 (comment)

This was the only failure of this test that I could find in the last 2500 runs using the script, and a search of issues on this repo indicates it was the only case reported ever.

Happy to run this N times to prove no reproduction, but nobody has defined what N should be. Given it's only failed once in 12500 test runs and the probability is at least 1 in 20,000 or so (probably more) I think I'd have to test it at least 30,000 times to make sure it's "fixed". I don't think that's necessary but am willing to write a script to do so.

Copy link
Member Author

@dbwiddis dbwiddis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Poojita-Raj looked at it again and decided to just replace for loops with while checking size, to preserve original size. Realized the aliases were already unique because the index was used as part of the name, so backed out that change.

Should be GTG now. I don't think it's necessary to try to reproduce this again as the cause is well understood, the reproducing duplicate keys still occur with the random seed in the linked issue, and the test doesn't fail.

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 24m 46s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@dbwiddis
Copy link
Member Author

Test failure may have been fixed by #9079

Will rebase

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 32m 12s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.action.admin.indices.create.CreateIndexIT.testCreateAndDeleteIndexConcurrently
      1 org.opensearch.action.admin.indices.create.CreateIndexIT.classMethod

@dbwiddis
Copy link
Member Author

Both CreateIndexIT failures tracked in #5425

@dbwiddis dbwiddis requested a review from Poojita-Raj August 15, 2023 15:25
Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Daniel Widdis <widdis@gmail.com>
@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 27s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries
      1 org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testShardAlreadyReplicating

@dbwiddis
Copy link
Member Author

org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries #7934
org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testShardAlreadyReplicating #8928

@dbwiddis
Copy link
Member Author

This has fallen off the "front page" of PR reviews, so bumping for attention. Minor fix, shouldn't be controversial.

@Poojita-Raj (answered your earlier review), @mch2 @andrross @saratvemulapalli @dblock @ryanbogan @owaiskazi19 :-)

@andrross andrross merged commit e1c40b4 into opensearch-project:main Aug 17, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 17, 2023
…9184)

* Prevent duplicate keys in random mapping fields

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Aliases were already unique. Used while loop for indices to keep size

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
(cherry picked from commit e1c40b4)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@dbwiddis dbwiddis deleted the dupe-keys branch August 17, 2023 16:24
dbwiddis pushed a commit that referenced this pull request Aug 17, 2023
…9184) (#9406)

* Prevent duplicate keys in random mapping fields



* Aliases were already unique. Used while loop for indices to keep size



---------


(cherry picked from commit e1c40b4)

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
austintlee pushed a commit to austintlee/OpenSearch that referenced this pull request Aug 25, 2023
…pensearch-project#9184)

* Prevent duplicate keys in random mapping fields

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Aliases were already unique. Used while loop for indices to keep size

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
kkmr pushed a commit to kkmr/OpenSearch that referenced this pull request Aug 28, 2023
…pensearch-project#9184)

* Prevent duplicate keys in random mapping fields

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Aliases were already unique. Used while loop for indices to keep size

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Kiran Reddy <kkreddy@amazon.com>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…pensearch-project#9184)

* Prevent duplicate keys in random mapping fields

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Aliases were already unique. Used while loop for indices to keep size

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…pensearch-project#9184)

* Prevent duplicate keys in random mapping fields

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Aliases were already unique. Used while loop for indices to keep size

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…pensearch-project#9184)

* Prevent duplicate keys in random mapping fields

Signed-off-by: Daniel Widdis <widdis@gmail.com>

* Aliases were already unique. Used while loop for indices to keep size

Signed-off-by: Daniel Widdis <widdis@gmail.com>

---------

Signed-off-by: Daniel Widdis <widdis@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch skip-changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] org.opensearch.client.indices.GetIndexTemplatesResponseTests.testParsingFromOpenSearchResponse is flaky
3 participants