Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky ResourceAwareTasksTests.testBasicTaskResourceTracking test #8993

Merged
merged 1 commit into from
Aug 1, 2023

Conversation

ketanv3
Copy link
Contributor

@ketanv3 ketanv3 commented Jul 31, 2023

Description

The assertMemoryUsageWithinLimits method checks if the memory usage is within 5% (up to 200 KB) of the expected memory usage. This additional buffer is to account for the class loading overhead when new code is encountered at runtime. Over time, addition of new code paths has resulted in greater overhead. This PR bumps up the 200 KB limit to 500 KB.

Related Issues

Resolves #8213

Check List

  • New functionality includes testing.
    • All tests pass
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Copy link
Contributor

@Poojita-Raj Poojita-Raj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On #8213, this test failure seems hard to reproduce on local. How are we confident this change will resolve the issue on our agent nodes?

Currently this class suite is part of the retry classes - #8825 - We can merge in this change and monitor to see if any further failures occur for a fixed time period (~2 weeks) before we remove it from the list of retried classes.

@ketanv3
Copy link
Contributor Author

ketanv3 commented Jul 31, 2023

@Poojita-Raj I can reliably reproduce the issue with this command:

./gradlew ':server:test' --tests "org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests" -Dtests.seed=3A623BD634D99A0D -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-KW -Dtests.timezone=Europe/Paris

This test fails due to the following reasons – (1) addition of new code paths have increased the class loading overhead, (2) the randomized test seed influences the test execution order in this suite, which in turn influences when the class loading overhead is incurred. Using a different seed, for example, may not fail the test.

@Poojita-Raj
Copy link
Contributor

Thanks for clarifying @ketanv3 :) This PR lgtm!

@Poojita-Raj Poojita-Raj added the backport 2.x Backport to 2.x branch label Jul 31, 2023
@mch2 mch2 merged commit b2a7348 into opensearch-project:main Aug 1, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Aug 1, 2023
…8993)

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
(cherry picked from commit b2a7348)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kotwanikunal pushed a commit that referenced this pull request Aug 1, 2023
…8993) (#9014)

(cherry picked from commit b2a7348)

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…pensearch-project#8993)

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…pensearch-project#8993)

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…pensearch-project#8993)

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
3 participants