Bug/sbp cancellation #13474

kaushalmahi12 · 2024-04-30T17:48:58Z

Description

This PR is to address and fix the BUG: #13295

Changes

Refactor SearchBackpressureService to introduce resource wise cancellation when node in duress because of the resource
Move all resourceTrackers into a single class
Put the logic to calculate whether a resource usage is breaching for a task behind an interface and make it a instance member
Add an UT to cover the mentioned bug scenario

New Logic for Cancellation

Related Issues

Resolves #13295

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2024-04-30T18:25:57Z

❌ Gradle check result for aa4fd2b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-30T18:48:36Z

❌ Gradle check result for bf11c85: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-30T19:07:55Z

❌ Gradle check result for 5bcac55: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

ticheng-aws · 2024-05-01T21:27:46Z

Hi @kaushalmahi12, thank you for submitting this PR. Would you mind also creating a cancellation logic diagram similar to
what you've previously done #13295 (comment)? It would really help us grasp the changes for search backpressure.

server/src/test/java/org/opensearch/search/backpressure/stats/SearchShardTaskStatsTests.java

server/src/test/java/org/opensearch/search/backpressure/stats/SearchTaskStatsTests.java

server/src/main/java/org/opensearch/search/backpressure/trackers/NodeDuressTrackers.java

server/src/main/java/org/opensearch/search/backpressure/trackers/TaskResourceUsageTrackers.java

server/src/main/java/org/opensearch/search/backpressure/SearchBackpressureService.java

github-actions · 2024-05-21T18:45:54Z

❌ Gradle check result for 6b1c658: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-21T20:42:39Z

❌ Gradle check result for cd3e65b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-05-21T20:44:36Z

❌ Gradle check result for 2c3c4bc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-20T18:45:28Z

❌ Gradle check result for 3c3c64e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

github-actions · 2024-06-20T23:32:48Z

❕ Gradle check result for 214feba: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-06-20T23:51:45Z

❌ Gradle check result for becc022: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

github-actions · 2024-06-21T14:39:35Z

✅ Gradle check result for bd55e42: SUCCESS

github-actions · 2024-06-21T14:43:42Z

✅ Gradle check result for 0e38dee: SUCCESS

sohami · 2024-06-21T16:06:06Z

❌ Gradle check result for becc022: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Failure related to IndicesRequestCacheIT timeout which was recently fixed by #14369

* change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> (cherry picked from commit bcccedb) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Bug/sbp cancellation (#13474) * change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> (cherry picked from commit bcccedb) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Bug/sbp cancellation (#13474) * change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix compilation error Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

* Bug/sbp cancellation (opensearch-project#13474) * change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> (cherry picked from commit bcccedb) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Bug/sbp cancellation (opensearch-project#13474) * change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix compilation error Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: kkewwei <kkewwei@163.com>

* change cancellation logic to fix disparity bw trackers and resource duress Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional tests for searchBackpressureService and refactor code Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add enumMap instead of list for tracking taskResourceUsageTrackets Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add nodeNotInDuress test for nodeDuressTrackers class Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add entry in CHANGELOG Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * remove wildcard import Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * streamline imports Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * address comments Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add additional test case to test the circuit breaker for SBP logic Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add missing javadoc to resourece type enum Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * add javadoc to a method Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> * fix javadoc warnings Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com> --------- Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

kaushalmahi12 marked this pull request as ready for review May 1, 2024 02:31

kaushalmahi12 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners May 1, 2024 02:31

jainankitk reviewed May 8, 2024

View reviewed changes

add missing javadoc to resourece type enum

214feba

Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

kaushalmahi12 force-pushed the bug/sbpCancellation branch from 3c3c64e to 214feba Compare June 20, 2024 22:32

add javadoc to a method

becc022

Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

opensearch-ci-bot mentioned this pull request Jun 20, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

kaushalmahi12 added 2 commits June 21, 2024 06:35

fix javadoc warnings

bd55e42

Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

fix javadoc warnings

0e38dee

Signed-off-by: Kaushal Kumar <ravi.kaushal97@gmail.com>

sohami approved these changes Jun 21, 2024

View reviewed changes

sohami merged commit bcccedb into opensearch-project:main Jun 21, 2024
29 checks passed

opensearch-trigger-bot bot mentioned this pull request Jun 21, 2024

[Backport 2.x] Bug/sbp cancellation #14493

Closed

opensearch-ci-bot mentioned this pull request Jun 21, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteStoreStatsIT #14310

Open

kaushalmahi12 mentioned this pull request Jun 21, 2024

[Backport 2.x] Bug/sbp cancellation #14502

Merged

finnegancarroll mentioned this pull request Jul 15, 2024

Make remote snapshot (local)block size configurable #14753

Closed

3 tasks

BrewTestBot mentioned this pull request Aug 8, 2024

opensearch 2.16.0 Homebrew/homebrew-core#180446

Merged

github-actions bot added bug Something isn't working enhancement Enhancement or improvement to existing feature or request Search:Performance Search:Resiliency labels Aug 27, 2024

This was referenced Sep 24, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SearchTimeoutIT #16056

Closed

[AUTOCUT] Gradle Check Flaky Test Report for RemoteIndexShardTests #16116

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug/sbp cancellation #13474

Bug/sbp cancellation #13474

kaushalmahi12 commented Apr 30, 2024 •

edited by jed326

Loading

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

ticheng-aws commented May 1, 2024

github-actions bot commented May 21, 2024

github-actions bot commented May 21, 2024

github-actions bot commented May 21, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024

sohami commented Jun 21, 2024

Bug/sbp cancellation #13474

Bug/sbp cancellation #13474

Conversation

kaushalmahi12 commented Apr 30, 2024 • edited by jed326 Loading

Description

Changes

New Logic for Cancellation

Related Issues

Check List

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

github-actions bot commented Apr 30, 2024

ticheng-aws commented May 1, 2024

github-actions bot commented May 21, 2024

github-actions bot commented May 21, 2024

github-actions bot commented May 21, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 20, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024

sohami commented Jun 21, 2024

kaushalmahi12 commented Apr 30, 2024 •

edited by jed326

Loading