Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] Cluster Manager task throttling #5041

Merged
merged 8 commits into from
Nov 4, 2022
Merged

[Backport 2.x] Cluster Manager task throttling #5041

merged 8 commits into from
Nov 4, 2022

Conversation

dhwanilpatel
Copy link
Contributor

Description

Back porting cluster manager task throttling changes to 2.x branch.
PR for main: #4986

Changes were already reviewed and merged in feature branch. Below is list of changes.

Issues Resolved

#479

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

dhwanilpatel and others added 5 commits November 2, 2022 21:24
@dhwanilpatel @peterzhuamazon
Add basic thorttler/exponential backoff policy for retry/Defination o…
1b8f6e5 
…f throttling exception (opensearch-project#3856)

* Add basic thorttler/exponential backoff policy for retry/Defination of throttling exception

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Incorporated comments

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Incorporated minor comments

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Removed overall thorttlingEnabled flag from Throttler

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Corrected Java doc for Throttler

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Incorporated comments

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Changed the default behaviour of Throttler to return Optional

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Removed generics from Throttler and used String as key

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Ignore backport / autocut / dependabot branches for gradle checks on push

Signed-off-by: Peter Zhu <zhujiaxi@amazon.com>

Co-authored-by: Peter Zhu <zhujiaxi@amazon.com>
@dhwanilpatel
Master node changes for master task throttling (opensearch-project#3882)
8e5ef05 
* Master node changes for master task throttling

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@dhwanilpatel
Data node changes for master task throttling (opensearch-project#4204)
3ac3d37 
* Data node changes for master task throttling

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>

* Using Retryable action for retries
* Used RemoteAddress instead of new field for checking local Request
@dhwanilpatel
Onboarding of few task types to throttling (opensearch-project#4542)
13f7487 
* Onboarding of few task types to throttling

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@dhwanilpatel
Fix timeout exception and Add Integ test for Master task throttling (o…
04731bc 
…pensearch-project#4588)

* Fix timeout exception and Add Integ test for Master task throttling

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel dhwanilpatel force-pushed the master_throttling_backport_2x branch from ccfadd5 to 26c9395 Compare November 2, 2022 16:44
@dhwanilpatel dhwanilpatel marked this pull request as ready for review November 2, 2022 16:46
@dhwanilpatel dhwanilpatel requested review from a team and reta as code owners November 2, 2022 16:46
@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel dhwanilpatel force-pushed the master_throttling_backport_2x branch from 26c9395 to 163ffe0 Compare November 2, 2022 17:32
@github-actions
Copy link
Contributor

github-actions bot commented Nov 2, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel
Copy link
Contributor Author 

[org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}](https://build.ci.opensearch.org/job/gradle-check/6234/testReport/junit/org.opensearch.repositories.s3/RepositoryS3ClientYamlTestSuiteIT/test__yaml_repository_s3_20_repository_permanent_credentials_Snapshot_and_Restore_with_repository_s3_using_permanent_credentials_/)
    [org.opensearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/20_repository_permanent_credentials/Snapshot and Restore with repository-s3 using permanent credentials}](https://build.ci.opensearch.org/job/gradle-check/6234/testReport/junit/org.opensearch.repositories.s3/RepositoryS3ClientYamlTestSuiteIT/test__yaml_repository_s3_20_repository_permanent_credentials_Snapshot_and_Restore_with_repository_s3_using_permanent_credentials__2/)
    [org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testIndexDeletionDuringSnapshotCreationInQueue](https://build.ci.opensearch.org/job/gradle-check/6234/testReport/junit/org.opensearch.snapshots/DedicatedClusterSnapshotRestoreIT/testIndexDeletionDuringSnapshotCreationInQueue/)

@dhwanilpatel dhwanilpatel force-pushed the master_throttling_backport_2x branch from 163ffe0 to 84296cd Compare November 3, 2022 04:34
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel
Copy link
Contributor Author

gradle check failed with some jenkins issue.

java.lang.NullPointerException
	at hudson.FilePath.isAbsolute(FilePath.java:296)
	at hudson.FilePath.resolvePathIfRelative(FilePath.java:281)
	at hudson.FilePath.<init>(FilePath.java:272)
	at hudson.FilePath.child(FilePath.java:1522)
	at hudson.tools.AbstractCommandInstaller.performInstallation(AbstractCommandInstaller.java:86)
	at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:70)
	at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:108)
	at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:221)
	at hudson.model.JDK.forNode(JDK.java:149)
	at hudson.model.JDK.forNode(JDK.java:59)
	at org.jenkinsci.plugins.workflow.steps.ToolStep$Execution.run(ToolStep.java:157)
	at org.jenkinsci.plugins.workflow.steps.ToolStep$Execution.run(ToolStep.java:138)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

@dhwanilpatel dhwanilpatel force-pushed the master_throttling_backport_2x branch from 84296cd to b442e5f Compare November 3, 2022 06:04
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel dhwanilpatel force-pushed the master_throttling_backport_2x branch from b442e5f to 5ef3ff5 Compare November 3, 2022 08:06
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel
Complete TODO for version change and removed unused classes(Throttler…
1e022a2 
… and Semaphore) (opensearch-project#4846)

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@dhwanilpatel dhwanilpatel force-pushed the master_throttling_backport_2x branch from 5ef3ff5 to 1e022a2 Compare November 3, 2022 11:50
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel
Remove V1 version from throttling testcase
5f77021 
Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Gradle Check (Jenkins) Run Completed with:

@dhwanilpatel
Merge remote-tracking branch 'upstream/2.x' into master_throttling_ba…
a428942 
…ckport_2x

Signed-off-by: Dhwanil Patel <dhwanip@amazon.com>
@github-actions
Copy link
Contributor

github-actions bot commented Nov 4, 2022

Gradle Check (Jenkins) Run Completed with:

@Bukhtawar Bukhtawar changed the title [Backport] Cluster Manager task throttling 2x [Backport 2.x] Cluster Manager task throttling Nov 4, 2022
@Bukhtawar Bukhtawar merged commit d10bc9f into opensearch-project:2.x Nov 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Bukhtawar Bukhtawar Bukhtawar approved these changes

@reta reta Awaiting requested review from reta reta is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dhwanilpatel @Bukhtawar