Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Workload Management] Add QueryGroupServiceSettings #15028

Merged
merged 7 commits into from
Aug 21, 2024

Conversation

ruai0511
Copy link
Contributor

Description

This change introduces the QueryGroup service settings which we will use to enforce node level resiliency as part of this RFC: #12342.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

  • Functionality includes testing.
  • API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ruai0511 ruai0511 added the backport 2.x Backport to 2.x branch label Jul 30, 2024
@ruai0511 ruai0511 changed the title [Workload Management] add QeryGroupServiceSettings [Workload Management] Add QeryGroupServiceSettings Jul 30, 2024
@ruai0511 ruai0511 changed the title [Workload Management] Add QeryGroupServiceSettings [Workload Management] Add QueryGroupServiceSettings Jul 30, 2024
Copy link
Contributor

❌ Gradle check result for f8e758d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 5324ca5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 07aff39: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

✅ Gradle check result for ff9d123: SUCCESS

Copy link
Contributor

✅ Gradle check result for 3fa208c: SUCCESS

Copy link

codecov bot commented Jul 31, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.89%. Comparing base (13163ab) to head (47b91d6).
Report is 2 commits behind head on main.

Additional details and impacted files
@@             Coverage Diff              @@
##               main   #15028      +/-   ##
============================================
- Coverage     71.92%   71.89%   -0.03%     
- Complexity    63194    63198       +4     
============================================
  Files          5213     5214       +1     
  Lines        295884   295935      +51     
  Branches      42754    42754              
============================================
- Hits         212820   212777      -43     
- Misses        65584    65688     +104     
+ Partials      17480    17470      -10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

✅ Gradle check result for 20e56f1: SUCCESS

CHANGELOG.md Outdated Show resolved Hide resolved
@jainankitk
Copy link
Collaborator

@ruai0511 - Can you create issue for documentation PR as well?

Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
Copy link
Contributor

✅ Gradle check result for fcedaec: SUCCESS

Copy link
Contributor

✅ Gradle check result for 47b91d6: SUCCESS

@jainankitk jainankitk merged commit 01184de into opensearch-project:main Aug 21, 2024
33 of 35 checks passed
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-15028-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 01184decb897dcdfc1b1d36980c4fc4eeddb8e54
# Push it to GitHub
git push --set-upstream origin backport/backport-15028-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-15028-to-2.x.

@ruai0511
Copy link
Contributor Author

@ruai0511 - Can you create issue for documentation PR as well?
Opened an issue here: opensearch-project/documentation-website#8066

ruai0511 added a commit to kaushalmahi12/OpenSearch that referenced this pull request Aug 21, 2024
ruai0511 added a commit to kaushalmahi12/OpenSearch that referenced this pull request Aug 21, 2024
…ct#15028)

Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
(cherry picked from commit 01184de)
ruai0511 added a commit to kaushalmahi12/OpenSearch that referenced this pull request Aug 21, 2024
…ct#15028)

Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
(cherry picked from commit 01184de)
ruai0511 added a commit to kaushalmahi12/OpenSearch that referenced this pull request Aug 21, 2024
…ct#15028)

Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
(cherry picked from commit 01184de)
wdongyu pushed a commit to wdongyu/OpenSearch that referenced this pull request Aug 22, 2024
…ct#15028)

* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
jainankitk added a commit that referenced this pull request Aug 22, 2024
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
(cherry picked from commit 01184de)

Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
Signed-off-by: Ankit Jain <akjain@amazon.com>
Co-authored-by: Ankit Jain <akjain@amazon.com>
shiv0408 added a commit to shiv0408/OpenSearch that referenced this pull request Sep 2, 2024
* Optimize global ordinal includes/excludes for prefix matching (opensearch-project#14371)

* Optimize global ordinal includes/excludes for prefix matching

If an aggregration specifies includes or excludes based on a regular
expression, and the regular expression has a finite expansion followed
by .*, then we can optimize the global ordinal filter.

Specifically, in this case, we can expand the matching prefixes, then
include/exclude the range of global ordinals that start with each
prefix.

Signed-off-by: Michael Froh <froh@amazon.com>

* Add unit test

Signed-off-by: Michael Froh <froh@amazon.com>

* Add changelog entry

Signed-off-by: Michael Froh <froh@amazon.com>

* Improve test coverage

Updated the unit test to be functionally equivalent, but it covers
more of the regex logic.

Signed-off-by: Michael Froh <froh@amazon.com>

* Improve test coverage

Signed-off-by: Michael Froh <froh@amazon.com>

* Fix bug in exclude-only case with no doc values in segment

Signed-off-by: Michael Froh <froh@amazon.com>

* Address comments from @mch2

Signed-off-by: Michael Froh <froh@amazon.com>

---------

Signed-off-by: Michael Froh <froh@amazon.com>

* Adding access to noSubMatches and noOverlappingMatches in Hyphenation… (opensearch-project#13895)

* Adding access to noSubMatches and noOverlappingMatches in HyphenationCompoundWordTokenFilter

Signed-off-by: Evan Kielley <evankielley@gmail.com>

* Add Changelog Entry

Signed-off-by: Mohammad Hasnain Mohsin Rajan <hasnain2808@gmail.com>

* test: add hyphenation decompounder tests

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* test: refactor tests

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* test: reformat test files

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* chore: add changelog entry for 2.X

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* chore: remove 3.x changelog

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* chore: commonify settingsarr

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* chore: commonify settingsarr

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

* chore: linting

Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>

---------

Signed-off-by: Evan Kielley <evankielley@gmail.com>
Signed-off-by: Mohammad Hasnain Mohsin Rajan <hasnain2808@gmail.com>
Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>
Co-authored-by: Evan Kielley <evankielley@gmail.com>

* Add Settings related to Workload Management feature (opensearch-project#15028)

* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* Update affiliation for @nknize. (opensearch-project#15322)

Signed-off-by: dblock <dblock@amazon.com>

* Add log when download completes with file size (opensearch-project#15224)

Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>

* Support Filtering on Large List encoded by Bitmap (version update) (opensearch-project#15352)

Signed-off-by: Andriy Redko <andriy.redko@aiven.io>

* Add support for index level slice count setting (opensearch-project#15336)

Signed-off-by: Ganesh Ramadurai <gramadur@amazon.com>

* Adding allowlist setting for ingest-useragent and ingest-geoip processors (opensearch-project#15325)

* Adding allowlist setting for user-agent, geo-ip and updated tests for ingest-common.

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Remove duplicate test in ingest-common

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Adding changelog

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

---------

Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>

* Add Delete QueryGroup API Logic (opensearch-project#14735)

* Add Delete QueryGroup API Logic
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* include comments from create pr
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove delete all
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* rebase and address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* rebase
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add UT coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* [Star Tree] Lucene Abstractions for Star Tree File Formats  (opensearch-project#15278)

---------
Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com>

* [Star tree] Changes to handle derived metrics such as avg as part of star tree mapping (opensearch-project#15152)

---------
Signed-off-by: Bharathwaj G <bharath78910@gmail.com>

* relaxing the join validation for nodes which have only store disabled but only publication enabled

* relaxing the join validation for nodes which have only store disabled but only publication enabled

Signed-off-by: Rajiv Kumar Vaidyanathan <rajivkv@amazon.com>

---------

Signed-off-by: Michael Froh <froh@amazon.com>
Signed-off-by: Evan Kielley <evankielley@gmail.com>
Signed-off-by: Mohammad Hasnain Mohsin Rajan <hasnain2808@gmail.com>
Signed-off-by: Mohammad Hasnain <hasnain2808@gmail.com>
Signed-off-by: dblock <dblock@amazon.com>
Signed-off-by: Gaurav Bafna <gbbafna@amazon.com>
Signed-off-by: Andriy Redko <andriy.redko@aiven.io>
Signed-off-by: Ganesh Ramadurai <gramadur@amazon.com>
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
Signed-off-by: Rajiv Kumar Vaidyanathan <rajivkv@amazon.com>
Co-authored-by: Michael Froh <froh@amazon.com>
Co-authored-by: Mohammad Hasnain Mohsin Rajan <hasnain2808@gmail.com>
Co-authored-by: Evan Kielley <evankielley@gmail.com>
Co-authored-by: Ruirui Zhang <mariazrr@amazon.com>
Co-authored-by: Daniel (dB.) Doubrovkine <dblock@amazon.com>
Co-authored-by: Gaurav Bafna <85113518+gbbafna@users.noreply.github.com>
Co-authored-by: Andriy Redko <andriy.redko@aiven.io>
Co-authored-by: Ganesh Krishna Ramadurai <gramadur@icloud.com>
Co-authored-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>
Co-authored-by: Bharathwaj G <bharath78910@gmail.com>
Co-authored-by: Rajiv Kumar Vaidyanathan <rajivkv@amazon.com>
akolarkunnu pushed a commit to akolarkunnu/OpenSearch that referenced this pull request Sep 10, 2024
…ct#15028)

* add QeryGroup Service tests
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add PR to changelog
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* change the test directory
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* modify comments to be more specific
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* add test coverage
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* remove QUERY_GROUP_RUN_INTERVAL_SETTING as we'll define it in QueryGroupService
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>

* address comments
Signed-off-by: Ruirui Zhang <mariazrr@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants