Backport - Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy (#9992) #10312

rishabhmaurya · 2023-10-02T20:44:56Z

Backport #9992 to 2.x

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-10-02T21:08:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26632/
CommitID: 8514e37
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-02T21:21:21Z

Compatibility status:

Checks if related components are compatible with change ae503b8

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

github-actions · 2023-10-02T22:44:47Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26645/
CommitID: 8514e37
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-03T01:09:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.index.shard.RemoteIndexShardTests.testRepicaCleansUpOldCommitsWhenReceivingNew

URL: https://build.ci.opensearch.org/job/gradle-check/26658/
CommitID: 8514e37
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-10-03T01:11:04Z

Codecov Report

Merging #10312 (ae503b8) into 2.x (ee84945) will decrease coverage by 0.13%.
Report is 8 commits behind head on 2.x.
The diff coverage is 81.81%.

@@             Coverage Diff              @@
##                2.x   #10312      +/-   ##
============================================
- Coverage     70.92%   70.80%   -0.13%     
+ Complexity    58394    58361      -33     
============================================
  Files          4816     4817       +1     
  Lines        275745   275855     +110     
  Branches      40531    40540       +9     
============================================
- Hits         195572   195314     -258     
- Misses        63436    63874     +438     
+ Partials      16737    16667      -70

Files	Coverage Δ
...rg/opensearch/common/settings/ClusterSettings.java	`92.85% <ø> (ø)`
...pensearch/common/settings/IndexScopedSettings.java	`100.00% <ø> (ø)`
...rch/extensions/rest/RestActionsRequestHandler.java	`100.00% <100.00%> (ø)`
...extensions/rest/RestInitializeExtensionAction.java	`65.00% <100.00%> (-0.44%)`	⬇️
...rch/extensions/rest/RestSendToExtensionAction.java	`42.50% <100.00%> (ø)`
.../main/java/org/opensearch/index/engine/Engine.java	`75.00% <100.00%> (+0.42%)`	⬆️
...in/java/org/opensearch/index/shard/IndexShard.java	`69.57% <100.00%> (+0.70%)`	⬆️
...a/org/opensearch/extensions/ExtensionsManager.java	`50.94% <62.50%> (+7.07%)`	⬆️
...in/java/org/opensearch/index/merge/MergeStats.java	`78.35% <70.00%> (-1.43%)`	⬇️
.../main/java/org/opensearch/index/IndexSettings.java	`87.47% <89.83%> (+0.28%)`	⬆️
... and 2 more

... and 466 files with indirect coverage changes

…roject#9992) * Configurable merge policy for index * additional setting to configure merge policy for timestamp based index * introduction of logbytesize merge policy as an option Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * remove the trace log not required anymore Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Refactor the merge policy extraction logic Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Rename constant DEFAULT to DEFAULT_POLICY Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Simplify merge policy extraction and selection logic Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * missing javadoc error Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Renaming log byte size policy setting with mb Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Move validation exception to enum from setting defn Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * rename time_index to time_series_index Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> --------- Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> (cherry picked from commit fa66beb)

github-actions · 2023-10-03T17:04:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26753/
CommitID: ae503b8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-03T17:04:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26752/
CommitID: b259f8f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-03T18:28:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26762/
CommitID: ae503b8
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

rishabhmaurya · 2023-10-03T18:37:43Z

the gradle check failure looks related to #7687 and #4184

github-actions · 2023-10-03T19:43:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/26772/
CommitID: ae503b8

rishabhmaurya mentioned this pull request Oct 2, 2023

Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy #9992

Merged

6 tasks

rishabhmaurya changed the title ~~Backport - Use of LogByteSizeMergePolicy for data stream use cases (#9992)~~ Backport - Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy (#9992) Oct 2, 2023

reta approved these changes Oct 2, 2023

View reviewed changes

reta added the v2.11.0 Issues and PRs related to version 2.11.0 label Oct 3, 2023

rishabhmaurya force-pushed the backport/backport-9992-to-2.x branch from 8514e37 to b259f8f Compare October 3, 2023 16:19

rishabhmaurya force-pushed the backport/backport-9992-to-2.x branch from b259f8f to ae503b8 Compare October 3, 2023 16:32

reta merged commit 81cd1f2 into opensearch-project:2.x Oct 3, 2023
13 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport - Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy (#9992) #10312

Backport - Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy (#9992) #10312

rishabhmaurya commented Oct 2, 2023 •

edited

Loading

github-actions bot commented Oct 2, 2023

github-actions bot commented Oct 2, 2023 •

edited

Loading

github-actions bot commented Oct 2, 2023

github-actions bot commented Oct 3, 2023

codecov bot commented Oct 3, 2023 •

edited

Loading

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

rishabhmaurya commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

Backport - Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy (#9992) #10312

Backport - Configurable merge policy and option to choose between LogByteSize and Tiered MergePolicy (#9992) #10312

Conversation

rishabhmaurya commented Oct 2, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Oct 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 2, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Oct 2, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Oct 3, 2023 • edited Loading

Codecov Report

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

rishabhmaurya commented Oct 3, 2023

github-actions bot commented Oct 3, 2023

Gradle Check (Jenkins) Run Completed with:

rishabhmaurya commented Oct 2, 2023 •

edited

Loading

github-actions bot commented Oct 2, 2023 •

edited

Loading

codecov bot commented Oct 3, 2023 •

edited

Loading