Adds implementation for the delay feature in rollup jobs #147
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexRunner.kt
Outdated
Show resolved
Hide resolved
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/rollup/RollupSearchService.kt
Outdated
Show resolved
Hide resolved
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/rollup/util/RollupUtils.kt
Outdated
Show resolved
Hide resolved
thalurur
reviewed
Sep 21, 2021
thalurur
reviewed
Sep 21, 2021
src/main/kotlin/org/opensearch/indexmanagement/rollup/util/RollupUtils.kt
Outdated
Show resolved
Hide resolved
5 tasks
Signed-off-by: Clay Downs <downsrob@amazon.com>
Signed-off-by: Clay Downs <downsrob@amazon.com>
Signed-off-by: Robert Downs <downsrob@amazon.com>
Signed-off-by: Clay Downs <downsrob@amazon.com>
Signed-off-by: Clay Downs <downsrob@amazon.com>
Signed-off-by: Clay Downs <downsrob@amazon.com>
downsrob
force-pushed
the
rollups-delay
branch
from
947b1e4 to
659dfbc
Compare
Codecov Report
@@ Coverage Diff @@
## main #147 +/- ##
============================================
+ Coverage 75.70% 75.80% +0.10%
- Complexity 1921 1934 +13
============================================
Files 260 260
Lines 11016 11041 +25
Branches 1724 1735 +11
============================================
+ Hits 8340 8370 +30
+ Misses 1721 1713 -8
- Partials 955 958 +3
Continue to review full report at Codecov.
|
Signed-off-by: Clay Downs <downsrob@amazon.com>
thalurur
approved these changes
Oct 11, 2021
downsrob
added a commit
to downsrob/index-management
that referenced
this pull request
Oct 16, 2021
…project#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <downsrob@amazon.com>
1 task
downsrob
added a commit
that referenced
this pull request
Oct 25, 2021
downsrob
added a commit
that referenced
this pull request
Nov 5, 2021
* Add integTest script to the repo (#94) Signed-off-by: Peter Zhu <zhujiaxi@amazon.com> * Removing Usages of Action Get Call and using listeners (#100) Signed-off-by: Aditya Jindal <aditjind@amazon.com> * Enhance ISM template (#105) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Explain response still use old opendistro policy id (#109) * Explain response still use old opendistro policy id * Use hardcoded policyid setting in tests for explain response * Trying to fix flaky tests * Storing user information as part of the job when security plugin is installed (#113) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * ISM/Notification channel support (#117) * Updates NAME of transport actions * Upgrades Kotlin version, updates dependencies on other OS plugins, adds notification plugin as a test resource and includes it in test clusters * Adds support for Channels in error notifications and notification actions * Adds support for sending notifications to channels * Adds support for publishing notifications to the legacy destinations through the Notification plugin and some cleanup * Removes notification alerting jar dependency * Adds compile only dep on commons codec for digest utils sha1 method in ism rollup * Updates Error Notification to make channel/destination nullable, and adds helper methods for publish calls * Constructs URL for legacy custom webhook Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes Feature enum and dep Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Trying something else Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Addresses comments Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Storing user object in all APIs and enabling filter of response based on user (#115) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Upgrade dependencies to 1.1 and build snapshot by default. (#121) Signed-off-by: dblock <dblock@amazon.com> * Security improvements (#126) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Removes support for notification plugin (#136) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Updating security filtering logic (#137) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Release note for 1.1.0.0 release. (#139) * Release note for 1.1.0.0 release. Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Correct copyright notices * Uses published daily snapshot dependencies (#141) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * License header check (#142) * Provide default copyright header using IDE feature Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address #103 history write index is rolled over even if the history indices are disabled * Removed integtest.sh. (#148) Signed-off-by: dblock <dblock@dblock.org> * Adds mavenLocal back to repositories (#158) Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Making snapshot name to scripted input in template (#77) Signed-off-by: Ravi Thaluru <ravi1092@gmail.com> * Fix issues with security changes in rollup runnner (#161) * Updates index management version to 1.2 (#157) * Updates index management version to 1.2 * Updates job scheduler snapshot to 1.2 in test resources Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds setting to search all rollup jobs on a target index (#165) * Adds cluster setting to search all rollup jobs Signed-off-by: Clay Downs <downsrob@amazon.com> * Adds implementation for the delay feature in rollup jobs (#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Updates testCompile mockito version, adds AwaitsFix annotation to MetadataRegressionIT tests (#168) * Updates testCompile mockito version to match OpenSearch changes * AwaitsFix on the failing MetadataRegressionIT tests Signed-off-by: Robert Downs <downsrob@amazon.com> * Adds cluster setting to configure index state management jitter (#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <downsrob@amazon.com> * Allows out of band rollovers on an index without causing ISM to fail (#180) * Allows out of band rollovers on an index without causing ISM to fail Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Fixes detekt issue Signed-off-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> * Remove policy API on read only indices (#182) * In explain API not showing the total count to all users (#185) Co-authored-by: Peter Zhu <zhujiaxi@amazon.com> Co-authored-by: Aditya Jindal <13850971+aditjind@users.noreply.github.com> Co-authored-by: Bowen Lan <62091230+bowenlan-amzn@users.noreply.github.com> Co-authored-by: Ravi <6005951+thalurur@users.noreply.github.com> Co-authored-by: Drew Baugher <46505179+dbbaughe@users.noreply.github.com> Co-authored-by: Daniel Doubrovkine (dB.) <dblock@dblock.org>
wuychn
pushed a commit
to ochprince/index-management
that referenced
this pull request
Mar 16, 2023
…project#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <downsrob@amazon.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds implementation for the delay parameter in rollup jobs. Previously, the delay parameter was included in the api and stored with the job, but implementation was never added to use this delay when searching for windows of data to be used in the rollup job. In this PR, the rollup job excludes data from the delay time period, and the delay is passed to the job scheduler to handle delaying the actual execution.
There were job scheduler changes required as a prerequisite to this PR. This Job Scheduler PR. included changes which enabled the delay to be passed to the job scheduler to actually delay execution.
CheckList:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.