[Backport 2.x] Backports Job Scheduler Communication mechanism to 2.x #382
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Job Details from Extension for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Commnunication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> * Communication Mechanism for JS Signed-off-by: Varun Jain <varunudr@amazon.com> Signed-off-by: Varun Jain <varunudr@amazon.com>
…sService map (#299) * Added JobDetailsService as an indexOperationListener to synchronize metadata index with internal job details m and indicesToListen set. Signed-off-by: Joshua Palis <jpalis@amazon.com> * Changes indexToJobDetails to a ConcurrentMap, adds getter method for indexToJobDetails Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, fixing log message Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added test to updateIndexToJobDetails Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, changing extensionId to extensionUniqueId Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR Comments : Updating Job Details index Name and mapping file to opensearch-job-scheduler-job-details Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, enabling extensions to submit more than 1 job details entry to support extensions registering multiple types of jobs, updated all integration, multinode tests now that the rest response value is the document Id Signed-off-by: Joshua Palis <jpalis@amazon.com> * Renaming TestHelper base URI name Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, made multinode test request strings constant Signed-off-by: Joshua Palis <jpalis@amazon.com> Signed-off-by: Joshua Palis <jpalis@amazon.com>
… invoke corresponding registered Job Detail actions (#306) * Draft : Added JobParameterRequest to translate ScheduledJobParser parameters nto compatible inputs for the ExtensionActionRequest. Completed initial proxy scheduled job parser implementation. Added ExtensionJobParameter class to deserialize ExtensionActionResponsebyte array into an object of type ScheduledJobParameter Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added JobRunnerRequest, modified JobExecutionContext to implement writeable, created initial proxy ScheduledJobRunner, fixed failing tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added generic ExtensionJobActionRequest that extends ExtensionActionRequest, modified JobParameter/Runner request to implement writeable, refactored proxy object creation so that the requests are added to the ExtensionJobActionRequest, which in turn upcasts the request into an ExtensionActionRequest Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added byte array constructors for the JobRunner/Parameter requests, added javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added placeholder string for an eventual access token to be sent to extensions to validate prior to invoking an action Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added ExtensionJobActionResponse, JobParameterResponse, JobRunnerResponse to facilitate the response of extension actions Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added JobRunnerResponse handling Signed-off-by: Joshua Palis <jpalis@amazon.com> * Separating updateIndexToJobProviders into separate methods Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> * SpotlessApply Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added tests for serialization/deserialization of JobExecutionContext, JobDocVersion, ExtensionJobParameter, ExtensionJobActionRequest, ExtensionJobActionResponse, JobRunner/JobParameter/Request/Response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Writing ExtensionActionRequest/Response to bytestream output to test deserialization Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing imports Signed-off-by: Joshua Palis <jpalis@amazon.com> * Changing to extensionActionResponse constructor Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding tests for updateIndexToJobProviders Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments, added getters for lock duration seconds and jitter values, added javadocs to ScheduleType enum and made this public Signed-off-by: Joshua Palis <jpalis@amazon.com> * Removing Strings dependency from lock model to fix BWC tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixes BWC fullRestartClusterTask, rollingUpgradeClusterTask, oldVersionCluster task. mixedClusterTask is still failing Signed-off-by: Joshua Palis <jpalis@amazon.com> * Modified createProxyScheduledJobRunner to return the document Id of the extension job parameter entry within the registered job index, modifed the ExtensionJobParameter to null check the jitter value and setting this to 0.0 if null Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com>
…re a lock model for their job execution (#311) * Added RestGetLockAction API, added AcquireLockRequest, enables extensions to retrieve a lock model object to run their Job Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding Lock ID field to RestGetLockAction response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added multi node integration tests for GetLockApi Signed-off-by: Joshua Palis <jpalis@amazon.com> * Making lock service in RestGetLockAction private Signed-off-by: Joshua Palis <jpalis@amazon.com> * Added Rest integration tests for RestGetLockAction Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Updating get lock rest path in tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com>
* Release lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Release Lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Release Lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Release Lock API Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> * Reformatting Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Dan's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Dan's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Sarat's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Addressing Sarat's Comments Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing test cases Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing test cases Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing ReleaseLocktestcase Signed-off-by: Varun Jain <varunudr@amazon.com> * Fixing ReleaseLocktestcase Signed-off-by: Varun Jain <varunudr@amazon.com> --------- Signed-off-by: Varun Jain <varunudr@amazon.com>
#342) * Modifies the RestGetLockAction response to include all fields of the lock model object, rather than the lock model object itself Signed-off-by: Joshua Palis <jpalis@amazon.com> * Removing unused fields Signed-off-by: Joshua Palis <jpalis@amazon.com> * reverting field removal Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding lockID to back to response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding checks to multinode get lock rest integration tests to ensure that all response fields are populated Signed-off-by: Joshua Palis <jpalis@amazon.com> * fixing lock time parsing Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adds a new Response class to house serde logic for RestGetLockAction response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Moving parser.nextToken() within AcquireLockResponse.parse() Signed-off-by: Joshua Palis <jpalis@amazon.com> * Implementing toXContentObject for AcquireLockRequest Signed-off-by: Joshua Palis <jpalis@amazon.com> * Moving AcquireLockResponse to transpor package Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com>
…nitial job detail registration request will initialize the job details metadata index, which takes some time (#346) Signed-off-by: Joshua Palis <jpalis@amazon.com>
…nses (#349) Signed-off-by: Joshua Palis <jpalis@amazon.com>
…xtra trimming of request bytes for the streaminput constructor for the JobParameter/RunnerRequest, as this is already trimmed by the SDK prior to forwarding the actionrequest to the transport action (#351) Signed-off-by: Joshua Palis <jpalis@amazon.com>
…d from ActionRequest/Response (#352) * Replaces ExtensionActionRequest class name with the JobParameter/RunnerRequest fully qualified class names, modifes JobParameter/Runner/Request/Response classes to extend from ActionRequest/Response Signed-off-by: Joshua Palis <jpalis@amazon.com> * Removes ExtensionJobActionResponse and fixes affected test classes Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing javadocs Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com>
Signed-off-by: Sarat Vemulapalli <vemulapallisarat@gmail.com>
joshpalis
requested review from
saratvemulapalli,
dbwiddis,
kaituo and
vibrantvarun
as code owners
Signed-off-by: Joshua Palis <jpalis@amazon.com>
Signed-off-by: Joshua Palis <jpalis@amazon.com>
Signed-off-by: Joshua Palis <jpalis@amazon.com>
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## 2.x #382 +/- ##
=============================================
- Coverage 52.11% 28.77% -23.34%
- Complexity 64 97 +33
=============================================
Files 8 22 +14
Lines 449 1178 +729
Branches 50 109 +59
=============================================
+ Hits 234 339 +105
- Misses 195 818 +623
- Partials 20 21 +1
|
dbwiddis
approved these changes
May 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Backports all Job Scheduler communication mechanism commits for extensions from main to 2.x
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.