Adds Create Ingest Pipeline Step #44

joshpalis · 2023-09-18T17:44:29Z

Description

Adds CreateIngestPipelineStep for workflow set up. In this current state, this step is hardcoded to parse the necessary information from the WorkflowData related to the text_embedding processor.

Issues Resolved

#24

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Joshua Palis <jpalis@amazon.com>

…ments Signed-off-by: Joshua Palis <jpalis@amazon.com>

Signed-off-by: Joshua Palis <jpalis@amazon.com>

…rather than from content Signed-off-by: Joshua Palis <jpalis@amazon.com>

Signed-off-by: Joshua Palis <jpalis@amazon.com>

…avadoc error Signed-off-by: Joshua Palis <jpalis@amazon.com>

codecov · 2023-09-20T21:27:21Z

Codecov Report

Merging #44 (9662632) into main (a530739) will increase coverage by 5.54%.
The diff coverage is 87.87%.

@@             Coverage Diff              @@
##               main      #44      +/-   ##
============================================
+ Coverage     68.33%   73.87%   +5.54%     
- Complexity       48       65      +17     
============================================
  Files             7        8       +1     
  Lines           180      245      +65     
  Branches         18       22       +4     
============================================
+ Hits            123      181      +58     
- Misses           48       54       +6     
- Partials          9       10       +1

Files	Coverage Δ
.../opensearch/flowframework/FlowFrameworkPlugin.java	`0.00% <0.00%> (ø)`
...owframework/workflow/CreateIngestPipelineStep.java	`90.62% <90.62%> (ø)`

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis

Initial review. Looks good so far!

src/main/java/org/opensearch/flowframework/FlowFrameworkPlugin.java

src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java

…kflow data for required fields Signed-off-by: Joshua Palis <jpalis@amazon.com>

Signed-off-by: Joshua Palis <jpalis@amazon.com>

…st body generator Signed-off-by: Joshua Palis <jpalis@amazon.com>

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis

Some similar comments to #38

src/main/java/org/opensearch/flowframework/FlowFrameworkPlugin.java

src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java

...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java

Signed-off-by: Joshua Palis <jpalis@amazon.com>

owaiskazi19

Overall LGTM with few comments.

src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java

...rg/opensearch/flowframework/workflow/CreateIngestPipeline/CreateIngestPipelineStepTests.java

Signed-off-by: Joshua Palis <jpalis@amazon.com>

src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java

* Initial create ingest pipeline implementation Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding TODOs for storing response to global context index, fixing comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Updating workflowData interface, modifying CreateIngestPipelineStep Signed-off-by: Joshua Palis <jpalis@amazon.com> * updating workflow data extraction to read pipelineId from parameters rather than from content Signed-off-by: Joshua Palis <jpalis@amazon.com> * removing unecessary cast Signed-off-by: Joshua Palis <jpalis@amazon.com> * Pulls all required data from content rather than from params, fixes javadoc error Signed-off-by: Joshua Palis <jpalis@amazon.com> * fixing comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * addressing PR comments, adding switch statement to handle parsing workflow data for required fields Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding entry import Signed-off-by: Joshua Palis <jpalis@amazon.com> * fixing comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adds unit tests for create ingest pipeline step, fixes pipeline request body generator Signed-off-by: Joshua Palis <jpalis@amazon.com> * Adding failure tests Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Addressing PR comments Signed-off-by: Joshua Palis <jpalis@amazon.com> * Fixing workflow data Signed-off-by: Joshua Palis <jpalis@amazon.com> --------- Signed-off-by: Joshua Palis <jpalis@amazon.com> (cherry picked from commit 1d22bee) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Adds Create Ingest Pipeline Step (#44) * Initial create ingest pipeline implementation * Adding TODOs for storing response to global context index, fixing comments * Updating workflowData interface, modifying CreateIngestPipelineStep * updating workflow data extraction to read pipelineId from parameters rather than from content * removing unecessary cast * Pulls all required data from content rather than from params, fixes javadoc error * fixing comments * addressing PR comments, adding switch statement to handle parsing workflow data for required fields * Adding entry import * fixing comments * Adds unit tests for create ingest pipeline step, fixes pipeline request body generator * Adding failure tests * Addressing PR comments * Addressing PR comments * Fixing workflow data --------- (cherry picked from commit 1d22bee) Signed-off-by: Joshua Palis <jpalis@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Joshua Palis <jpalis@amazon.com>

#47) * Add WorkflowStepFactory class Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add XContent classes representing Template JSON Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add parse methods for the Template XContent Signed-off-by: Daniel Widdis <widdis@gmail.com> * Cleanup parsing, javadocs, and demo output Signed-off-by: Daniel Widdis <widdis@gmail.com> * Refactor to use field name constants, get tests working again Signed-off-by: Daniel Widdis <widdis@gmail.com> * Separate WorkflowNode and ProcessNode functionality Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix demos to align with template field names Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add workflow, node, and edge tests Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add Template tests Signed-off-by: Daniel Widdis <widdis@gmail.com> * Refactor TemplateParser to WorkflowProcessSorter Signed-off-by: Daniel Widdis <widdis@gmail.com> * Test exceptional cases Signed-off-by: Daniel Widdis <widdis@gmail.com> * Finish up exceptional cases Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix a template field name bug in demo Signed-off-by: Daniel Widdis <widdis@gmail.com> * Rebase with #34 Signed-off-by: Daniel Widdis <widdis@gmail.com> * Rebase changes from #54 Signed-off-by: Daniel Widdis <widdis@gmail.com> * Integrate thread pool executor service Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix flaky ProcessNodeTests by removing orTimeout Signed-off-by: Daniel Widdis <widdis@gmail.com> * Rebase and refactor with #44 Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix demos and remove DataDemo Signed-off-by: Daniel Widdis <widdis@gmail.com> * Use non-deprecated mapping method for CreateIndexStep Signed-off-by: Daniel Widdis <widdis@gmail.com> * Eliminate casting and deprecation warnings on test classes Signed-off-by: Daniel Widdis <widdis@gmail.com> * Remove unused/leftover demo class Signed-off-by: Daniel Widdis <widdis@gmail.com> * Typo Signed-off-by: Daniel Widdis <widdis@gmail.com> * Don't offer steps as an alternative to nodes Signed-off-by: Daniel Widdis <widdis@gmail.com> * Move Workflow into package with all the other parsing classes Signed-off-by: Daniel Widdis <widdis@gmail.com> * Move process sequencing classes into workflow package Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add PipelineProcessor class and XContent parsing, rename package Signed-off-by: Daniel Widdis <widdis@gmail.com> --------- Signed-off-by: Daniel Widdis <widdis@gmail.com>

#47) * Add WorkflowStepFactory class Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add XContent classes representing Template JSON Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add parse methods for the Template XContent Signed-off-by: Daniel Widdis <widdis@gmail.com> * Cleanup parsing, javadocs, and demo output Signed-off-by: Daniel Widdis <widdis@gmail.com> * Refactor to use field name constants, get tests working again Signed-off-by: Daniel Widdis <widdis@gmail.com> * Separate WorkflowNode and ProcessNode functionality Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix demos to align with template field names Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add workflow, node, and edge tests Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add Template tests Signed-off-by: Daniel Widdis <widdis@gmail.com> * Refactor TemplateParser to WorkflowProcessSorter Signed-off-by: Daniel Widdis <widdis@gmail.com> * Test exceptional cases Signed-off-by: Daniel Widdis <widdis@gmail.com> * Finish up exceptional cases Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix a template field name bug in demo Signed-off-by: Daniel Widdis <widdis@gmail.com> * Rebase with #34 Signed-off-by: Daniel Widdis <widdis@gmail.com> * Rebase changes from #54 Signed-off-by: Daniel Widdis <widdis@gmail.com> * Integrate thread pool executor service Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix flaky ProcessNodeTests by removing orTimeout Signed-off-by: Daniel Widdis <widdis@gmail.com> * Rebase and refactor with #44 Signed-off-by: Daniel Widdis <widdis@gmail.com> * Fix demos and remove DataDemo Signed-off-by: Daniel Widdis <widdis@gmail.com> * Use non-deprecated mapping method for CreateIndexStep Signed-off-by: Daniel Widdis <widdis@gmail.com> * Eliminate casting and deprecation warnings on test classes Signed-off-by: Daniel Widdis <widdis@gmail.com> * Remove unused/leftover demo class Signed-off-by: Daniel Widdis <widdis@gmail.com> * Typo Signed-off-by: Daniel Widdis <widdis@gmail.com> * Don't offer steps as an alternative to nodes Signed-off-by: Daniel Widdis <widdis@gmail.com> * Move Workflow into package with all the other parsing classes Signed-off-by: Daniel Widdis <widdis@gmail.com> * Move process sequencing classes into workflow package Signed-off-by: Daniel Widdis <widdis@gmail.com> * Add PipelineProcessor class and XContent parsing, rename package Signed-off-by: Daniel Widdis <widdis@gmail.com> --------- Signed-off-by: Daniel Widdis <widdis@gmail.com> (cherry picked from commit 734f9c2) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…Template Parsing (#60) Add WorkflowStep Factory and implement XContent-based Template Parsing (#47) * Add WorkflowStepFactory class * Add XContent classes representing Template JSON * Add parse methods for the Template XContent * Cleanup parsing, javadocs, and demo output * Refactor to use field name constants, get tests working again * Separate WorkflowNode and ProcessNode functionality * Fix demos to align with template field names * Add workflow, node, and edge tests * Add Template tests * Refactor TemplateParser to WorkflowProcessSorter * Test exceptional cases * Finish up exceptional cases * Fix a template field name bug in demo * Rebase with #34 * Rebase changes from #54 * Integrate thread pool executor service * Fix flaky ProcessNodeTests by removing orTimeout * Rebase and refactor with #44 * Fix demos and remove DataDemo * Use non-deprecated mapping method for CreateIndexStep * Eliminate casting and deprecation warnings on test classes * Remove unused/leftover demo class * Typo * Don't offer steps as an alternative to nodes * Move Workflow into package with all the other parsing classes * Move process sequencing classes into workflow package * Add PipelineProcessor class and XContent parsing, rename package --------- (cherry picked from commit 734f9c2) Signed-off-by: Daniel Widdis <widdis@gmail.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

joshpalis added 3 commits September 14, 2023 22:46

Initial create ingest pipeline implementation

764c5af

Signed-off-by: Joshua Palis <jpalis@amazon.com>

Adding TODOs for storing response to global context index, fixing com…

c728f0d

…ments Signed-off-by: Joshua Palis <jpalis@amazon.com>

Updating workflowData interface, modifying CreateIngestPipelineStep

6b2a0ac

Signed-off-by: Joshua Palis <jpalis@amazon.com>

github-actions bot added the backport 2.x backport PRs to 2.x branch label Sep 18, 2023

joshpalis mentioned this pull request Sep 18, 2023

Topological Sorting and Sequenced Execution #26

Merged

joshpalis added 4 commits September 18, 2023 23:27

updating workflow data extraction to read pipelineId from parameters …

d81dc84

…rather than from content Signed-off-by: Joshua Palis <jpalis@amazon.com>

removing unecessary cast

8ce90fd

Signed-off-by: Joshua Palis <jpalis@amazon.com>

merge main

9fabe58

Signed-off-by: Joshua Palis <jpalis@amazon.com>

Pulls all required data from content rather than from params, fixes j…

5f1cc5b

…avadoc error Signed-off-by: Joshua Palis <jpalis@amazon.com>

fixing comments

fdd2dbb

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis reviewed Sep 21, 2023

View reviewed changes

joshpalis added 4 commits September 21, 2023 21:31

addressing PR comments, adding switch statement to handle parsing wor…

d9566d4

…kflow data for required fields Signed-off-by: Joshua Palis <jpalis@amazon.com>

Adding entry import

edd0b0f

Signed-off-by: Joshua Palis <jpalis@amazon.com>

fixing comments

7fcb718

Signed-off-by: Joshua Palis <jpalis@amazon.com>

Adds unit tests for create ingest pipeline step, fixes pipeline reque…

be67209

…st body generator Signed-off-by: Joshua Palis <jpalis@amazon.com>

joshpalis marked this pull request as ready for review September 22, 2023 16:51

joshpalis requested review from owaiskazi19, ohltyler, amitgalitz and jackiehanyang as code owners September 22, 2023 16:51

Adding failure tests

ff889ed

Signed-off-by: Joshua Palis <jpalis@amazon.com>

dbwiddis reviewed Sep 22, 2023

View reviewed changes

Addressing PR comments

a38773e

Signed-off-by: Joshua Palis <jpalis@amazon.com>

This was referenced Sep 25, 2023

[FEATURE] Simplify WorkflowData to a concrete class #53

Closed

Change WorkflowData from interface to class #54

Merged

dbwiddis approved these changes Sep 25, 2023

View reviewed changes

dbwiddis mentioned this pull request Sep 25, 2023

[FEATURE] Create an XContent parser for Processors in pipelines. #55

Closed

Merge main

b2338de

Signed-off-by: Joshua Palis <jpalis@amazon.com>

owaiskazi19 reviewed Sep 25, 2023

View reviewed changes

joshpalis added 3 commits September 25, 2023 20:38

Addressing PR comments

c1d4d27

Signed-off-by: Joshua Palis <jpalis@amazon.com>

Merge branch 'main' into ingest

8236580

Fixing workflow data

9662632

Signed-off-by: Joshua Palis <jpalis@amazon.com>

owaiskazi19 reviewed Sep 25, 2023

View reviewed changes

src/main/java/org/opensearch/flowframework/workflow/CreateIngestPipelineStep.java Show resolved Hide resolved

owaiskazi19 approved these changes Sep 25, 2023

View reviewed changes

owaiskazi19 merged commit 1d22bee into opensearch-project:main Sep 25, 2023
10 checks passed

opensearch-trigger-bot bot mentioned this pull request Sep 25, 2023

[Backport 2.x] Adds Create Ingest Pipeline Step #59

Merged

dbwiddis mentioned this pull request Sep 25, 2023

Add WorkflowStep Factory and implement XContent-based Template Parsing #47

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds Create Ingest Pipeline Step #44

Adds Create Ingest Pipeline Step #44

joshpalis commented Sep 18, 2023 •

edited

Loading

codecov bot commented Sep 20, 2023 •

edited

Loading

dbwiddis left a comment

dbwiddis left a comment

owaiskazi19 left a comment

Adds Create Ingest Pipeline Step #44

Adds Create Ingest Pipeline Step #44

Conversation

joshpalis commented Sep 18, 2023 • edited Loading

Description

Issues Resolved

codecov bot commented Sep 20, 2023 • edited Loading

Codecov Report

dbwiddis left a comment

Choose a reason for hiding this comment

dbwiddis left a comment

Choose a reason for hiding this comment

owaiskazi19 left a comment

Choose a reason for hiding this comment

joshpalis commented Sep 18, 2023 •

edited

Loading

codecov bot commented Sep 20, 2023 •

edited

Loading