[Search Pipelines] Split search pipeline processor factories by type #7597

msfroh · 2023-05-17T02:07:46Z

Description

In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake.

We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier.

As a bonus, we'll be able to reuse processor names across different types of processor.

Related Issues

Resolves #7576

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-05-17T02:35:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15542/
CommitID: 89c4732
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-05-17T02:42:19Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15543/
CommitID: 139e591
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineInfo.java

github-actions · 2023-05-17T18:19:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15598/
CommitID: 76e3245

codecov · 2023-05-17T18:23:48Z

Codecov Report

Merging #7597 (63f0fa7) into main (df7cf27) will increase coverage by 0.06%.
The diff coverage is 83.52%.

❗ Current head 63f0fa7 differs from pull request most recent head faabc42. Consider uploading reports for the commit faabc42 to get more accurate results

@@             Coverage Diff              @@
##               main    #7597      +/-   ##
============================================
+ Coverage     70.80%   70.87%   +0.06%     
- Complexity    56085    56118      +33     
============================================
  Files          4676     4676              
  Lines        265927   265895      -32     
  Branches      39053    39051       -2     
============================================
+ Hits         188292   188446     +154     
+ Misses        61678    61474     -204     
- Partials      15957    15975      +18

Impacted Files	Coverage Δ
...h/pipeline/common/FilterQueryRequestProcessor.java	`85.29% <ø> (ø)`
.../pipeline/common/RenameFieldResponseProcessor.java	`94.59% <ø> (ø)`
...eline/common/SearchPipelineCommonModulePlugin.java	`0.00% <0.00%> (ø)`
...a/org/opensearch/plugins/SearchPipelinePlugin.java	`0.00% <0.00%> (ø)`
...java/org/opensearch/search/pipeline/Processor.java	`100.00% <ø> (ø)`
...opensearch/search/pipeline/SearchPipelineInfo.java	`72.41% <84.44%> (+36.69%)`	⬆️
...nsearch/search/pipeline/SearchPipelineService.java	`82.77% <88.23%> (-0.04%)`	⬇️
.../java/org/opensearch/search/pipeline/Pipeline.java	`86.20% <100.00%> (+1.59%)`	⬆️

... and 463 files with indirect coverage changes

github-actions · 2023-05-18T06:53:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.remotestore.SegmentReplicationRemoteStoreIT.testReplicaHasDiffFilesThanPrimary

URL: https://build.ci.opensearch.org/job/gradle-check/15656/
CommitID: 4a5c9bc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-05-19T18:41:28Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/15780/
CommitID: 63f0fa7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

msfroh · 2023-05-19T21:00:00Z

a flaky test

Flaky test: #7592

msfroh · 2023-05-19T22:12:18Z

Looking for some feedback on this change. @reta @andrross @navneet1v ?

I would like to get it in for 2.8, since it will be much harder to change if/when we release the search pipelines feature more broadly.

macohen · 2023-05-21T17:07:48Z

can we have a reviewer in addition to @reta and @andrross take a look at this please? @VachaShah @kartg @saratvemulapalli @tlfeng, perhaps?

thanks!

nknize

Pretty straightforward, overall LGTM. Just one Q on thread safety.

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java

nknize · 2023-05-22T18:06:55Z

Le sigh.. another unrelated failure. #7592

These "flaky" tests need some attention. They're starting to get unmanageable and gridlock our PRs.

github-actions · 2023-05-22T19:11:44Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15937/
CommitID: 63f0fa7

nknize · 2023-05-22T19:12:21Z

I'm happy to approve w/ the minor changes above. Don't worry about unrelated flaky tests here. We'll fix those separately.

In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake. We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier. As a bonus, we'll be able to reuse processor names across different types of processor. Closes opensearch-project#7576 Signed-off-by: Michael Froh <froh@amazon.com>

Signed-off-by: Michael Froh <froh@amazon.com>

@nknize

Thanks @nknize! Signed-off-by: Michael Froh <froh@amazon.com>

nknize

LGTM. 🤞🏻 we don't hit flaky test hell...

github-actions · 2023-05-22T19:54:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/15943/
CommitID: faabc42

…7597) In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake. We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier. As a bonus, we'll be able to reuse processor names across different types of processor. Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit eb51f2b) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…tories by type (#7669) * [Search Pipelines] Split search pipeline processor factories by type (#7597) In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake. We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier. As a bonus, we'll be able to reuse processor names across different types of processor. Signed-off-by: Michael Froh <froh@amazon.com> (cherry picked from commit eb51f2b) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Update SearchPipelineInfo serialization check to < 2.8 We needed to check for onOrBefore 2.8 in order to pass back-compat tests on main. Now that we're backporting to 2.8, we need to update the check (on 2.x and on main). Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#7597) In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake. We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier. As a bonus, we'll be able to reuse processor names across different types of processor. Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com>

…pensearch-project#7597) In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake. We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier. As a bonus, we'll be able to reuse processor names across different types of processor. Signed-off-by: Michael Froh <froh@amazon.com>

…pensearch-project#7597) In the initial search pipelines commit, I threw request and response processor factories into one combined map. I think that was a mistake. We should embrace type-safety by making sure that the kind of processor is clear from end to end. As we add more processor types (e.g. search phase processor), throwing them all in one big map would get messier. As a bonus, we'll be able to reuse processor names across different types of processor. Signed-off-by: Michael Froh <froh@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

msfroh force-pushed the search_pipelines_namespaces branch from 139e591 to 76e3245 Compare May 17, 2023 17:40

msfroh commented May 17, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineInfo.java Show resolved Hide resolved

msfroh marked this pull request as ready for review May 17, 2023 18:20

msfroh requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng and VachaShah as code owners May 17, 2023 18:20

msfroh changed the title ~~Split search pipeline processor factories by type~~ [Search Pipelines] Split search pipeline processor factories by type May 18, 2023

msfroh force-pushed the search_pipelines_namespaces branch from 4a5c9bc to 63f0fa7 Compare May 19, 2023 18:06

nknize reviewed May 22, 2023

View reviewed changes

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java Show resolved Hide resolved

server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Show resolved Hide resolved

msfroh added 4 commits May 22, 2023 19:15

Add tests for SearchPipelineInfo

4354ffd

Signed-off-by: Michael Froh <froh@amazon.com>

Add processor type to RenameFieldResponseProcessor.Factory

da7baac

Signed-off-by: Michael Froh <froh@amazon.com>

Ensure processor lists are unmodifiable

faabc42

Thanks @nknize! Signed-off-by: Michael Froh <froh@amazon.com>

msfroh force-pushed the search_pipelines_namespaces branch from 63f0fa7 to faabc42 Compare May 22, 2023 19:15

nknize approved these changes May 22, 2023

View reviewed changes

nknize merged commit eb51f2b into opensearch-project:main May 22, 2023

msfroh added the backport 2.x Backport to 2.x branch label May 22, 2023

opensearch-trigger-bot bot mentioned this pull request May 22, 2023

[Backport 2.x] [Search Pipelines] Split search pipeline processor factories by type #7667

Closed

This was referenced May 22, 2023

Adding the SearchPhaseResultsProcessor interface in Search Pipeline #7283

Merged

[Backport 2.x] [Search Pipelines] Split search pipeline processor factories by type #7669

Merged

Update SearchPipelineInfo backcompat check #7672

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Search Pipelines] Split search pipeline processor factories by type #7597

[Search Pipelines] Split search pipeline processor factories by type #7597

msfroh commented May 17, 2023 •

edited

Loading

github-actions bot commented May 17, 2023

github-actions bot commented May 17, 2023

github-actions bot commented May 17, 2023

codecov bot commented May 17, 2023 •

edited

Loading

github-actions bot commented May 18, 2023

github-actions bot commented May 19, 2023

msfroh commented May 19, 2023

msfroh commented May 19, 2023

macohen commented May 21, 2023

nknize left a comment

nknize commented May 22, 2023

github-actions bot commented May 22, 2023

nknize commented May 22, 2023

nknize left a comment

github-actions bot commented May 22, 2023

[Search Pipelines] Split search pipeline processor factories by type #7597

[Search Pipelines] Split search pipeline processor factories by type #7597

Conversation

msfroh commented May 17, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 17, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented May 17, 2023 • edited Loading

Codecov Report

github-actions bot commented May 18, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented May 19, 2023

Gradle Check (Jenkins) Run Completed with:

msfroh commented May 19, 2023

msfroh commented May 19, 2023

macohen commented May 21, 2023

nknize left a comment

Choose a reason for hiding this comment

nknize commented May 22, 2023

github-actions bot commented May 22, 2023

Gradle Check (Jenkins) Run Completed with:

nknize commented May 22, 2023

nknize left a comment

Choose a reason for hiding this comment

github-actions bot commented May 22, 2023

Gradle Check (Jenkins) Run Completed with:

msfroh commented May 17, 2023 •

edited

Loading

codecov bot commented May 17, 2023 •

edited

Loading