[Backport 2.x] Initial search pipelines implementation (#6587) #7075

msfroh · 2023-04-10T20:34:01Z

Initial search pipelines implementation

This commit includes the basic features of search pipelines (see opensearch-project/search-processor#80).

Search pipelines are modeled after ingest pipelines and provide a simple, clean API for components to modify search requests and responses.

With this commit we can:

Can create, retrieve, update, and delete search pipelines.
Transform search requests and responses by explicitly referencing a pipeline.

Later work will include:

Adding an index setting to specify a default search pipeline.
Allowing search pipelines to be defined within a search request (for development/testing purposes, akin to simulating an ingest pipeline).
Adding a collection of search pipeline processors to support common useful transformations. (Suggestions welcome!)

Signed-off-by: Michael Froh froh@amazon.com

Incorporate feedback from @reta and @navneet1v

SearchPipelinesClient: JavaDoc fix
SearchRequest: Check versions when (de)serializing new "pipeline" property.
Rename SearchPipelinesPlugin -> SearchPipelinePlugin.
Pipeline: Change visibility to package private
SearchPipelineProcessingException: New exception type to wrap exceptions thrown when executing a pipeline.

Bonus: Added an integration test for filter_query request processor.

Signed-off-by: Michael Froh froh@amazon.com

Register SearchPipelineProcessingException

Also added more useful messages to unit tests to explicitly explain what hoops need to be jumped through in order to add a new serializable exception.

Signed-off-by: Michael Froh froh@amazon.com

Remove unneeded dependencies from search-pipeline-common

I had copied some dependencies from ingest-common, but they are not used by search-pipeline-common (yet).

Signed-off-by: Michael Froh froh@amazon.com

Avoid cloning SearchRequest if no SearchRequestProcessors

Also, add tests to confirm that a pipeline with no processors works fine (as a no-op).

Signed-off-by: Michael Froh froh@amazon.com

Use NamedWritableRegistry to deserialize SearchRequest

Queries are serialized as NamedWritables, so we need to use a NamedWritableRegistry to deserialize.

Signed-off-by: Michael Froh froh@amazon.com

Check for empty pipeline with CollectionUtils.isEmpty

Signed-off-by: Michael Froh froh@amazon.com

Update server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java

Co-authored-by: Navneet Verma vermanavneet003@gmail.com
Signed-off-by: Michael Froh froh@amazon.com

Incorporate feedback from @noCharger

Signed-off-by: Michael Froh froh@amazon.com

Incorporate feedback from @reta

Renamed various classes from "SearchPipelinesSomething" to "SearchPipelineSomething" to be consistent.
Refactored NodeInfo construction in NodeService to avoid ternary operator and improved readability.

Signed-off-by: Michael Froh froh@amazon.com

Gate search pipelines behind a feature flag

Also renamed SearchPipelinesRequestConverters.

Signed-off-by: Michael Froh froh@amazon.com

More feature flag fixes for search pipeline testing

Don't use system properties for SearchPipelineServiceTests.
Enable feature flag for multinode smoke tests.

Signed-off-by: Michael Froh froh@amazon.com

Move feature flag into constructor parameter

Thanks for the suggestion, @reta!

Signed-off-by: Michael Froh froh@amazon.com

Move REST handlers behind feature flag

Signed-off-by: Michael Froh froh@amazon.com

Signed-off-by: Michael Froh froh@amazon.com
Co-authored-by: Navneet Verma vermanavneet003@gmail.com
(cherry picked from commit ee990bd)

@reta

* Initial search pipelines implementation This commit includes the basic features of search pipelines (see opensearch-project/search-processor#80). Search pipelines are modeled after ingest pipelines and provide a simple, clean API for components to modify search requests and responses. With this commit we can: 1. Can create, retrieve, update, and delete search pipelines. 2. Transform search requests and responses by explicitly referencing a pipeline. Later work will include: 1. Adding an index setting to specify a default search pipeline. 2. Allowing search pipelines to be defined within a search request (for development/testing purposes, akin to simulating an ingest pipeline). 3. Adding a collection of search pipeline processors to support common useful transformations. (Suggestions welcome!) Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @reta and @navneet1v 1. SearchPipelinesClient: JavaDoc fix 2. SearchRequest: Check versions when (de)serializing new "pipeline" property. 3. Rename SearchPipelinesPlugin -> SearchPipelinePlugin. 4. Pipeline: Change visibility to package private 5. SearchPipelineProcessingException: New exception type to wrap exceptions thrown when executing a pipeline. Bonus: Added an integration test for filter_query request processor. Signed-off-by: Michael Froh <froh@amazon.com> * Register SearchPipelineProcessingException Also added more useful messages to unit tests to explicitly explain what hoops need to be jumped through in order to add a new serializable exception. Signed-off-by: Michael Froh <froh@amazon.com> * Remove unneeded dependencies from search-pipeline-common I had copied some dependencies from ingest-common, but they are not used by search-pipeline-common (yet). Signed-off-by: Michael Froh <froh@amazon.com> * Avoid cloning SearchRequest if no SearchRequestProcessors Also, add tests to confirm that a pipeline with no processors works fine (as a no-op). Signed-off-by: Michael Froh <froh@amazon.com> * Use NamedWritableRegistry to deserialize SearchRequest Queries are serialized as NamedWritables, so we need to use a NamedWritableRegistry to deserialize. Signed-off-by: Michael Froh <froh@amazon.com> * Check for empty pipeline with CollectionUtils.isEmpty Signed-off-by: Michael Froh <froh@amazon.com> * Update server/src/main/java/org/opensearch/search/pipeline/SearchPipelineService.java Co-authored-by: Navneet Verma <vermanavneet003@gmail.com> Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @noCharger Signed-off-by: Michael Froh <froh@amazon.com> * Incorporate feedback from @reta - Renamed various classes from "SearchPipelinesSomething" to "SearchPipelineSomething" to be consistent. - Refactored NodeInfo construction in NodeService to avoid ternary operator and improved readability. Signed-off-by: Michael Froh <froh@amazon.com> * Gate search pipelines behind a feature flag Also renamed SearchPipelinesRequestConverters. Signed-off-by: Michael Froh <froh@amazon.com> * More feature flag fixes for search pipeline testing - Don't use system properties for SearchPipelineServiceTests. - Enable feature flag for multinode smoke tests. Signed-off-by: Michael Froh <froh@amazon.com> * Move feature flag into constructor parameter Thanks for the suggestion, @reta! Signed-off-by: Michael Froh <froh@amazon.com> * Move REST handlers behind feature flag Signed-off-by: Michael Froh <froh@amazon.com> --------- Signed-off-by: Michael Froh <froh@amazon.com> Co-authored-by: Navneet Verma <vermanavneet003@gmail.com> (cherry picked from commit ee990bd)

github-actions · 2023-04-10T20:41:16Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13788/
CommitID: a6c16d5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

msfroh · 2023-04-10T21:09:10Z

Oops -- looks like the merge misunderstood how to handle these version checks:

        if (in.getVersion().onOrAfter(LegacyESVersion.V_7_10_0)) {
            addInfoIfNonNull(AggregationInfo.class, in.readOptionalWriteable(AggregationInfo::new));
        if (in.getVersion().onOrAfter(Version.V_3_0_0)) { // TODO: Change if/when we backport to 2.x
            addInfoIfNonNull(SearchPipelineInfo.class, in.readOptionalWriteable(SearchPipelineInfo::new));
        }
        }

I'll go back and pull the one if out of the other.

1. Can't reference version 3.0.0. 2. Bad merges of adjacent version checks. 3. Use of Apache HTTP client 4 (vs 5). 4. Use of old cluster manager naming in REST params. 5. CollectionUtils didn't have isEmpty for collections. Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-04-10T22:12:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13801/
CommitID: c1e8cea
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-10T22:20:51Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/13802/
CommitID: 744d5f9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Michael Froh <froh@amazon.com>

github-actions · 2023-04-10T23:26:02Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica
      1 org.opensearch.indices.replication.SegmentReplicationIT.testReplicationPostDeleteAndForceMerge

URL: https://build.ci.opensearch.org/job/gradle-check/13808/
CommitID: 5ba734e
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-04-10T23:55:09Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=pit/10_basic/Delete all}

URL: https://build.ci.opensearch.org/job/gradle-check/13814/
CommitID: 4126cfc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov-commenter · 2023-04-10T23:59:41Z

Codecov Report

Merging #7075 (4126cfc) into 2.x (932f47b) will increase coverage by 0.46%.
The diff coverage is 53.37%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##                2.x    #7075      +/-   ##
============================================
+ Coverage     70.35%   70.81%   +0.46%     
- Complexity    59492    60043     +551     
============================================
  Files          4822     4849      +27     
  Lines        285965   286731     +766     
  Branches      41562    41649      +87     
============================================
+ Hits         201178   203042    +1864     
+ Misses        67995    67079     -916     
+ Partials      16792    16610     -182

Impacted Files	Coverage Δ
...search/client/SearchPipelineRequestConverters.java	`0.00% <0.00%> (ø)`
...eline/common/SearchPipelineCommonModulePlugin.java	`0.00% <0.00%> (ø)`
...ion/admin/cluster/node/info/NodesInfoResponse.java	`3.03% <0.00%> (-0.10%)`	⬇️
...search/action/search/GetSearchPipelineRequest.java	`0.00% <0.00%> (ø)`
...earch/action/search/GetSearchPipelineResponse.java	`0.00% <0.00%> (ø)`
.../org/opensearch/client/support/AbstractClient.java	`31.33% <0.00%> (-0.43%)`	⬇️
...search/cluster/service/ClusterManagerTaskKeys.java	`0.00% <ø> (ø)`
...pensearch/common/settings/FeatureFlagSettings.java	`50.00% <ø> (ø)`
...va/org/opensearch/common/util/CollectionUtils.java	`78.87% <0.00%> (-0.56%)`	⬇️
...a/org/opensearch/plugins/SearchPipelinePlugin.java	`0.00% <0.00%> (ø)`
... and 37 more

... and 478 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

reta · 2023-04-11T01:29:14Z

server/src/main/java/org/opensearch/action/admin/cluster/node/info/NodeInfo.java

@@ -102,6 +103,9 @@ public NodeInfo(StreamInput in) throws IOException {
        if (in.getVersion().onOrAfter(LegacyESVersion.V_7_10_0)) {
            addInfoIfNonNull(AggregationInfo.class, in.readOptionalWriteable(AggregationInfo::new));
        }
+        if (in.getVersion().onOrAfter(Version.V_2_7_0)) {


@msfroh we would need to bring these changes to main as well, could you please prepare the pull request so we will merge it right after this one? thank you

Done, thanks! Here we go: #7135

msfroh · 2023-04-11T15:54:55Z

I'm canceling this PR for now, just to make sure it doesn't go out in its current state for 2.7. I'll see about reopening after 2.7 is released.

msfroh · 2023-04-12T23:17:02Z

Talked it over with a colleague, who mentioned that the feature flag is enough that it should be safe to ship this code in 2.7.

github-actions · 2023-04-12T23:52:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=pit/10_basic/Delete all}

URL: https://build.ci.opensearch.org/job/gradle-check/13968/
CommitID: 4126cfc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

reta · 2023-04-13T00:26:17Z

@andrross just to confirm, good to go for 2.7.0? thank you

saratvemulapalli · 2023-04-13T21:16:39Z

Talked it over with a colleague, who mentioned that the feature flag is enough that it should be safe to ship this code in 2.7.

Late here. Im the mystery colleague. As long as we dont break semver and the feature is gated we should be good.

msfroh requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners April 10, 2023 20:34

reta approved these changes Apr 10, 2023

View reviewed changes

msfroh force-pushed the backport/backport-6587-to-2.x branch from c1e8cea to 744d5f9 Compare April 10, 2023 21:48

Support deprecated master_timeout parameter

4126cfc

Signed-off-by: Michael Froh <froh@amazon.com>

msfroh force-pushed the backport/backport-6587-to-2.x branch from 5ba734e to 4126cfc Compare April 10, 2023 23:12

reta reviewed Apr 11, 2023

View reviewed changes

msfroh closed this Apr 11, 2023

msfroh reopened this Apr 12, 2023

msfroh mentioned this pull request Apr 13, 2023

[SearchPipelines] Lower compatibility version to 2.7 #7135

Merged

6 tasks

andrross approved these changes Apr 13, 2023

View reviewed changes

reta merged commit 6024485 into opensearch-project:2.x Apr 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] Initial search pipelines implementation (#6587) #7075

[Backport 2.x] Initial search pipelines implementation (#6587) #7075

msfroh commented Apr 10, 2023

github-actions bot commented Apr 10, 2023

msfroh commented Apr 10, 2023 •

edited

Loading

github-actions bot commented Apr 10, 2023

github-actions bot commented Apr 10, 2023

github-actions bot commented Apr 10, 2023

github-actions bot commented Apr 10, 2023

codecov-commenter commented Apr 10, 2023 •

edited

Loading

reta Apr 11, 2023

msfroh Apr 13, 2023

msfroh commented Apr 11, 2023

msfroh commented Apr 12, 2023

github-actions bot commented Apr 12, 2023

reta commented Apr 13, 2023

saratvemulapalli commented Apr 13, 2023

[Backport 2.x] Initial search pipelines implementation (#6587) #7075

[Backport 2.x] Initial search pipelines implementation (#6587) #7075

Conversation

msfroh commented Apr 10, 2023

github-actions bot commented Apr 10, 2023

Gradle Check (Jenkins) Run Completed with:

msfroh commented Apr 10, 2023 • edited Loading

github-actions bot commented Apr 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 10, 2023

Gradle Check (Jenkins) Run Completed with:

codecov-commenter commented Apr 10, 2023 • edited Loading

Codecov Report

reta Apr 11, 2023

Choose a reason for hiding this comment

msfroh Apr 13, 2023

Choose a reason for hiding this comment

msfroh commented Apr 11, 2023

msfroh commented Apr 12, 2023

github-actions bot commented Apr 12, 2023

Gradle Check (Jenkins) Run Completed with:

reta commented Apr 13, 2023

saratvemulapalli commented Apr 13, 2023

msfroh commented Apr 10, 2023 •

edited

Loading

codecov-commenter commented Apr 10, 2023 •

edited

Loading