Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Support "search-pipeline" in Multi-Search API #15748

Closed
DougTrajano opened this issue Sep 5, 2024 · 7 comments · Fixed by #15923
Closed

[Feature Request] Support "search-pipeline" in Multi-Search API #15748

DougTrajano opened this issue Sep 5, 2024 · 7 comments · Fixed by #15923
Assignees
Labels
enhancement Enhancement or improvement to existing feature or request good first issue Good for newcomers Roadmap:Search Project-wide roadmap label Search Search query, autocomplete ...etc v2.18.0 Issues and PRs related to version 2.18.0

Comments

@DougTrajano
Copy link

Is your feature request related to a problem? Please describe

When using Hybrid Search, we must configure a Search Pipeline with a normalization-processor and inform it in the Search API operation using the search_pipeline=<search-pipeline-name> query parameter.

The Multi-search does not support this parameter, which means that we need to specify the Search Pipeline as a default search pipeline in the index. It's ok for some use cases, but it limits our capability to experiment with multiple normalization-processor.

Describe the solution you'd like

I want to have the search_pipeline in the URL parameters and metadata options of the Multi-search API operation.

Related component

Search

Describe alternatives you've considered

The current way to run multiple hybrid search objects using the Multi-search API operation is by defining the search pipeline as a default search pipeline in the index.

Additional context

No response

@DougTrajano DougTrajano added enhancement Enhancement or improvement to existing feature or request untriaged labels Sep 5, 2024
@github-actions github-actions bot added the Search Search query, autocomplete ...etc label Sep 5, 2024
@epugh
Copy link

epugh commented Sep 10, 2024

I just stumbled across the same situation. In my case, I'm using a ReactiveSearch library that assumes you are using the _msearch end point. I don't want to redo the entire communication layer just to use a pipeline, and I can't assume that a default pipeline makes sense since I want to experiment with different settings.

@msfroh
Copy link
Collaborator

msfroh commented Sep 10, 2024

@martin-gaievski and I were discussing this very idea a few weeks ago. Also, it came up in a chat with @joellerobinson at Haystack this year after her talk about combining results from multiple indices.

Multi-search opens up some interesting options for search pipelines. You could apply the same pipeline to all requests or apply different pipelines to each individual request. On top of that, you could do a "multi-request" or "multi-response" processor that transforms the full set of requests or responses. In particular, a "multi-response" processor could do result interleaving.

That said, @DougTrajano and @epugh -- I'm assuming that being able to specify a single search pipeline in the URL and having it apply to all requests (and their responses) would be sufficient. Is that correct?

@epugh
Copy link

epugh commented Sep 10, 2024

A single search pipeline would at least bring the overall "feel" of the Multi-search more in line with regular search. It was quite a surprise to me. It would at least solve my specific problem!

Having said that, being able to pass these types of parameters as either url OR as part of the query seems like a "good thing". The search pipeline is a core part of how you express your query, it isn't just an add on ;-). Especially in the direction that we appear to be going in with more and more ranker type features enabled via the pipeline!

@DougTrajano
Copy link
Author

@msfroh specifying it in the URL meets my use case, but I believe it will not cover additional use cases when different queries are performed.

To complete what @epugh said, since we will need to implement this, I believe it's more beneficial to add it at the query level.

@msfroh msfroh removed the untriaged label Sep 11, 2024
@msfroh
Copy link
Collaborator

msfroh commented Sep 11, 2024

[Search Triage] This might actually be a "Good First Issue". We have the ability to specify an ad-hoc pipeline as a JSON object. I believe that when we parse the search request, we could check for a JSON string instead of an object. If it's a string, move it into the search request (as though it was a URL parameter).

@owaiskazi19
Copy link
Member

@DougTrajano @epugh @msfroh raised #15923. Let me know what you all think.

@DougTrajano
Copy link
Author

@DougTrajano @epugh @msfroh raised #15923. Let me know what you all think.

Hey buddy! It looks great. Good work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request good first issue Good for newcomers Roadmap:Search Project-wide roadmap label Search Search query, autocomplete ...etc v2.18.0 Issues and PRs related to version 2.18.0
Projects
Status: New
Archived in project
Development

Successfully merging a pull request may close this issue.

5 participants