Fixed Hybrid query for cases when it's wrapped into other compound queries

[martin-gaievski]

Description

Nested fields have few problems that we're addressing in this PR:

• query is wrapped into the Bool query inside the core code (ref), that happening behind the scenes and depends only on presents of the field in the index mapping. This case should work.

• wrapping up hybrid search query into another query in general. That is not how hybrid query were designed, it must be a top level query. We need to track and block such queries.

We're mainly addressing case 1, and trying our best for case 2.

Below is snippet of example from original github issue to illustrate case 1. Without this PR search query returns wrong scores:

index mapping

PUT /index-test
{
    "settings": {
        "index.knn": true,
        "number_of_shards": 2,
        "default_pipeline": "pipeline-test"
    },
    "mappings": {
        "properties": {
            "id": {
                "type": "text"
            },
            "passage_embedding": {
                "type": "knn_vector",
                "dimension": 768,
                "method": {
                    "name": "hnsw",
                    "space_type": "cosinesimil",
                    "engine": "lucene",
                    "parameters": {
                        "ef_construction": 512,
                        "m": 8
                    }
                }
            },
            "name": {
                "type": "text"
            },
            "passage_text": {
                "type": "text"
            },
            "test": {
                "type": "text"
            },
            "user": {
                "type": "nested"
            }
        }
    }
}

add doc to index

POST /index-test/_doc/1
{
    "name": "A West Virginia university women 's basketball team , officials , and a small gathering of fans are in a West Virginia arena .",
    "id": "4319130149.jpg"
}

search request

GET /index-test/_search?search_pipeline=nlp-search-pipeline

{
    "_source": {
        "exclude": [
            "passage_embedding"
        ]
    },
    "query": {
        "hybrid": {
            "queries": [
                {
                    "match": {
                        "name": {
                            "query": "wild west"
                        }
                    }
                },
                {
                    "neural": {
                        "passage_embedding": {
                            "query_text": "wild west",
                            "model_id": "{{model_id}}",
                            "k": 20
                        }
                    }
                }
            ]
        }
    }
}

Before this change this is the search query response that system can return

{
    "took": 1081,
    "timed_out": false,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 5,
            "relation": "eq"
        },
        "max_score": 2.3997707,
        "hits": [
            {
                "_index": "index-test",
                "_id": "1",
                "_score": 2.3997707,
                "_source": {
                    "name": "A West Virginia university women 's basketball team , officials , and a small gathering of fans are in a West Virginia arena .",
                    "id": "4319130149.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "2",
                "_score": 1.6187637,
                "_source": {
                    "name": "A wild animal races across an uncut field with a minimal amount of trees .",
                    "id": "1775029934.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "5",
                "_score": 1.5765835,
                "_source": {
                    "name": "A rodeo cowboy , wearing a cowboy hat , is being thrown off of a wild white horse .",
                    "id": "2691147709.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "4",
                "_score": 1.1599709,
                "_source": {
                    "name": "A man who is riding a wild horse in the rodeo is very near to falling off .",
                    "id": "4427058951.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "3",
                "_score": 0.82945794,
                "_source": {
                    "name": "People line the stands which advertise Freemont 's orthopedics , a cowboy rides a light brown bucking bronco .",
                    "id": "2664027527.jpg"
                }
            }
        ]
    }
}

below is response from system with this fix

{
    "took": 1079,
    "timed_out": false,
    "_shards": {
        "total": 2,
        "successful": 2,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 5,
            "relation": "eq"
        },
        "max_score": 0.82329726,
        "hits": [
            {
                "_index": "index-test",
                "_id": "1",
                "_score": 0.82329726,
                "_source": {
                    "name": "A West Virginia university women 's basketball team , officials , and a small gathering of fans are in a West Virginia arena .",
                    "id": "4319130149.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "2",
                "_score": 0.5499638,
                "_source": {
                    "name": "A wild animal races across an uncut field with a minimal amount of trees .",
                    "id": "1775029934.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "5",
                "_score": 0.49218822,
                "_source": {
                    "name": "A rodeo cowboy , wearing a cowboy hat , is being thrown off of a wild white horse .",
                    "id": "2691147709.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "4",
                "_score": 0.3007,
                "_source": {
                    "name": "A man who is riding a wild horse in the rodeo is very near to falling off .",
                    "id": "4427058951.jpg"
                }
            },
            {
                "_index": "index-test",
                "_id": "3",
                "_score": 3.0E-4,
                "_source": {
                    "name": "People line the stands which advertise Freemont 's orthopedics , a cowboy rides a light brown bucking bronco .",
                    "id": "2664027527.jpg"
                }
            }
        ]
    }
}

Issues Resolved

#466

Check List

• New functionality includes testing.
  • All tests pass
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (b3c73bd) 84.65% compared to head (dbac80e) 84.24%.

❗ Current head dbac80e differs from pull request most recent head 379478a. Consider uploading reports for the commit 379478a to get more accurate results

Files	Patch %	Lines
...lsearch/search/query/HybridQueryPhaseSearcher.java	71.11%	1 Missing and 12 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #498      +/-   ##
============================================
- Coverage     84.65%   84.24%   -0.41%     
- Complexity      508      529      +21     
============================================
  Files            40       40              
  Lines          1505     1549      +44     
  Branches        234      245      +11     
============================================
+ Hits           1274     1305      +31     
- Misses          128      129       +1     
- Partials        103      115      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[navneet1v]

@martin-gaievski Please add the index mapping and query that you tested via curl or postman in the issue description so that we understand how this was tested.

[navneet1v]

@martin-gaievski fix the title of the PR it doesn't provide info around what this PR is about.

Other than that code look good to me

[martin-gaievski]

@martin-gaievski fix the title of the PR it doesn't provide info around what this PR is about.

Other than that code look good to me

fixed the title @navneet1v

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-498-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 3991fe7c04e2f2c104c0be0365428c5747f21f6c
# Push it to GitHub
git push --set-upstream origin backport/backport-498-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-498-to-2.x.