Reciprocal Rank Fusion (RRF) normalization technique in hybrid query

[Johnsonisaacn]

Description

Adding ability to process and combine scores from multiple subqueries in neural search using the reciprocal rank fusion (RRF) technique. Built with a new processor and processor factory class apart from NormalizationProcessor. Changes to API included in RFC. Does not currently support weights when combining processed subquery scores, based on lack of examples in existing literature.

Example of usage for RRF processor:

create index

PUT /index-test
{
  "settings": {
    "index": {
      "knn": true
    }
  },
  "mappings": {
    "properties": {
      "vector": {
        "type": "knn_vector",
        "dimension": 3,
        "method": {
          "name": "hnsw",
          "space_type": "l2",
          "engine": "lucene"
        }
      },
      "field1": {
        "type": "integer"
      }
    }
  }
}

create pipeline with rrf processor and all defaults

PUT /_search/pipeline/nlp-search-pipeline
{
    "description": "Post processor for hybrid search",
    "phase_results_processors": [
        {
            "score-ranker-processor": {
                "combination": {
                    "technique": "rrf",
                    "parameters": {
                    }
                }
            }
        }
    ]
}

ingest 4 documents

POST /index-test/_doc/?refresh=true
{
    "field1": 2,
    "vector": [0.4, 0.5, 0.2],
    "title": "basic"
}

{
    "field1": 10,
    "vector": [0.2, 0.2, 0.3],
    "title": "java"
}

{
    "field1": 50,
    "vector": [4.2, 5.5, 8.9]
}

{
    "vector": [0.3, 0.12, 3.3],
    "title": "python"
}

run search request

GET /index-test/_search?search_pipeline=nlp-search-pipeline
{
    "query": {
        "hybrid": {
            "queries": [
                {
                    "knn": {
                        "vector": {
                            "vector": [
                                4.2,
                                5.0,
                                8.5
                            ],
                            "k": 10
                        }
                    }
                },
                {
                    "range": {
                        "field1": {
                            "gte": 10,
                            "lte": 50
                        }
                    }
                }
            ]
        }
    }
}

you'll get following response

{
    "took": 6,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 4,
            "relation": "eq"
        },
        "max_score": 0.032522473,
        "hits": [
            {
                "_index": "index-test",
                "_id": "fSgBmJIB3rlMI6kPNIQL",
                "_score": 0.032522473,
                "_source": {
                    "field1": 50,
                    "vector": [
                        4.2,
                        5.5,
                        8.9
                    ]
                }
            },
            {
                "_index": "index-test",
                "_id": "fCgBmJIB3rlMI6kPK4QS",
                "_score": 0.03201844,
                "_source": {
                    "field1": 10,
                    "vector": [
                        0.2,
                        0.2,
                        0.3
                    ],
                    "title": "java"
                }
            },
            {
                "_index": "index-test",
                "_id": "figBmJIB3rlMI6kPPITH",
                "_score": 0.016129032,
                "_source": {
                    "vector": [
                        0.3,
                        0.12,
                        3.3
                    ],
                    "title": "python"
                }
            },
            {
                "_index": "index-test",
                "_id": "eygBmJIB3rlMI6kPIYQm",
                "_score": 0.015873017,
                "_source": {
                    "field1": 2,
                    "vector": [
                        0.4,
                        0.5,
                        0.2
                    ],
                    "title": "basic"
                }
            }
        ]
    }
}

if you change rank to something smaller, like '1' your scores all will be scalled up
update rank contant

{
    "description": "Post processor for hybrid search",
    "phase_results_processors": [
        {
            "score-ranker-processor": {
                "combination": {
                    "technique": "rrf",
                    "parameters": {
                        "rank_constant": 1
                    }
                }
            }
        }
    ]
}

and search response is

{
    "took": 10,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 4,
            "relation": "eq"
        },
        "max_score": 0.8333334,
        "hits": [
            {
                "_index": "index-test",
                "_id": "fSgBmJIB3rlMI6kPNIQL",
                "_score": 0.8333334,
                "_source": {
                    "field1": 50,
                    "vector": [
                        4.2,
                        5.5,
                        8.9
                    ]
                }
            },
            {
                "_index": "index-test",
                "_id": "fCgBmJIB3rlMI6kPK4QS",
                "_score": 0.7,
                "_source": {
                    "field1": 10,
                    "vector": [
                        0.2,
                        0.2,
                        0.3
                    ],
                    "title": "java"
                }
            },
            {
                "_index": "index-test",
                "_id": "figBmJIB3rlMI6kPPITH",
                "_score": 0.33333334,
                "_source": {
                    "vector": [
                        0.3,
                        0.12,
                        3.3
                    ],
                    "title": "python"
                }
            },
            {
                "_index": "index-test",
                "_id": "eygBmJIB3rlMI6kPIYQm",
                "_score": 0.25,
                "_source": {
                    "field1": 2,
                    "vector": [
                        0.4,
                        0.5,
                        0.2
                    ],
                    "title": "basic"
                }
            }
        ]
    }
}

for comparison this is the response for same query if we use normalization processor with default techniques.

Important difference is that delta between document scores with RRF is much smaller, this is because it's based on document rank that are typically close in value comparing to scores where delta can be huge.

{
    "took": 16,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 4,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [
            {
                "_index": "index-test",
                "_id": "fSgBmJIB3rlMI6kPNIQL",
                "_score": 1.0,
                "_source": {
                    "field1": 50,
                    "vector": [
                        4.2,
                        5.5,
                        8.9
                    ]
                }
            },
            {
                "_index": "index-test",
                "_id": "fCgBmJIB3rlMI6kPK4QS",
                "_score": 0.5005,
                "_source": {
                    "field1": 10,
                    "vector": [
                        0.2,
                        0.2,
                        0.3
                    ],
                    "title": "java"
                }
            },
            {
                "_index": "index-test",
                "_id": "figBmJIB3rlMI6kPPITH",
                "_score": 0.0039931787,
                "_source": {
                    "vector": [
                        0.3,
                        0.12,
                        3.3
                    ],
                    "title": "python"
                }
            },
            {
                "_index": "index-test",
                "_id": "eygBmJIB3rlMI6kPIYQm",
                "_score": 1.7192177E-4,
                "_source": {
                    "field1": 2,
                    "vector": [
                        0.4,
                        0.5,
                        0.2
                    ],
                    "title": "basic"
                }
            }
        ]
    }
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]
#865
#659

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[martin-gaievski]

we should be merging to feature branch https://github.com/opensearch-project/neural-search/tree/feature/rrf-score-normalization, not main.

[yuye-aws]

This class not completed?

[martin-gaievski]

we planned to have very simple implementation for this one, I'll be finishing this PR and address all misses if

[martin-gaievski]

I finished the class, please take a look @yuye-aws

[vibrantvarun]

Can't we shift the code in this class to HybridQueryUtil?

[martin-gaievski]

I don't think it belongs there, my view is - anything related to the query itself should go to that class, like parsing score collection into multiple sub query results.

[vibrantvarun]

Suggested change

	* in NormalizationProcessorWorkflow. Field rankConstant
	* in NormalizationProcessorWorkflow.

[vibrantvarun]

Just one minor comment.

LGTM.

[yuye-aws]

Is there somewhere to validate that RRFNormalizationTechnique is used together with RRFScoreCombinationTechnique? The execute method in NormalizationProcessorWorkflow class doing normalization and them combination.

[yuye-aws]

Why are you adding these method in this testing? I think you can simply with a few examples like 1 plus 1 is 2.

[martin-gaievski]

it's added to be compatible with https://github.com/opensearch-project/neural-search/blob/main/src/test/java/org/opensearch/neuralsearch/processor/combination/BaseScoreCombinationTechniqueTests.java and be able to use all test cases it provides. We need to ensure in better possible test coverage if it's a low hanging fruit

[yuye-aws]

Did not get your point. The private randomScore outputs non-deterministic results.

[yuye-aws]

Since you are already using big decimal, please remove the delta here

[martin-gaievski]

not sure what do you mean, assert requires third parameter in case we're comparing floats, and both arguments are float

[yuye-aws]

I mean you are using big decimal in the test rrfNorm method. You can be more strict, and the delta can be set to 0.

[martin-gaievski]

Is there somewhere to validate that RRFNormalizationTechnique is used together with RRFScoreCombinationTechnique? The execute method in NormalizationProcessorWorkflow class doing normalization and them combination.

We do not retrieve normalization technique from user input, it's hardcoded and passed to processor class by the factory, check out code snippet
https://github.com/Johnsonisaacn/neural-search/blob/RRF/src/main/java/org/opensearch/neuralsearch/processor/factory/RRFProcessorFactory.java#L51-L69

I want to keep NormalizationProcessorWorkflow generic, maybe later refactor it to more abstract class not specific to normalization.

[martin-gaievski]

I've addressed all comments, and most of them were minor in recent reviews. I'll be merging this one to feature branch and we'll start one more related to RRF soon, with focus on testing

[yuye-aws]

Nice work @martin-gaievski