#639: allow metadata fields and score opensearch function (#228)

[acarbonetto]

Description

OpenSearch reserved fields (_id, _index, _sort, _score, _max_score) are not allowed to be used in SQL clauses (SELECT, WHERE, ORDER BY) because the field format starting with underscore _ is not allowed.

• This ticket adds specific identifiers to the language, and opens up support for OpenSearch reserved identifiers.
• As an aside, identifiers with double underscore at the start (such as __myCoolField) is acceptable as an identifier.

This ticket allows for the score(), score_query() and scorequery() function to be used to wrap around relevance-search queries to force the _score and _max_score metadata fields to be returned with values. For some queries, _score returns null unless the score() function is included.

• The score function also allows for an optional boost argument to be included that boosts the score of the child relevance function.

Example - metadata fields returned:

SELECT calcs.key, str0, _id, _sort, _score, _maxscore FROM calcs WHERE _id="5"
Result:
{ "key04", "OFFICE SUPPLIES", "5", -2, null, null }

Example - Metadata fields not requested are not displayed:

SELECT *, _id FROM bigint WHERE _id="2"
Result (assuming bigint only has one field):
[9223372026854775807, "2"]

Example - relevance search without and with score function

SELECT _id, _index, _score, _maxscore, str0, str1 FROM calcs WHERE QUERY_STRING([\"*\"], `BINDING SUPPLIES`);
result: 
[
    "7",
    "calcs",
    null,
    null,
    "OFFICE SUPPLIES",
    "BINDING SUPPLIES"
]

SELECT _id, _index, _score, _maxscore, str0, str1 FROM calcs WHERE SCORE(QUERY_STRING([\"*\"], `BINDING SUPPLIES`));
[
    "7",
    "calcs",
    2.4849067,
    2,
    "OFFICE SUPPLIES",
    "BINDING SUPPLIES"
]

Example - boost score on the _sql/_explain call:

SELECT _id, _score, _maxscore FROM calcs WHERE SCORE(QUERY_STRING([\"*\"], `BINDING SUPPLIES`, boost=2.5));
result: 
[
    "7",
    6.2122664,
    6
]

SELECT _id, _score, _maxscore FROM calcs WHERE SCOREQUERY(QUERY_STRING([\"*\"], `BINDING SUPPLIES`, boost=2.5), 2.0);
result: 
[
    "7",
    12.424533,
    12
]

SELECT _id, _index, _score, _maxscore, str0, str1 FROM calcs WHERE SCORE_QUERY(QUERY_STRING([\"*\"], `BINDING SUPPLIES`), 2.5);
result:
[
    "7",
    6.2122664,
    6
]

Issues Resolved

• [BUG] _id is not available in where condition #639
• Support for _routing field in SQL queries #339 (TBD)
• [BUG] ORDER BY meta fields, such as _id, regression in ODFE 1.13 and removed in OpenSearch 1.0 #432
• [BUG] fields starting with __ is missing in the returned result when query by SQL with extra filter #783
• [BUG] Underscore in identifier is ignored by ANTLR parser #1133

Check List

• New functionality includes testing.
  • All tests pass, including unit test, integration test and doctest
• New functionality has been documented.
  • New functionality has javadoc added
  • New functionality has user manual doc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov-commenter]

Codecov Report

Merging #1456 (05c42f7) into main (23cc0f6) will increase coverage by 0.02%.
The diff coverage is 100.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

@@             Coverage Diff              @@
##               main    #1456      +/-   ##
============================================
+ Coverage     98.46%   98.49%   +0.02%     
- Complexity     3869     3928      +59     
============================================
  Files           345      347       +2     
  Lines          9603     9771     +168     
  Branches        616      645      +29     
============================================
+ Hits           9456     9624     +168     
  Misses          142      142              
  Partials          5        5              

Flag	Coverage Δ
sql-engine	98.49% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ain/java/org/opensearch/sql/analysis/Analyzer.java	100.00% <100.00%> (ø)
...rg/opensearch/sql/analysis/ExpressionAnalyzer.java	100.00% <100.00%> (ø)
...rch/sql/analysis/ExpressionReferenceOptimizer.java	100.00% <100.00%> (ø)
...a/org/opensearch/sql/analysis/TypeEnvironment.java	100.00% <100.00%> (ø)
...c/main/java/org/opensearch/sql/expression/DSL.java	100.00% <100.00%> (ø)
...h/sql/expression/function/BuiltinFunctionName.java	100.00% <100.00%> (ø)
...h/sql/expression/function/OpenSearchFunctions.java	100.00% <100.00%> (ø)
...rc/main/java/org/opensearch/sql/storage/Table.java	100.00% <100.00%> (ø)
...sql/opensearch/request/OpenSearchQueryRequest.java	100.00% <100.00%> (ø)
...l/opensearch/request/OpenSearchRequestBuilder.java	100.00% <100.00%> (ø)
... and 7 more

... and 4 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[MaxKsyunz]

@acarbonetto Do metafields work with aliases? E.g. SELECT h._id FROM tableA AS h

Is there a test that confirms this?

[acarbonetto]

@acarbonetto Do metafields work with aliases? E.g. SELECT h._id FROM tableA AS h

Is there a test that confirms this?

IT test for this added in f3d89b8

[dai-chen]

Does this comment match the rule below? So ID_LITERAL won't accept token starting with single underscore?

[acarbonetto]

Good catch! I need to update this comment since I've updated the pattern.
Thank you!

[Yury-Fridlyand]

May be do this in the query builder?

[acarbonetto]

OpenSearchQueryRequest and OpenSearchScrollRequest needs a refactor before this makes any sense.

[penghuo]

Please assist in creating an issue to monitor and manage future modifications.