Add FlatObject FieldMapper

[mingshl]

Description

To fulfill #1018, we implement the approach by store the entire nested object as a String. A flat_object creates exactly two internal Lucene StringField ( "._value" and "._valueAndPath" ) in regards of how many nested fields in the flat field.

Background

The current object field types available for JSON objects in OpenSearch are object , nested and join. However, these field types can create individual indexable fields for each component, resulting in a large number of fields. When dealing with complex JSON objects, mapping many individual fields can lead to heavy RAM usage and thrashing, ultimately causing a mapping explosion. Additionally, storing a large number of fields can take up valuable space, and migrating indexes from the system can be a difficult task.

To address this problem, we are introducing a new field type called flat-object. This new field type will flatten the index in mapping, meaning that the components will not be indexed. Instead, the values and their paths will be stored in two string fields, no matter how complex the JSON object may be. While the values within the JSON object can still be accessed in the flat field using standard dot path notation in DSL and SQL, they will not be indexed for faster lookup. This will provide a more efficient way of handling complex JSON objects and will ultimately help to improve performance.

High Level Design

• The FlatObjectFieldMapper stores the entire nested JSON object as a String. A flat-object field creates exactly two internal Lucene StringField ( "._value" and "._valueAndPath" ) in regards of how many nested fields the flat field has.
  catalog { ._value :
  ._valueAndPath }

  • value: a keyword field field contains the leaf values of all subfields, allowing for efficient searching of leaf values without specifying the path. (e.g: catalog = 'Mike').

  • valueAndPath: : a keyword field field contains the path to the leaf value and its value, enabling efficient searching when the query includes the path to the leaf. (e.g: catalog.author.given = 'Mike')

Supported Query:

Term query
Terms query
Termset query
Prefix query
Range query
Match query
Multi_match query
Query_string query
Simplequery_string query
Exists query

Performance Evaluation:

FlatObject can upload document with more than 1000 fields, while dynamicmapping cannot go above 1000 fields. FlatObject support searching values without dotpath(searching with global fieldname), while dynamic mapping requires searching with exact dot path.

In testing 100 runs, flatObject takes 15% more time in creating index, 6% to 9% slower when searching with dot path depending on the complexity of the nested JSON. But flatObject takes 6% less time in uploading documents.

Benchmark Mode Cnt Score Error Units
FlatObjectMappingBenchmark.CreateDynamicIndex avgt 100 222.730 ± 11.891 ms/op
FlatObjectMappingBenchmark.CreateFlatObjectIndex avgt 100 256.178 ± 9.294 ms/op
FlatObjectMappingBenchmark.indexDynamicMapping avgt 100 337.405 ± 15.383 ms/op
FlatObjectMappingBenchmark.indexFlatObjectMapping avgt 100 316.860 ± 17.534 ms/op
FlatObjectMappingBenchmark.searchDynamicMappingWithOneHundredNestedJSON avgt 100 281.482 ± 11.176 ms/op
FlatObjectMappingBenchmark.searchFlatObjectMappingInValueWithOneHundredNestedJSON avgt 100 308.626 ± 11.710 ms/op

Limitation and Future Development:

• enable searching in PainlessScript, we will need to direct the fielddatabuilder to fetch docvalues within the two stringfields in memory
• open parameters setting, such as normalizer, docValues, ignoreAbove, nullValue, similarity, and depthlimit.
• enable wildcard query

Issues Resolved

#1018

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/11799/
• CommitID: 334dd6f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/12059/
• CommitID: 67a367f
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/12739/
• CommitID: 3eb4cf5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/12771/
• CommitID: 44c3b5e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=pit/10_basic/Delete all}

• URL: https://build.ci.opensearch.org/job/gradle-check/12784/
• CommitID: 33209b7
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[mingshl]

@reta This FlatObjectfieldMapper is default using Lucene.KEYWORD_ANALYZER, if you look at the method that initiate the FlatObjectFieldType in here and here, the rest of the public FlatObjectFieldType are used in different tests file and they are useful.

If you look at the keywordfieldmapper, it's similar design, the only difference is that it's extending from ParamterizedFieldMapper, so it accepts open parameters for normalizer, but it's also using the same Lucene.KEYWORD_ANALYZER.

So it shouldn't be a surprise because it only does KEYWORD_ANALYZER in this field mapper, which only do exact match and will not allow tokenizers.

I hope this help.

[reta]

@mingshl I am confused now

So it shouldn't be a surprise because it only does KEYWORD_ANALYZER in this field mapper, which only do exact match and will not allow tokenizers.

That's fine for now - no questions, but you said that previously

In FlatObject FieldMapper, we are intended to open parameters in the future,

So would these parameters include the possibility to specify normalizer? If yes - the reindexing will be required.

[mingshl]

@reta Yes, the first PR is using default parameters, and Lucene.KEYWORD_ANALYZER. We don't have any open parameters -- that's said normalizers can not be used in this first release. Similar to other parameters, such as docValue, ignoreAbove, nullValue. Changing any one of the parameters, yes, users will need to re-index, that's why it's a wider discussion on the open parameters, and we should better to add the open parameters all together in a next PR.

[reta]

@reta Yes, the first PR is using default parameters, and Lucene.KEYWORD_ANALYZER. We don't have any open parameters -- that's said normalizers can not be used in this first release. Similar to other parameters, such as docValue, ignoreAbove, nullValue. Changing any one of the parameters, yes, users will need to re-index, that's why it's a wider discussion on the open parameters, and we should better to add the open parameters all together in a next PR.

All set, I just wanted to highlight that

[mingshl]

@reta Yes, the first PR is using default parameters, and Lucene.KEYWORD_ANALYZER. We don't have any open parameters -- that's said normalizers can not be used in this first release. Similar to other parameters, such as docValue, ignoreAbove, nullValue. Changing any one of the parameters, yes, users will need to re-index, that's why it's a wider discussion on the open parameters, and we should better to add the open parameters all together in a next PR.

All set, I just wanted to highlight that

Thank you!!! I also added "normalizers" as one of the example of open parameters in the description in the PR.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      2 org.opensearch.cluster.service.MasterServiceTests.classMethod
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes
      1 org.opensearch.cluster.service.MasterServiceTests.testThrottlingForMultipleTaskTypes

• URL: https://build.ci.opensearch.org/job/gradle-check/13661/
• CommitID: de2f32b
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation

• URL: https://build.ci.opensearch.org/job/gradle-check/13664/
• CommitID: 543755f
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[reta]

LGTM @mingshl , I have nothing to add (taking into account the limitations and future work), thank you!

[dblock]

I merged it, great work @mingshl!

[dblock]

@mingshl What's next for this feature? Do you have a list of GH issues/improvements as next steps written up?

[mingshl]

@mingshl What's next for this feature? Do you have a list of GH issues/improvements as next steps written up?

Created new issues for future enhancement:
#7138
#7137
#7136

[macohen]

@mingshl awesome! Can you label those three issues "Search" and then close this if you think there's no more work do be done here please?