Add FlatObject FieldMapper (opensearch-project#6507)

* Add FlatObject FieldMapper Signed-off-by: Mingshi Liu <mingshl@amazon.com> * resolve import package for HttpHost Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Dynamic Create FlatObjectFieldType for dotpath field Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Rename flat-object to flat_object and fix CI tests Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Organized package Signed-off-by: Mingshi Liu <mingshl@amazon.com> * resolved compile error Signed-off-by: Mingshi Liu <mingshl@amazon.com> * organize package Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Add integration tests and remove benchmark Signed-off-by: Mingshi Liu <mingshl@amazon.com> * fix IT tests Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Skip IT tests before 2.7.0 Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Revert "Skip IT tests before 2.7.0" Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Add more IT tests for supported queries Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Removed license head and add tests for wildcard query Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Add tests for array, nested-arrary, number and float Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Upgrade FlatObjectFieldMapperTests to MapperTestCase - Upgrading `FlatObjectFieldMapperTests` from `MapperServiceTestCase` to `MapperTestCase`. The `MapperTestCase` explicitly forces us to: - Test parameter updates (empty now) - Explicitly specify if the field supports Meta and Boost (if not, relevant tests are automatically skipped) - Test also the substring fields - Add new test `testMapperServiceHasParser` to verify the new `flat_object` field mapper is present in mapper service registry - Remove duplicated test and assertions methods - Removed `testExistsQueryDocValuesDisabledWithNorms` as this test was not adding much. We shall reintroduce similar test later if we decide that we want to support ExistsQuery and what to do if DocValue are disabled and Norms enabled. Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> * Add exist query in FlatObjectFieldMapperTests and FlatObjectFieldDataTests Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Add IT tests for painless query in testDocValues Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Add filter script (Painless) test for flat_object While it is not possible to use flat_object field in scripting filter context to access doc values (like `doc[<flat_object>.<field_x>]`) it is possible to call `doc[<flat_object>].size()` to get number of fields inside the flat_object field. - Reorganize flat_object yaml tests into sections: - Mappings - Supported - Unsupported - Scripting (Painless) yamlRest tests need to go into lang-painless module Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> * Removed Normalizer Signed-off-by: Mingshi Liu <mingshl@amazon.com> * removed unused codes Signed-off-by: Mingshi Liu <mingshl@amazon.com> * Remove non-relevant Javadoc from DynamicKeyFieldMapper Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> * Improve flat_object scripting test Make it more obvious what the `doc[<flat_field>].size()` number represents. Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> * Add test for mapping parameters Mapping parameters are not allowed in the initial version. This commit adds a test to demonstrate that trying to specify index/search analyzer for the flat_object field will fail. Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> * remove IndexAnalyzer from Builder Signed-off-by: Mingshi Liu <mingshl@amazon.com> * remove IndexAnalyzer from Builder Signed-off-by: Mingshi Liu <mingshl@amazon.com> --------- Signed-off-by: Mingshi Liu <mingshl@amazon.com> Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io> Co-authored-by: Lukáš Vlček <lukas.vlcek@aiven.io>
austintlee · Apr 28, 2023 · 69db885 · 69db885
1 parent 40d2c0d
commit 69db885
Show file tree

Hide file tree

Showing 12 changed files with 2,099 additions and 10 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -118,6 +118,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 ### Changed
 - Require MediaType in Strings.toString API ([#6009](https://github.com/opensearch-project/OpenSearch/pull/6009))
 - [Refactor] XContent base classes from xcontent to core library ([#5902](https://github.com/opensearch-project/OpenSearch/pull/5902))
+- Added a new field type: flat_object ([#6507](https://github.com/opensearch-project/OpenSearch/pull/6507))
 
 ### Deprecated
 - Map, List, and Set in org.opensearch.common.collect ([#6609](https://github.com/opensearch-project/OpenSearch/pull/6609))

diff --git a/modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/30_search.yml b/modules/lang-painless/src/yamlRestTest/resources/rest-api-spec/test/painless/30_search.yml
@@ -482,3 +482,100 @@
                         }]
     - match: { error.root_cause.0.type: "illegal_argument_exception" }
     - match: { error.root_cause.0.reason: "script score function must not produce negative scores, but got: [-9.0]"}
+
+---
+
+"Flat-object fields from within the scripting":
+  - skip:
+      version: " - 2.99.99"
+      reason: "flat_object is introduced in 3.0.0 in main branch"
+
+  - do:
+      indices.create:
+        index: test
+        body:
+          mappings:
+            properties:
+              flat:
+                type : "flat_object"
+
+  # This document has 6 distinct parts in its flat_object field paths:
+  # - flat.field_1
+  # - flat.field_2
+  # - flat.field_3
+  # - flat.inner
+  # - flat.field_A
+  # - flat.field_B
+  - do:
+      index:
+        index: test
+        id:    1
+        body:  {
+          "flat": {
+            "field_1": "John Doe",
+            "field_2": 33,
+            "field_3": false,
+            "inner": {
+              "field_A": ["foo", "bar"],
+              "field_B": false
+            }
+          }
+        }
+
+  - do:
+      index:
+        index: test
+        id:    2
+        body:  {
+          "flat": {
+            "field_1": "Joe Public",
+            "field_2": 45
+          }
+        }
+
+  - do:
+      indices.refresh:
+        index: test
+
+  # It is possible to filter based on the number of distinct parts of flat_object field paths
+  - do:
+      search:
+        body: {
+          _source: true,
+          query: {
+            bool: {
+              filter: {
+                script: {
+                  script: {
+                    source: "doc['flat'].size() == 6",
+                    lang: "painless"
+                  }
+                }
+              }
+            }
+          }
+        }
+
+  - length:   { hits.hits: 1  }
+  - match: { hits.hits.0._source.flat.field_1: "John Doe" }
+
+  - do:
+      search:
+        body: {
+          _source: true,
+          query: {
+            bool: {
+              filter: {
+                script: {
+                  script: {
+                    source: "doc['flat'].size() < 6",
+                    lang: "painless"
+                  }
+                }
+              }
+            }
+          }
+        }
+
+  - length:   { hits.hits: 1  }
+  - match: { hits.hits.0._source.flat.field_1: "Joe Public" }