Skip to content

Commit

Permalink
Add FlatObject FieldMapper (opensearch-project#6507)
Browse files Browse the repository at this point in the history
* Add FlatObject FieldMapper

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* resolve import package for HttpHost

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Dynamic Create FlatObjectFieldType for dotpath field

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Rename flat-object to flat_object and fix CI tests

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Organized package

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* resolved compile error

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* organize package

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Add integration tests and remove benchmark

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* fix IT tests

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Skip IT tests before 2.7.0

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Revert "Skip IT tests before 2.7.0"

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Add more IT tests for supported queries

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Removed license head and add tests for wildcard query

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Add tests for array, nested-arrary, number and float

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Upgrade FlatObjectFieldMapperTests to  MapperTestCase

- Upgrading `FlatObjectFieldMapperTests` from `MapperServiceTestCase` to `MapperTestCase`. The `MapperTestCase` explicitly forces us to:
	- Test parameter updates (empty now)
	- Explicitly specify if the field supports Meta and Boost (if not, relevant tests are automatically skipped)
- Test also the substring fields
- Add new test `testMapperServiceHasParser` to verify the new `flat_object` field mapper is present in mapper service registry
- Remove duplicated test and assertions methods
- Removed `testExistsQueryDocValuesDisabledWithNorms` as this test was not adding much. We shall reintroduce similar test later if we decide that we want to support ExistsQuery and what to do if DocValue are disabled and Norms enabled.

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>

* Add exist query in FlatObjectFieldMapperTests and FlatObjectFieldDataTests

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Add IT tests for painless query in testDocValues

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Add filter script (Painless) test for flat_object

While it is not possible to use flat_object field in scripting filter context to access doc values (like `doc[<flat_object>.<field_x>]`) it is possible to call `doc[<flat_object>].size()` to get number of fields inside the flat_object field.

- Reorganize flat_object yaml tests into sections:
  - Mappings
  - Supported
  - Unsupported
- Scripting (Painless) yamlRest tests need to go into lang-painless module

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>

* Removed Normalizer

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* removed unused codes

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* Remove non-relevant Javadoc from DynamicKeyFieldMapper

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>

* Improve flat_object scripting test

Make it more obvious what the `doc[<flat_field>].size()` number represents.

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>

* Add test for mapping parameters

Mapping parameters are not allowed in the initial version. This commit adds a test to demonstrate that trying to specify index/search analyzer for the flat_object field will fail.

Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>

* remove IndexAnalyzer from Builder

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

* remove IndexAnalyzer from Builder

Signed-off-by: Mingshi Liu <mingshl@amazon.com>

---------

Signed-off-by: Mingshi Liu <mingshl@amazon.com>
Signed-off-by: Lukáš Vlček <lukas.vlcek@aiven.io>
Co-authored-by: Lukáš Vlček <lukas.vlcek@aiven.io>
  • Loading branch information
2 people authored and austintlee committed Apr 28, 2023
1 parent 40d2c0d commit 69db885
Show file tree
Hide file tree
Showing 12 changed files with 2,099 additions and 10 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
### Changed
- Require MediaType in Strings.toString API ([#6009](https://github.com/opensearch-project/OpenSearch/pull/6009))
- [Refactor] XContent base classes from xcontent to core library ([#5902](https://github.com/opensearch-project/OpenSearch/pull/5902))
- Added a new field type: flat_object ([#6507](https://github.com/opensearch-project/OpenSearch/pull/6507))

### Deprecated
- Map, List, and Set in org.opensearch.common.collect ([#6609](https://github.com/opensearch-project/OpenSearch/pull/6609))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -482,3 +482,100 @@
}]
- match: { error.root_cause.0.type: "illegal_argument_exception" }
- match: { error.root_cause.0.reason: "script score function must not produce negative scores, but got: [-9.0]"}

---

"Flat-object fields from within the scripting":
- skip:
version: " - 2.99.99"
reason: "flat_object is introduced in 3.0.0 in main branch"

- do:
indices.create:
index: test
body:
mappings:
properties:
flat:
type : "flat_object"

# This document has 6 distinct parts in its flat_object field paths:
# - flat.field_1
# - flat.field_2
# - flat.field_3
# - flat.inner
# - flat.field_A
# - flat.field_B
- do:
index:
index: test
id: 1
body: {
"flat": {
"field_1": "John Doe",
"field_2": 33,
"field_3": false,
"inner": {
"field_A": ["foo", "bar"],
"field_B": false
}
}
}

- do:
index:
index: test
id: 2
body: {
"flat": {
"field_1": "Joe Public",
"field_2": 45
}
}

- do:
indices.refresh:
index: test

# It is possible to filter based on the number of distinct parts of flat_object field paths
- do:
search:
body: {
_source: true,
query: {
bool: {
filter: {
script: {
script: {
source: "doc['flat'].size() == 6",
lang: "painless"
}
}
}
}
}
}

- length: { hits.hits: 1 }
- match: { hits.hits.0._source.flat.field_1: "John Doe" }

- do:
search:
body: {
_source: true,
query: {
bool: {
filter: {
script: {
script: {
source: "doc['flat'].size() < 6",
lang: "painless"
}
}
}
}
}
}

- length: { hits.hits: 1 }
- match: { hits.hits.0._source.flat.field_1: "Joe Public" }
Loading

0 comments on commit 69db885

Please sign in to comment.