Skip to content

Commit

Permalink
Rename 'embedded JSON' to 'flattened object'.
Browse files Browse the repository at this point in the history
The code refers to 'flat object' in some places for brevity.
  • Loading branch information
jtibshirani committed Jun 7, 2019
1 parent 74ebc19 commit 064e26f
Show file tree
Hide file tree
Showing 31 changed files with 316 additions and 316 deletions.
5 changes: 2 additions & 3 deletions docs/reference/mapping/types.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ string:: <<text,`text`>> and <<keyword,`keyword`>>
<<array>>:: Array support does not require a dedicated `type`
<<object>>:: `object` for single JSON objects
<<nested>>:: `nested` for arrays of JSON objects
<<flattened>>:: Allows an entire JSON object to be indexed as a single field.

[float]
=== Geo datatypes
Expand All @@ -44,8 +45,6 @@ string:: <<text,`text`>> and <<keyword,`keyword`>>

<<alias>>:: Defines an alias to an existing field.

<<embedded-json>>:: Allows an entire JSON object to be indexed as a single field.

<<rank-feature>>:: Record numeric feature to boost hits at query time.

<<rank-features>>:: Record numeric features to boost hits at query time.
Expand Down Expand Up @@ -84,7 +83,7 @@ include::types/date.asciidoc[]

include::types/date_nanos.asciidoc[]

include::types/embedded-json.asciidoc[]
include::types/flattened.asciidoc[]

include::types/geo-point.asciidoc[]

Expand Down
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[[embedded-json]]
=== Embedded JSON datatype
[[flattened]]
=== Flattened datatype

By default, each subfield in an object is mapped and indexed separately. If
the names or types of the subfields are not known in advance, then they are
<<dynamic-mapping, mapped dynamically>>.

The `embedded_json` type provides an alternative approach, where the entire
object is mapped as a single field. Given an object, the `embedded_json`
The `flattened` type provides an alternative approach, where the entire
object is mapped as a single field. Given an object, the `flattened`
mapping will parse out its leaf values and index them into one field as
keywords. The object's contents can then be searched through simple queries
and aggregations.
Expand All @@ -16,17 +16,17 @@ number of unique keys. Only one field mapping is created for the whole JSON
object, which can help prevent a <<mapping-limit-settings, mappings explosion>>
from having too many distinct field mappings.

On the other hand, embedded JSON fields present a trade-off in terms of search
functionality. Only basic queries are allowed, with no support for numeric
range queries or highlighting. Further information on the limitations can be
found in the <<supported-operations, Supported operations>> section.
On the other hand, flattened object fields present a trade-off in terms of
search functionality. Only basic queries are allowed, with no support for
numeric range queries or highlighting. Further information on the limitations
can be found in the <<supported-operations, Supported operations>> section.

NOTE: The `embedded_json` mapping type should **not** be used for indexing all
JSON content, as it treats all values as keywords and does not provide full
NOTE: The `flattened` mapping type should **not** be used for indexing all
document content, as it treats all values as keywords and does not provide full
search functionality. The default approach, where each subfield has its own
entry in the mappings, works well in the majority of cases.

An embedded JSON field can be created as follows:
An flattened object field can be created as follows:
[source,js]
--------------------------------
PUT bug_reports
Expand All @@ -37,7 +37,7 @@ PUT bug_reports
"type": "text"
},
"labels": {
"type": "embedded_json"
"type": "flattened"
}
}
}
Expand All @@ -63,7 +63,7 @@ During indexing, tokens are created for each leaf value in the JSON object. The
values are indexed as string keywords, without analysis or special handling for
numbers or dates.

Querying the top-level `embedded_json` field searches all leaf values in the
Querying the top-level `flattened` field searches all leaf values in the
object:

[source,js]
Expand All @@ -77,7 +77,7 @@ POST bug_reports/_search
--------------------------------
// CONSOLE

To query on a specific key in the JSON object, object dot notation is used:
To query on a specific key in the flattened object, object dot notation is used:
[source,js]
--------------------------------
POST bug_reports/_search
Expand All @@ -92,11 +92,11 @@ POST bug_reports/_search
[[supported-operations]]
==== Supported operations

Because of the similarities in the way values are indexed, `embedded_json`
Because of the similarities in the way values are indexed, `flattened`
fields share much of the same mapping and search functionality as
<<keyword, `keyword`>> fields.

Currently, embedded JSON fields can be used with the following query types:
Currently, flattened object fields can be used with the following query types:

- `term`, `terms`, and `terms_set`
- `prefix`
Expand All @@ -108,19 +108,19 @@ Currently, embedded JSON fields can be used with the following query types:
When querying, it is not possible to refer to field keys using wildcards, as in
`{ "term": {"labels.time*": 1541457010}}`. Note that all queries, including
`range`, treat the values as string keywords. Highlighting is not supported on
`embedded_json` fields.
`flattened` fields.

It is possible to sort on an embedded JSON field, as well as perform simple
It is possible to sort on an flattened object field, as well as perform simple
keyword-style aggregations such as `terms`. As with queries, there is no
special support for numerics -- all values in the JSON object are treated as
keywords. When sorting, this implies that values are compared
lexicographically.

Embedded JSON fields currently cannot be stored. It is not possible to specify
the <<mapping-store, `store`>> parameter in the mapping.
Flattened object fields currently cannot be stored. It is not possible to
specify the <<mapping-store, `store`>> parameter in the mapping.

[[json-params]]
==== Parameters for JSON fields
[[flattened-params]]
==== Parameters for flattened object fields

The following mapping parameters are accepted:

Expand All @@ -133,9 +133,9 @@ The following mapping parameters are accepted:

`depth_limit`::

The maximum allowed depth of the JSON field, in terms of nested inner
objects. If a JSON field exceeds this limit, then an error will be
thrown. Defaults to `20`.
The maximum allowed depth of the flattened object field, in terms of nested
inner objects. If a flattened object field exceeds this limit, then an
error will be thrown. Defaults to `20`.

<<doc-values,`doc_values`>>::

Expand All @@ -153,8 +153,8 @@ The following mapping parameters are accepted:

Leaf values longer than this limit will not be indexed. By default, there
is no limit and all values will be indexed. Note that this limit applies
to the leaf values within the JSON field, and not the length of the entire
field.
to the leaf values within the flattened object field, and not the length of
the entire field.

<<mapping-index,`index`>>::

Expand All @@ -170,8 +170,8 @@ The following mapping parameters are accepted:
<<null-value,`null_value`>>::

A string value which is substituted for any explicit `null` values within
the JSON field. Defaults to `null`, which means null sfields are treated as
if it were missing.
the flattened object field. Defaults to `null`, which means null sields are
treated as if it were missing.

<<similarity,`similarity`>>::

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1330,55 +1330,55 @@ setup:
- match: {hits.total: 3}

---
"Test exists query on JSON field":
"Test exists query on flat object field":
- skip:
version: " - 7.99.99"
reason: "JSON fields are currently only implemented in 8.0."
reason: "Flat object fields are currently only implemented in 8.0."

- do:
indices.create:
index: json_test
index: flat_object_test
body:
mappings:
dynamic: false
properties:
json:
type: embedded_json
flat_object:
type: flattened
- do:
index:
index: json_test
index: flat_object_test
id: 1
body:
json:
flat_object:
key: some_value
refresh: true

- do:
search:
index: json_test
index: flat_object_test
body:
query:
exists:
field: json
field: flat_object

- match: { hits.total.value: 1 }

- do:
search:
index: json_test
index: flat_object_test
body:
query:
exists:
field: json.key
field: flat_object.key

- match: { hits.total.value: 1 }

- do:
search:
index: json_test
index: flat_object_test
body:
query:
exists:
field: json.nonexistent_key
field: flat_object.nonexistent_key

- match: { hits.total.value: 0 }
Original file line number Diff line number Diff line change
Expand Up @@ -63,10 +63,10 @@
- match: {hits.total: 0}

---
"search on JSON field":
"search on flat object field":
- skip:
version: " - 7.99.99"
reason: "JSON fields are currently only implemented in 8.0."
reason: "Flat object fields are currently only implemented in 8.0."

- do:
indices.create:
Expand All @@ -75,7 +75,7 @@
mappings:
properties:
headers:
type: embedded_json
type: flattened

- do:
index:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,24 +38,24 @@ class FieldTypeLookup implements Iterable<MappedFieldType> {
final CopyOnWriteHashMap<String, MappedFieldType> fullNameToFieldType;
private final CopyOnWriteHashMap<String, String> aliasToConcreteName;

private final CopyOnWriteHashMap<String, JsonFieldMapper> fullNameToJsonMapper;
private final int maxJsonFieldDepth;
private final CopyOnWriteHashMap<String, FlatObjectFieldMapper> flatObjectMappers;
private final int maxFlatObjectDepth;

FieldTypeLookup() {
fullNameToFieldType = new CopyOnWriteHashMap<>();
aliasToConcreteName = new CopyOnWriteHashMap<>();
fullNameToJsonMapper = new CopyOnWriteHashMap<>();
maxJsonFieldDepth = 0;
flatObjectMappers = new CopyOnWriteHashMap<>();
maxFlatObjectDepth = 0;
}

private FieldTypeLookup(CopyOnWriteHashMap<String, MappedFieldType> fullNameToFieldType,
CopyOnWriteHashMap<String, String> aliasToConcreteName,
CopyOnWriteHashMap<String, JsonFieldMapper> fullNameToJsonMapper,
int maxJsonFieldDepth) {
CopyOnWriteHashMap<String, FlatObjectFieldMapper> flatObjectMappers,
int maxFlatObjectDepth) {
this.fullNameToFieldType = fullNameToFieldType;
this.aliasToConcreteName = aliasToConcreteName;
this.fullNameToJsonMapper = fullNameToJsonMapper;
this.maxJsonFieldDepth = maxJsonFieldDepth;
this.flatObjectMappers = flatObjectMappers;
this.maxFlatObjectDepth = maxFlatObjectDepth;
}

/**
Expand All @@ -74,7 +74,7 @@ public FieldTypeLookup copyAndAddAll(String type,

CopyOnWriteHashMap<String, MappedFieldType> fullName = this.fullNameToFieldType;
CopyOnWriteHashMap<String, String> aliases = this.aliasToConcreteName;
CopyOnWriteHashMap<String, JsonFieldMapper> jsonMappers = this.fullNameToJsonMapper;
CopyOnWriteHashMap<String, FlatObjectFieldMapper> flatObjectMappers = this.flatObjectMappers;

for (FieldMapper fieldMapper : fieldMappers) {
String fieldName = fieldMapper.name();
Expand All @@ -85,8 +85,8 @@ public FieldTypeLookup copyAndAddAll(String type,
fullName = fullName.copyAndPut(fieldType.name(), fieldType);
}

if (fieldMapper instanceof JsonFieldMapper) {
jsonMappers = fullNameToJsonMapper.copyAndPut(fieldName, (JsonFieldMapper) fieldMapper);
if (fieldMapper instanceof FlatObjectFieldMapper) {
flatObjectMappers = this.flatObjectMappers.copyAndPut(fieldName, (FlatObjectFieldMapper) fieldMapper);
}
}

Expand All @@ -100,13 +100,13 @@ public FieldTypeLookup copyAndAddAll(String type,
}
}

int maxFieldDepth = getMaxJsonFieldDepth(aliases, jsonMappers);
int maxFlatObjectDepth = getMaxFlatObjectDepth(aliases, flatObjectMappers);

return new FieldTypeLookup(fullName, aliases, jsonMappers, maxFieldDepth);
return new FieldTypeLookup(fullName, aliases, flatObjectMappers, maxFlatObjectDepth);
}

private static int getMaxJsonFieldDepth(CopyOnWriteHashMap<String, String> aliases,
CopyOnWriteHashMap<String, JsonFieldMapper> jsonMappers) {
private static int getMaxFlatObjectDepth(CopyOnWriteHashMap<String, String> aliases,
CopyOnWriteHashMap<String, FlatObjectFieldMapper> jsonMappers) {
int maxFieldDepth = 0;
for (Map.Entry<String, String> entry : aliases.entrySet()) {
String aliasName = entry.getKey();
Expand Down Expand Up @@ -152,22 +152,22 @@ public MappedFieldType get(String field) {
return fieldType;
}

// If the mapping contains JSON fields, check if this could correspond
// to a keyed JSON field of the form 'path_to_json_field.path_to_key'.
return !fullNameToJsonMapper.isEmpty() ? getKeyedJsonField(field) : null;
// If the mapping contains flat object fields, check if this could correspond
// to a keyed field of the form 'path_to_flat_object.path_to_key'.
return !flatObjectMappers.isEmpty() ? getKeyedFlatObjectField(field) : null;
}

/**
* Check if the given field corresponds to a keyed JSON field of the form
* 'path_to_json_field.path_to_key'. If so, returns a field type that can
* be used to perform searches on this field.
* Check if the given field corresponds to a keyed flat object field of the
* form 'path_to_flat_object.path_to_key'. If so, returns a field type that
* can be used to perform searches on this field.
*/
private MappedFieldType getKeyedJsonField(String field) {
private MappedFieldType getKeyedFlatObjectField(String field) {
int dotIndex = -1;
int fieldDepth = 0;

while (true) {
if (++fieldDepth > maxJsonFieldDepth) {
if (++fieldDepth > maxFlatObjectDepth) {
return null;
}

Expand All @@ -178,7 +178,7 @@ private MappedFieldType getKeyedJsonField(String field) {

String parentField = field.substring(0, dotIndex);
String concreteField = aliasToConcreteName.getOrDefault(parentField, parentField);
JsonFieldMapper mapper = fullNameToJsonMapper.get(concreteField);
FlatObjectFieldMapper mapper = flatObjectMappers.get(concreteField);

if (mapper != null) {
String key = field.substring(dotIndex + 1);
Expand Down Expand Up @@ -209,18 +209,18 @@ public Collection<String> simpleMatchToFullName(String pattern) {
public Iterator<MappedFieldType> iterator() {
Iterator<MappedFieldType> concreteFieldTypes = fullNameToFieldType.values().iterator();

if (fullNameToJsonMapper.isEmpty()) {
if (flatObjectMappers.isEmpty()) {
return concreteFieldTypes;
} else {
Iterator<MappedFieldType> keyedJsonFieldTypes = fullNameToJsonMapper.values().stream()
Iterator<MappedFieldType> keyedFlatObjectTypes = flatObjectMappers.values().stream()
.<MappedFieldType>map(mapper -> mapper.keyedFieldType(""))
.iterator();
return Iterators.concat(concreteFieldTypes, keyedJsonFieldTypes);
return Iterators.concat(concreteFieldTypes, keyedFlatObjectTypes);
}
}

// Visible for testing.
int maxJsonFieldDepth() {
return maxJsonFieldDepth;
int maxFlatObjectDepth() {
return maxFlatObjectDepth;
}
}
Loading

0 comments on commit 064e26f

Please sign in to comment.