Skip to content

Commit

Permalink
Add setting to ignore dynamic fields when field limit is reached (ela…
Browse files Browse the repository at this point in the history
…stic#96235)

Adds a new `index.mapping.total_fields.ignore_dynamic_beyond_limit`
index setting.

When set to `true`, new fields are added to the mapping as long as the
field limit (`index.mapping.total_fields.limit`) is not exceeded. Fields
that would exceed the limit are not added to the mapping, similar to
`dynamic: false`.  Ignored fields are added to the `_ignored` metadata
field.

Relates to elastic#89911

To make this easier to review, this is split into the following PRs: -
[x] elastic#102915 - [x]
elastic#102936 - [x]
elastic#104769

Related but not a prerequisite: - [ ]
elastic#102885
  • Loading branch information
felixbarny authored Feb 2, 2024
1 parent 75e1181 commit f642b8a
Show file tree
Hide file tree
Showing 24 changed files with 795 additions and 128 deletions.
5 changes: 5 additions & 0 deletions docs/changelog/96235.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 96235
summary: Add `index.mapping.total_fields.ignore_dynamic_beyond_limit` setting to ignore dynamic fields when field limit is reached
area: Mapping
type: enhancement
issues: []
7 changes: 5 additions & 2 deletions docs/reference/mapping/fields/ignored-field.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@
The `_ignored` field indexes and stores the names of every field in a document
that has been ignored when the document was indexed. This can, for example,
be the case when the field was malformed and <<ignore-malformed,`ignore_malformed`>>
was turned on, or when a `keyword` fields value exceeds its optional
<<ignore-above,`ignore_above`>> setting.
was turned on, when a `keyword` field's value exceeds its optional
<<ignore-above,`ignore_above`>> setting, or when
<<mapping-settings-limit,`index.mapping.total_fields.limit`>> has been reached and
<<mapping-settings-limit,`index.mapping.total_fields.ignore_dynamic_beyond_limit`>>
is set to `true`.

This field is searchable with <<query-dsl-term-query,`term`>>,
<<query-dsl-terms-query,`terms`>> and <<query-dsl-exists-query,`exists`>>
Expand Down
11 changes: 10 additions & 1 deletion docs/reference/mapping/mapping-settings-limit.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,18 @@ limits the maximum number of clauses in a query.
+
[TIP]
====
If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type.
If your field mappings contain a large, arbitrary set of keys, consider using the <<flattened,flattened>> data type,
or setting the index setting `index.mapping.total_fields.ignore_dynamic_beyond_limit` to `true`.
====

`index.mapping.total_fields.ignore_dynamic_beyond_limit`::
This setting determines what happens when a dynamically mapped field would exceed the total fields limit.
When set to `false` (the default), the index request of the document that tries to add a dynamic field to the mapping will fail with the message `Limit of total fields [X] has been exceeded`.
When set to `true`, the index request will not fail.
Instead, fields that would exceed the limit are not added to the mapping, similar to <<dynamic, `dynamic: false`>>.
The fields that were not added to the mapping will be added to the <<mapping-ignored-field, `_ignored` field>>.
The default value is `false`.

`index.mapping.depth.limit`::
The maximum depth for a field, which is measured as the number of inner
objects. For instance, if all fields are defined at the root object level,
Expand Down
7 changes: 7 additions & 0 deletions docs/reference/mapping/params/dynamic.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -90,3 +90,10 @@ accepts the following parameters:
to the mapping, and new fields must be added explicitly.
`strict`:: If new fields are detected, an exception is thrown and the document
is rejected. New fields must be explicitly added to the mapping.

[[dynamic-field-limit]]
==== Behavior when reaching the field limit
Setting `dynamic` to either `true` or `runtime` will only add dynamic fields until <<mapping-settings-limit,`index.mapping.total_fields.limit`>> is reached.
By default, index requests for documents that would exceed the field limit will fail,
unless <<mapping-settings-limit,`index.mapping.total_fields.ignore_dynamic_beyond_limit`>> is set to `true`.
In that case, ignored fields are added to the <<mapping-ignored-field, `_ignored` metadata field>>.
Original file line number Diff line number Diff line change
Expand Up @@ -41,13 +41,16 @@ timing out in the browser's Developer Tools Network tab.
doesn't normally cause problems unless it's combined with overriding
<<mapping-settings-limit,`index.mapping.total_fields.limit`>>. The
default `1000` limit is considered generous, though overriding to `10000`
doesn't cause noticable impact depending on use case. However, to give
doesn't cause noticeable impact depending on use case. However, to give
a bad example, overriding to `100000` and this limit being hit
by mapping totals would usually have strong performance implications.

If your index mapped fields expect to contain a large, arbitrary set of
keys, you may instead consider:

* Setting <<mapping-settings-limit,`index.mapping.total_fields.ignore_dynamic_beyond_limit`>> to `true`.
Instead of rejecting documents that exceed the field limit, this will ignore dynamic fields once the limit is reached.

* Using the <<flattened,flattened>> data type. Please note,
however, that flattened objects is link:https://github.com/elastic/kibana/issues/25820[not fully supported in {kib}] yet. For example, this could apply to sub-mappings like { `host.name` ,
`host.os`, `host.version` }. Desired fields are still accessed by
Expand Down

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,8 @@ private static ClusterState applyRequest(
MapperService.mergeMappings(
mapperService.documentMapper(),
mapping,
request.autoUpdate() ? MergeReason.MAPPING_AUTO_UPDATE : MergeReason.MAPPING_UPDATE
request.autoUpdate() ? MergeReason.MAPPING_AUTO_UPDATE : MergeReason.MAPPING_UPDATE,
mapperService.getIndexSettings()
);
}
Metadata.Builder builder = Metadata.builder(metadata);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ public final class IndexScopedSettings extends AbstractScopedSettings {
Store.INDEX_STORE_STATS_REFRESH_INTERVAL_SETTING,
MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING,
MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING,
MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
MapperService.INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING,
MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING,
MapperService.INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING,
Expand Down
15 changes: 15 additions & 0 deletions server/src/main/java/org/elasticsearch/index/IndexSettings.java
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_DEPTH_LIMIT_SETTING;
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING;
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING;
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING;
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING;
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING;
import static org.elasticsearch.index.mapper.MapperService.INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING;
Expand Down Expand Up @@ -753,6 +754,7 @@ private void setRetentionLeaseMillis(final TimeValue retentionLease) {
private volatile long mappingNestedFieldsLimit;
private volatile long mappingNestedDocsLimit;
private volatile long mappingTotalFieldsLimit;
private volatile boolean ignoreDynamicFieldsBeyondLimit;
private volatile long mappingDepthLimit;
private volatile long mappingFieldNameLengthLimit;
private volatile long mappingDimensionFieldsLimit;
Expand Down Expand Up @@ -897,6 +899,7 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
mappingNestedFieldsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING);
mappingNestedDocsLimit = scopedSettings.get(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING);
mappingTotalFieldsLimit = scopedSettings.get(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING);
ignoreDynamicFieldsBeyondLimit = scopedSettings.get(INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING);
mappingDepthLimit = scopedSettings.get(INDEX_MAPPING_DEPTH_LIMIT_SETTING);
mappingFieldNameLengthLimit = scopedSettings.get(INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING);
mappingDimensionFieldsLimit = scopedSettings.get(INDEX_MAPPING_DIMENSION_FIELDS_LIMIT_SETTING);
Expand Down Expand Up @@ -976,6 +979,10 @@ public IndexSettings(final IndexMetadata indexMetadata, final Settings nodeSetti
scopedSettings.addSettingsUpdateConsumer(INDEX_SOFT_DELETES_RETENTION_LEASE_PERIOD_SETTING, this::setRetentionLeaseMillis);
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_FIELDS_LIMIT_SETTING, this::setMappingNestedFieldsLimit);
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_NESTED_DOCS_LIMIT_SETTING, this::setMappingNestedDocsLimit);
scopedSettings.addSettingsUpdateConsumer(
INDEX_MAPPING_IGNORE_DYNAMIC_BEYOND_LIMIT_SETTING,
this::setIgnoreDynamicFieldsBeyondLimit
);
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_TOTAL_FIELDS_LIMIT_SETTING, this::setMappingTotalFieldsLimit);
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_DEPTH_LIMIT_SETTING, this::setMappingDepthLimit);
scopedSettings.addSettingsUpdateConsumer(INDEX_MAPPING_FIELD_NAME_LENGTH_LIMIT_SETTING, this::setMappingFieldNameLengthLimit);
Expand Down Expand Up @@ -1519,6 +1526,14 @@ private void setMappingTotalFieldsLimit(long value) {
this.mappingTotalFieldsLimit = value;
}

private void setIgnoreDynamicFieldsBeyondLimit(boolean ignoreDynamicFieldsBeyondLimit) {
this.ignoreDynamicFieldsBeyondLimit = ignoreDynamicFieldsBeyondLimit;
}

public boolean isIgnoreDynamicFieldsBeyondLimit() {
return ignoreDynamicFieldsBeyondLimit;
}

public long getMappingDepthLimit() {
return mappingDepthLimit;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -629,11 +629,6 @@ public DeleteResult(Exception failure, long version, long term, long seqNo, bool
this.found = found;
}

public DeleteResult(Mapping requiredMappingUpdate, String id) {
super(Operation.TYPE.DELETE, requiredMappingUpdate, id);
this.found = false;
}

public boolean isFound() {
return found;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -514,7 +514,11 @@ private static void parseObjectDynamic(DocumentParserContext context, String cur

}
if (context.dynamic() != ObjectMapper.Dynamic.RUNTIME) {
context.addDynamicMapper(dynamicObjectMapper);
if (context.addDynamicMapper(dynamicObjectMapper) == false) {
failIfMatchesRoutingPath(context, currentFieldName);
context.parser().skipChildren();
return;
}
}
if (dynamicObjectMapper instanceof NestedObjectMapper && context.isWithinCopyTo()) {
throwOnCreateDynamicNestedViaCopyTo(dynamicObjectMapper, context);
Expand Down Expand Up @@ -556,7 +560,10 @@ private static void parseArrayDynamic(DocumentParserContext context, String curr
parseNonDynamicArray(context, currentFieldName, currentFieldName);
} else {
if (parsesArrayValue(objectMapperFromTemplate)) {
context.addDynamicMapper(objectMapperFromTemplate);
if (context.addDynamicMapper(objectMapperFromTemplate) == false) {
context.parser().skipChildren();
return;
}
context.path().add(currentFieldName);
parseObjectOrField(context, objectMapperFromTemplate);
context.path().remove();
Expand Down Expand Up @@ -674,7 +681,9 @@ private static void parseDynamicValue(final DocumentParserContext context, Strin
failIfMatchesRoutingPath(context, currentFieldName);
return;
}
context.dynamic().getDynamicFieldsBuilder().createDynamicFieldFromValue(context, currentFieldName);
if (context.dynamic().getDynamicFieldsBuilder().createDynamicFieldFromValue(context, currentFieldName) == false) {
failIfMatchesRoutingPath(context, currentFieldName);
}
}

private static void ensureNotStrict(DocumentParserContext context, String currentFieldName) {
Expand Down
Loading

0 comments on commit f642b8a

Please sign in to comment.