Remove types from index templates for x-pack features #38637
Labels
>non-issue
:Search Foundations/Mapping
Index mappings, including merging and defining field types
:Security/Security
Security issues without another label
Team:Search Foundations
Meta label for the Search Foundations team in Elasticsearch
v7.0.0-rc1
v7.2.0
v8.0.0-alpha1
Comments
jakelandis
added
:Search Foundations/Mapping
|
Pinging @elastic/es-search |
|
Pinging @elastic/es-security |
|
Pinging @elastic/es-core-features |
This was referenced Feb 8, 2019
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Feb 8, 2019
This commit changes the type from "doc" to "_doc" for the .logstash-management template. Since this is an internally managed template it does not always go through the REST layer for it's internal representation. The internal representation requires the default "_doc" type, which for external templates is added in the REST layer. Related elastic#38637
This was referenced Feb 23, 2019
|
Blocked by #39469 to allow clusters with mixed versions. |
bizybot
pushed a commit
to bizybot/elasticsearch
that referenced
this issue
Mar 5, 2019
As we are moving to single type indices, we need to address this change in security-related indexes. To address this, we are - updating index templates to use preferred type name `_doc` - updating the API calls to use preferred type name `_doc` Upgrade impact:- In case of an upgrade from 6.x, the security index has type `doc` and this will keep working as there is a single type and `_doc` works as an alias to an existing type. The change is handled in the `SecurityIndexManager` when we load mappings and settings from the template. Previously, we used to do a `PutIndexTemplateRequest` with the mapping source JSON with the type name. This has been modified to remove the type name from the source. So in the case of an upgrade, the `doc` type is updated whereas for fresh installs `_doc` is updated. This happens as backend handles `_doc` as an alias to the existing type name. An optional step is to `reindex` security index and update the type to `_doc`. Since we do not support the security audit log index, that template has been deleted. Relates: elastic#38637
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this issue
Mar 6, 2019
As we are moving to single type indices, we need to address this change in security-related indexes. To address this, we are - updating index templates to use preferred type name `_doc` - updating the API calls to use preferred type name `_doc` Upgrade impact:- In case of an upgrade from 6.x, the security index has type `doc` and this will keep working as there is a single type and `_doc` works as an alias to an existing type. The change is handled in the `SecurityIndexManager` when we load mappings and settings from the template. Previously, we used to do a `PutIndexTemplateRequest` with the mapping source JSON with the type name. This has been modified to remove the type name from the source. So in the case of an upgrade, the `doc` type is updated whereas for fresh installs `_doc` is updated. This happens as backend handles `_doc` as an alias to the existing type name. An optional step is to `reindex` security index and update the type to `_doc`. Since we do not support the security audit log index, that template has been deleted. Relates: elastic#38637
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this issue
Mar 6, 2019
As we are moving to single type indices, we need to address this change in security-related indexes. To address this, we are - updating index templates to use preferred type name `_doc` - updating the API calls to use preferred type name `_doc` Upgrade impact:- In case of an upgrade from 6.x, the security index has type `doc` and this will keep working as there is a single type and `_doc` works as an alias to an existing type. The change is handled in the `SecurityIndexManager` when we load mappings and settings from the template. Previously, we used to do a `PutIndexTemplateRequest` with the mapping source JSON with the type name. This has been modified to remove the type name from the source. So in the case of an upgrade, the `doc` type is updated whereas for fresh installs `_doc` is updated. This happens as backend handles `_doc` as an alias to the existing type name. An optional step is to `reindex` security index and update the type to `_doc`. Since we do not support the security audit log index, that template has been deleted. Relates: elastic#38637
bizybot
added a commit
to bizybot/elasticsearch
that referenced
this issue
Mar 6, 2019
As we are moving to single type indices, we need to address this change in security-related indexes. To address this, we are - updating index templates to use preferred type name `_doc` - updating the API calls to use preferred type name `_doc` Upgrade impact:- In case of an upgrade from 6.x, the security index has type `doc` and this will keep working as there is a single type and `_doc` works as an alias to an existing type. The change is handled in the `SecurityIndexManager` when we load mappings and settings from the template. Previously, we used to do a `PutIndexTemplateRequest` with the mapping source JSON with the type name. This has been modified to remove the type name from the source. So in the case of an upgrade, the `doc` type is updated whereas for fresh installs `_doc` is updated. This happens as backend handles `_doc` as an alias to the existing type name. An optional step is to `reindex` security index and update the type to `_doc`. Since we do not support the security audit log index, that template has been deleted. Relates: elastic#38637
bizybot
added a commit
that referenced
this issue
Mar 6, 2019
As we are moving to single type indices, we need to address this change in security-related indexes. To address this, we are - updating index templates to use preferred type name `_doc` - updating the API calls to use preferred type name `_doc` Upgrade impact:- In case of an upgrade from 6.x, the security index has type `doc` and this will keep working as there is a single type and `_doc` works as an alias to an existing type. The change is handled in the `SecurityIndexManager` when we load mappings and settings from the template. Previously, we used to do a `PutIndexTemplateRequest` with the mapping source JSON with the type name. This has been modified to remove the type name from the source. So in the case of an upgrade, the `doc` type is updated whereas for fresh installs `_doc` is updated. This happens as backend handles `_doc` as an alias to the existing type name. An optional step is to `reindex` security index and update the type to `_doc`. Since we do not support the security audit log index, that template has been deleted. Relates: #38637
bizybot
added a commit
that referenced
this issue
Mar 6, 2019
As we are moving to single type indices, we need to address this change in security-related indexes. To address this, we are - updating index templates to use preferred type name `_doc` - updating the API calls to use preferred type name `_doc` Upgrade impact:- In case of an upgrade from 6.x, the security index has type `doc` and this will keep working as there is a single type and `_doc` works as an alias to an existing type. The change is handled in the `SecurityIndexManager` when we load mappings and settings from the template. Previously, we used to do a `PutIndexTemplateRequest` with the mapping source JSON with the type name. This has been modified to remove the type name from the source. So in the case of an upgrade, the `doc` type is updated whereas for fresh installs `_doc` is updated. This happens as backend handles `_doc` as an alias to the existing type name. An optional step is to `reindex` security index and update the type to `_doc`. Since we do not support the security audit log index, that template has been deleted. Relates: #38637
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Mar 8, 2019
This commit changes the type from "doc" to "_doc" for the .logstash-management template. Since this is an internally managed template it does not always go through the REST layer for it's internal representation. The internal representation requires the default "_doc" type, which for external templates is added in the REST layer. Related elastic#38637
jakelandis
added a commit
that referenced
this issue
Mar 8, 2019
…9819) This commit changes the type from "doc" to "_doc" for the .logstash-management template. Since this is an internally managed template it does not always go through the REST layer for it's internal representation. The internal representation requires the default "_doc" type, which for external templates is added in the REST layer. Related #38637
jakelandis
added a commit
that referenced
this issue
Mar 8, 2019
…9820) This commit changes the type from "doc" to "_doc" for the .logstash-management template. Since this is an internally managed template it does not always go through the REST layer for it's internal representation. The internal representation requires the default "_doc" type, which for external templates is added in the REST layer. Related #38637
jakelandis
added a commit
that referenced
this issue
Mar 8, 2019
This commit removes the "doc" type from watcher internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This impacts the .watches, .triggered-watches, and .watch-history indexes. External consumers do not need any changes since all external calls go through the _watcher API, and should not interact with the the .index directly. Relates #38637
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Mar 8, 2019
This commit removes the "doc" type from watcher internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This impacts the .watches, .triggered-watches, and .watch-history indexes. External consumers do not need any changes since all external calls go through the _watcher API, and should not interact with the the .index directly. Relates elastic#38637
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Mar 8, 2019
This commit removes the "doc" type from watcher internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This impacts the .watches, .triggered-watches, and .watch-history indexes. External consumers do not need any changes since all external calls go through the _watcher API, and should not interact with the the .index directly. Relates elastic#38637
This was referenced Mar 8, 2019
jakelandis
added a commit
that referenced
this issue
Mar 8, 2019
This commit removes the "doc" type from watcher internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This impacts the .watches, .triggered-watches, and .watch-history indexes. External consumers do not need any changes since all external calls go through the _watcher API, and should not interact with the the .index directly. Relates #38637
jakelandis
added a commit
that referenced
this issue
Mar 8, 2019
This commit removes the "doc" type from watcher internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This impacts the .watches, .triggered-watches, and .watch-history indexes. External consumers do not need any changes since all external calls go through the _watcher API, and should not interact with the the .index directly. Relates #38637
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Mar 10, 2019
The template still carries the "_doc" type since that is needed for the internal representation. This change impacts the following templates: monitoring-alerts.json monitoring-beats.json monitoring-es.json monitoring-kibana.json monitoring-logstash.json As part of the required changes, the system_api_version has been bumped from "6" to "7" and support for version "2" has been dropped. A new empty pipeline is now introduced for the version "7", and the formerly empty "6" pipeline will now remove the type and re-direct the request to the "7" index. Additionally, to due to a difference in the internal representation (which requires the inclusion of "_doc" type) and external representation (which requires the exclusion of any type) a helper method is introduced to help convert internal to external representation, and used by the monitoring HTTP template exporter. Relates elastic#38637
jakelandis
added a commit
that referenced
this issue
Mar 11, 2019
…9888) This commit removes the "doc" type from monitoring internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This change impacts the following templates: monitoring-alerts.json monitoring-beats.json monitoring-es.json monitoring-kibana.json monitoring-logstash.json As part of the required changes, the system_api_version has been bumped from "6" to "7" and support for version "2" has been dropped. A new empty pipeline is now introduced for the version "7", and the formerly empty "6" pipeline will now remove the type and re-direct the request to the "7" index. Additionally, to due to a difference in the internal representation (which requires the inclusion of "_doc" type) and external representation (which requires the exclusion of any type) a helper method is introduced to help convert internal to external representation, and used by the monitoring HTTP template exporter. Relates #38637
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Mar 11, 2019
…astic#39888) This commit removes the "doc" type from monitoring internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This change impacts the following templates: monitoring-alerts.json monitoring-beats.json monitoring-es.json monitoring-kibana.json monitoring-logstash.json As part of the required changes, the system_api_version has been bumped from "6" to "7" and support for version "2" has been dropped. A new empty pipeline is now introduced for the version "7", and the formerly empty "6" pipeline will now remove the type and re-direct the request to the "7" index. Additionally, to due to a difference in the internal representation (which requires the inclusion of "_doc" type) and external representation (which requires the exclusion of any type) a helper method is introduced to help convert internal to external representation, and used by the monitoring HTTP template exporter. Relates elastic#38637
jakelandis
added a commit
to jakelandis/elasticsearch
that referenced
this issue
Mar 11, 2019
…astic#39888) This commit removes the "doc" type from monitoring internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This change impacts the following templates: monitoring-alerts.json monitoring-beats.json monitoring-es.json monitoring-kibana.json monitoring-logstash.json As part of the required changes, the system_api_version has been bumped from "6" to "7" and support for version "2" has been dropped. A new empty pipeline is now introduced for the version "7", and the formerly empty "6" pipeline will now remove the type and re-direct the request to the "7" index. Additionally, to due to a difference in the internal representation (which requires the inclusion of "_doc" type) and external representation (which requires the exclusion of any type) a helper method is introduced to help convert internal to external representation, and used by the monitoring HTTP template exporter. Relates elastic#38637
This was referenced Mar 11, 2019
jakelandis
added a commit
that referenced
this issue
Mar 11, 2019
…9888) (#39927) This commit removes the "doc" type from monitoring internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This change impacts the following templates: monitoring-alerts.json monitoring-beats.json monitoring-es.json monitoring-kibana.json monitoring-logstash.json As part of the required changes, the system_api_version has been bumped from "6" to "7" and support for version "2" has been dropped. A new empty pipeline is now introduced for the version "7", and the formerly empty "6" pipeline will now remove the type and re-direct the request to the "7" index. Additionally, to due to a difference in the internal representation (which requires the inclusion of "_doc" type) and external representation (which requires the exclusion of any type) a helper method is introduced to help convert internal to external representation, and used by the monitoring HTTP template exporter. Relates #38637
|
All the required changes have been made and back ported to the 7.0 branch. |
jakelandis
added a commit
that referenced
this issue
Mar 11, 2019
…9888) (#39926) This commit removes the "doc" type from monitoring internal indexes. The template still carries the "_doc" type since that is needed for the internal representation. This change impacts the following templates: monitoring-alerts.json monitoring-beats.json monitoring-es.json monitoring-kibana.json monitoring-logstash.json As part of the required changes, the system_api_version has been bumped from "6" to "7" and support for version "2" has been dropped. A new empty pipeline is now introduced for the version "7", and the formerly empty "6" pipeline will now remove the type and re-direct the request to the "7" index. Additionally, to due to a difference in the internal representation (which requires the inclusion of "_doc" type) and external representation (which requires the exclusion of any type) a helper method is introduced to help convert internal to external representation, and used by the monitoring HTTP template exporter. Relates #38637
javanna
added
the
Team:Search Foundations
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The following x-pack features use templates with explicit mappings in the templates:
[types removal] Logstash central management typeless API kibana#30546) (Central management typeless API logstash#10421) (Update logstash-management.json to use typeless template #38653)Remove types from internal monitoring templates and bump to api 7 #39888) (blocked byIngest ingest then create index #39607,update to send api version 6 logstash#10518,[Monitoring] Adapt monitoring to read from .monitoring-*-7-* indices kibana#32514)Types removal security index template #39705)Remove the index type from internal watcher indexes #39761)We can not simply remove the type from mapping since if any of the consumers use the (deprecated) typed API's they will start to fail. We need to update the consumers to the typeless API's before we can remove the types from the templates. Failure to update the consumers to the typeless API's will result in deprecation warnings that consumers have no means to resolve.
The types in the templates are relatively harmless since they still work with the type present. These internally managed templates differ from user defined/external defined templates since those with mappings with types will be rejected. This is due to an implementation detail, and ideally the internally managed templates follow the same rules as user defined/external templates.
Each of these areas requires some domain knowledge of which parts of the Elastic stack is consuming/generating the documents for these templates.
For example:
Logstash Central Configuration - Kibana writes the documents and Logstash reads the documents so both of these need to be updated to the typeless API's before we can remove the types.
The text was updated successfully, but these errors were encountered: