From a2b3278491bbf672e45b5f32cd62b18eb6b23250 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Wed, 1 Dec 2021 10:19:04 -0500 Subject: [PATCH 1/3] [DOCS] Update snapshot defaults for system indices --- .../migrate_8_0/rest-api-changes.asciidoc | 35 ++++++++++++++++ .../apis/create-snapshot-api.asciidoc | 7 +++- .../apis/restore-snapshot-api.asciidoc | 14 +++---- .../reference/snapshot-restore/index.asciidoc | 7 +++- .../restore-snapshot.asciidoc | 42 ++----------------- .../snapshot-restore/take-snapshot.asciidoc | 11 ++--- 6 files changed, 60 insertions(+), 56 deletions(-) diff --git a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc index bfda113d87fd..e35be9d05709 100644 --- a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc +++ b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc @@ -800,6 +800,41 @@ Assign users with the `kibana_user` role to the `kibana_admin` role. Discontinue use of the `kibana_user` role. ==== +[[snapshot-resolve-system-indices]] +.For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to system indices or system data streams. +[%collapsible] +==== +*Details* + +For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to +system indices or system data streams. +{ref}/snapshot-restore.html#feature-state[Feature states] are now the only way +to back up and restore system indices or system data streams from a snapshot. + +You can no longer use the `indices` parameter of the +{ref}/slm-api-put-policy.html[create {slm-init} policy API] or the +{ref}/create-snapshot-api.html[create snapshot API] to include a system index in +a snapshot. To back up a system index, use the `include_global_state` and +`feature_states` parameters to include the corresponding feature state instead. +By default, the `include_global_state` and `feature_states` parameters include +all system indices. + +Similarly, you can no longer use the {ref}/restore-snapshot-api.html[restore snapshot +API]'s `indices` parameter to restore a system index from a snapshot. To restore +a system index, use the `include_global_state` and `feature_states` parameters +to restore the corresponding feature state instead. By default, the +`include_global_state` and `feature_states` parameters don't restore any system +indices. + +*Impact* + +If you previously used the `indices` parameter to back up or restore system +indices, update your {slm-init} policies and application to use the +`include_global_state` and `feature_states` parameters instead. + +Snapshot API requests that explicitly specify a system index in the `indices` +parameter will return an error. If the `indices` value includes a wildcard (`*`) +pattern, the pattern will no longer match system indices. +==== + .Snapshots compress metadata files by default. [%collapsible] ==== diff --git a/docs/reference/snapshot-restore/apis/create-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/create-snapshot-api.asciidoc index 331231c0754a..200a5e9b520c 100644 --- a/docs/reference/snapshot-restore/apis/create-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/create-snapshot-api.asciidoc @@ -114,9 +114,12 @@ include::restore-snapshot-api.asciidoc[tag=cluster-state-contents] (Optional, string or array of strings) Comma-separated list of data streams and indices to include in the snapshot. Supports <>. Defaults to an empty array -(`[]`), which includes all data streams and indices, including system indices. +(`[]`), which includes all regular data streams and regular indices. To exclude +all data streams and indices, use `-*` or `none`. + -To exclude all data streams and indices, use `-*` or `none`. +You can't use this parameter to include or exclude <> from a snapshot. Use +<<{page-id}-feature-states,`feature_states`>> instead. [id="{page-id}-feature-states"] `feature_states`:: diff --git a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc index 23d7168a5e38..2f41a0819be9 100644 --- a/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc +++ b/docs/reference/snapshot-restore/apis/restore-snapshot-api.asciidoc @@ -53,14 +53,8 @@ POST /index_4/_close [source,console] ---- POST /_snapshot/my_repository/my_snapshot/_restore -{ - "indices": "*,-.*", - "feature_states": [ "geoip" ] -} ---- // TEST[s/_restore/_restore?wait_for_completion=true/] -// TEST[s/",/"/] -// TEST[s/"feature_states": \[ "geoip" \]//] [[restore-snapshot-api-request]] ==== {api-request-title} @@ -198,7 +192,13 @@ For data streams, this option only applies to restored backing indices. New backing indices are configured using the data stream's matching index template. `indices`:: -(Optional, string or array of strings) Comma-separated list of indices and data streams to restore. Supports <>. Defaults to all indices and data streams in the snapshot, including system indices. +(Optional, string or array of strings) Comma-separated list of indices and data +streams to restore. Supports <>. Defaults +to all regular indices and regular data streams in the snapshot. ++ +You can't use this parameter to restore <>. Use +<> instead. [[restore-snapshot-api-partial]] `partial`:: diff --git a/docs/reference/snapshot-restore/index.asciidoc b/docs/reference/snapshot-restore/index.asciidoc index 66e93cb21390..839ed32e7305 100644 --- a/docs/reference/snapshot-restore/index.asciidoc +++ b/docs/reference/snapshot-restore/index.asciidoc @@ -31,8 +31,8 @@ recover or transfer its data. [[snapshot-contents]] == Snapshot contents -By default, a snapshot of a cluster contains the cluster state, all data -streams, and all indices, including system indices. The cluster state includes: +By default, a snapshot of a cluster contains the cluster state, all regular data +streams, and all regular indices. The cluster state includes: include::apis/restore-snapshot-api.asciidoc[tag=cluster-state-contents] @@ -61,6 +61,9 @@ used by the feature. For example, a feature state may include a regular index that contains the feature's execution history. Storing this history in a regular index lets you more easily search it. +In {es} 8.0 and later versions, feature states are the only way to back up and +restore system indices and system data streams. + [discrete] [[how-snapshots-work]] == How snapshots work diff --git a/docs/reference/snapshot-restore/restore-snapshot.asciidoc b/docs/reference/snapshot-restore/restore-snapshot.asciidoc index 0a12f5966617..b80e0b69964c 100644 --- a/docs/reference/snapshot-restore/restore-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/restore-snapshot.asciidoc @@ -82,10 +82,10 @@ GET _snapshot/my_repository/*?verbose=false You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the <>. -By default, a restore request attempts to restore all indices and data streams -in a snapshot, including <>. In most cases, you only need to restore a specific index or data -stream from a snapshot. However, you can't restore an existing open index. +By default, a restore request attempts to restore all regular indices and +regular data streams in a snapshot. In most cases, you only need to restore a +specific index or data stream from a snapshot. However, you can't restore an +existing open index. If you're restoring data to a pre-existing cluster, use one of the following methods to avoid conflicts with existing indices and data streams: @@ -93,10 +93,6 @@ following methods to avoid conflicts with existing indices and data streams: * <> * <> -Some {es} features automatically create system indices on cluster startup. To -avoid conflicts with these system indices when restoring data to a new cluster, -see <>. - [discrete] [[delete-restore]] ==== Delete and restore @@ -210,37 +206,12 @@ POST _reindex ---- // TEST[continued] -[discrete] -[[restore-exclude-system-indices]] -==== Exclude system indices - -Some {es} features, such as the <>, -automatically create system indices at startup. To avoid naming conflicts with -these indices, use the `-.*` wildcard pattern to exclude system indices and -other dot (`.`) indices from your restore request. - -For example, the following request uses the `*,-.*` wildcard pattern to restore -all indices and data streams except dot indices. - -[source,console] ----- -POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore -{ - "indices": "*,-.*" -} ----- -// TEST[setup:setup-snapshots] -// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/] -// TEST[s/_restore/_restore?wait_for_completion=true/] - [discrete] [[restore-feature-state]] === Restore a feature state You can restore a <> to recover system indices, system data streams, and other configuration data for a feature from a snapshot. -Restoring a feature state is the preferred way to restore system indices and -system data streams. If you restore a snapshot's cluster state, the operation restores all feature states in the snapshot by default. Similarly, if you don't restore a snapshot's @@ -271,21 +242,16 @@ Console before restoring the `security` feature state. If you run {es} on your own hardware, <> to ensure you'll still be able to access your cluster. -To avoid accidentally restoring system indices, system data streams, and other -dot indices, append the `-.*` wildcard pattern to the `indices` value. - [source,console] ---- POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore { - "indices": "*,-.*", "feature_states": [ "geoip" ] } ---- // TEST[setup:setup-snapshots] // TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/] // TEST[s/_restore/_restore?wait_for_completion=true/] -// TEST[s/",/"/] // TEST[s/"feature_states": \[ "geoip" \]//] [discrete] diff --git a/docs/reference/snapshot-restore/take-snapshot.asciidoc b/docs/reference/snapshot-restore/take-snapshot.asciidoc index a4abef920d01..077763c29f24 100644 --- a/docs/reference/snapshot-restore/take-snapshot.asciidoc +++ b/docs/reference/snapshot-restore/take-snapshot.asciidoc @@ -207,9 +207,7 @@ PUT _slm/policy/nightly-snapshots naming conflicts, the policy also appends a UUID to each snapshot name. <3> <> used to store the policy's snapshots. -<4> Data streams and indices to include in the policy's snapshots. This - configuration includes all data streams and indices, including system - indices. +<4> Data streams and indices to include in the policy's snapshots. <5> If `true`, the policy's snapshots include the cluster state. This also includes all feature states by default. To only include specific feature states, see <>. @@ -498,7 +496,7 @@ PUT _slm/policy/nightly-cluster-state-snapshots <2> Excludes regular data streams and indices. If you take dedicated snapshots of the cluster state, you'll need to exclude the -cluster state and system indices from your other snapshots. For example: +cluster state from your other snapshots. For example: [source,console] ---- @@ -509,7 +507,7 @@ PUT _slm/policy/nightly-snapshots "repository": "my_repository", "config": { "include_global_state": false, <1> - "indices": "*,-.*" <2> + "indices": "*" <2> }, "retention": { "expire_after": "30d", @@ -521,8 +519,7 @@ PUT _slm/policy/nightly-snapshots <1> Excludes the cluster state. This also excludes all feature states by default. -<2> Includes all data streams and indices except system indices and other - indices that begin with a dot (`.`). +<2> Includes all regular data streams and indices. [discrete] [[create-snapshots-different-time-intervals]] From 40a0de7ec814753e931e69944f2f948e35c0a094 Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Fri, 3 Dec 2021 10:48:33 -0500 Subject: [PATCH 2/3] reword --- docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc index e35be9d05709..7b0bf6b64ab9 100644 --- a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc +++ b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc @@ -810,7 +810,7 @@ system indices or system data streams. {ref}/snapshot-restore.html#feature-state[Feature states] are now the only way to back up and restore system indices or system data streams from a snapshot. -You can no longer use the `indices` parameter of the +You can no longer use the `indices` parameter for the {ref}/slm-api-put-policy.html[create {slm-init} policy API] or the {ref}/create-snapshot-api.html[create snapshot API] to include a system index in a snapshot. To back up a system index, use the `include_global_state` and From eb630c0308eacc5cf55de559d9b37c9f2ed31dfd Mon Sep 17 00:00:00 2001 From: James Rodewig <40268737+jrodewig@users.noreply.github.com> Date: Fri, 3 Dec 2021 15:19:41 -0500 Subject: [PATCH 3/3] Clarify SLM policy behavior for #81235 --- .../migration/migrate_8_0/rest-api-changes.asciidoc | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc index 7b0bf6b64ab9..45a28ac29700 100644 --- a/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc +++ b/docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc @@ -830,9 +830,12 @@ If you previously used the `indices` parameter to back up or restore system indices, update your {slm-init} policies and application to use the `include_global_state` and `feature_states` parameters instead. -Snapshot API requests that explicitly specify a system index in the `indices` -parameter will return an error. If the `indices` value includes a wildcard (`*`) -pattern, the pattern will no longer match system indices. +An {slm-init} policy that explicitly specifies a system index in the `indices` +parameter will fail to create snapshots. Similarly, a create snapshot API or +restore snapshot API request that explicitly specifies a system index in the +`indices` parameter will fail and return an error. If the `indices` value +includes a wildcard (`*`) pattern, the pattern will no longer match system +indices. ==== .Snapshots compress metadata files by default.