Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Update snapshot defaults for system indices #81226

Merged
merged 3 commits into from
Dec 8, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions docs/reference/migration/migrate_8_0/rest-api-changes.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -800,6 +800,44 @@ Assign users with the `kibana_user` role to the `kibana_admin` role.
Discontinue use of the `kibana_user` role.
====

[[snapshot-resolve-system-indices]]
.For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to system indices or system data streams.
[%collapsible]
====
*Details* +
For snapshot and {slm-init} APIs, the `indices` parameter no longer resolves to
system indices or system data streams.
{ref}/snapshot-restore.html#feature-state[Feature states] are now the only way
to back up and restore system indices or system data streams from a snapshot.

You can no longer use the `indices` parameter for the
{ref}/slm-api-put-policy.html[create {slm-init} policy API] or the
{ref}/create-snapshot-api.html[create snapshot API] to include a system index in
a snapshot. To back up a system index, use the `include_global_state` and
`feature_states` parameters to include the corresponding feature state instead.
By default, the `include_global_state` and `feature_states` parameters include
all system indices.

Similarly, you can no longer use the {ref}/restore-snapshot-api.html[restore snapshot
API]'s `indices` parameter to restore a system index from a snapshot. To restore
a system index, use the `include_global_state` and `feature_states` parameters
to restore the corresponding feature state instead. By default, the
`include_global_state` and `feature_states` parameters don't restore any system
indices.

*Impact* +
If you previously used the `indices` parameter to back up or restore system
indices, update your {slm-init} policies and application to use the
`include_global_state` and `feature_states` parameters instead.

An {slm-init} policy that explicitly specifies a system index in the `indices`
parameter will fail to create snapshots. Similarly, a create snapshot API or
restore snapshot API request that explicitly specifies a system index in the
`indices` parameter will fail and return an error. If the `indices` value
includes a wildcard (`*`) pattern, the pattern will no longer match system
indices.
====

.Snapshots compress metadata files by default.
[%collapsible]
====
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -114,9 +114,12 @@ include::restore-snapshot-api.asciidoc[tag=cluster-state-contents]
(Optional, string or array of strings)
Comma-separated list of data streams and indices to include in the snapshot.
Supports <<api-multi-index,multi-index syntax>>. Defaults to an empty array
(`[]`), which includes all data streams and indices, including system indices.
(`[]`), which includes all regular data streams and regular indices. To exclude
all data streams and indices, use `-*` or `none`.
+
To exclude all data streams and indices, use `-*` or `none`.
You can't use this parameter to include or exclude <<system-indices,system
indices or system data streams>> from a snapshot. Use
<<{page-id}-feature-states,`feature_states`>> instead.

[id="{page-id}-feature-states"]
`feature_states`::
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,14 +53,8 @@ POST /index_4/_close
[source,console]
----
POST /_snapshot/my_repository/my_snapshot/_restore
{
"indices": "*,-.*",
"feature_states": [ "geoip" ]
}
----
// TEST[s/_restore/_restore?wait_for_completion=true/]
// TEST[s/",/"/]
// TEST[s/"feature_states": \[ "geoip" \]//]

[[restore-snapshot-api-request]]
==== {api-request-title}
Expand Down Expand Up @@ -198,7 +192,13 @@ For data streams, this option only applies to restored backing indices. New
backing indices are configured using the data stream's matching index template.

`indices`::
(Optional, string or array of strings) Comma-separated list of indices and data streams to restore. Supports <<api-multi-index,multi-target syntax>>. Defaults to all indices and data streams in the snapshot, including system indices.
(Optional, string or array of strings) Comma-separated list of indices and data
streams to restore. Supports <<api-multi-index,multi-target syntax>>. Defaults
to all regular indices and regular data streams in the snapshot.
+
You can't use this parameter to restore <<system-indices,system indices or
system data streams>>. Use
<<restore-snapshot-api-feature-states,`feature_states`>> instead.

[[restore-snapshot-api-partial]]
`partial`::
Expand Down
7 changes: 5 additions & 2 deletions docs/reference/snapshot-restore/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@ recover or transfer its data.
[[snapshot-contents]]
== Snapshot contents

By default, a snapshot of a cluster contains the cluster state, all data
streams, and all indices, including system indices. The cluster state includes:
By default, a snapshot of a cluster contains the cluster state, all regular data
streams, and all regular indices. The cluster state includes:

include::apis/restore-snapshot-api.asciidoc[tag=cluster-state-contents]

Expand Down Expand Up @@ -61,6 +61,9 @@ used by the feature. For example, a feature state may include a regular index
that contains the feature's execution history. Storing this history in a regular
index lets you more easily search it.

In {es} 8.0 and later versions, feature states are the only way to back up and
restore system indices and system data streams.

[discrete]
[[how-snapshots-work]]
== How snapshots work
Expand Down
42 changes: 4 additions & 38 deletions docs/reference/snapshot-restore/restore-snapshot.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -82,21 +82,17 @@ GET _snapshot/my_repository/*?verbose=false
You can restore a snapshot using {kib}'s *Snapshot and Restore* feature or the
<<restore-snapshot-api,restore snapshot API>>.

By default, a restore request attempts to restore all indices and data streams
in a snapshot, including <<system-indices,system indices and system data
streams>>. In most cases, you only need to restore a specific index or data
stream from a snapshot. However, you can't restore an existing open index.
By default, a restore request attempts to restore all regular indices and
regular data streams in a snapshot. In most cases, you only need to restore a
specific index or data stream from a snapshot. However, you can't restore an
existing open index.

If you're restoring data to a pre-existing cluster, use one of the
following methods to avoid conflicts with existing indices and data streams:

* <<delete-restore>>
* <<rename-on-restore>>

Some {es} features automatically create system indices on cluster startup. To
avoid conflicts with these system indices when restoring data to a new cluster,
see <<restore-exclude-system-indices>>.

[discrete]
[[delete-restore]]
==== Delete and restore
Expand Down Expand Up @@ -210,37 +206,12 @@ POST _reindex
----
// TEST[continued]

[discrete]
[[restore-exclude-system-indices]]
==== Exclude system indices

Some {es} features, such as the <<geoip-processor,GeoIP processor>>,
automatically create system indices at startup. To avoid naming conflicts with
these indices, use the `-.*` wildcard pattern to exclude system indices and
other dot (`.`) indices from your restore request.

For example, the following request uses the `*,-.*` wildcard pattern to restore
all indices and data streams except dot indices.

[source,console]
----
POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
{
"indices": "*,-.*"
}
----
// TEST[setup:setup-snapshots]
// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
// TEST[s/_restore/_restore?wait_for_completion=true/]

[discrete]
[[restore-feature-state]]
=== Restore a feature state

You can restore a <<feature-state,feature state>> to recover system indices,
system data streams, and other configuration data for a feature from a snapshot.
Restoring a feature state is the preferred way to restore system indices and
system data streams.

If you restore a snapshot's cluster state, the operation restores all feature
states in the snapshot by default. Similarly, if you don't restore a snapshot's
Expand Down Expand Up @@ -271,21 +242,16 @@ Console before restoring the `security` feature state. If you run {es} on your
own hardware, <<restore-create-file-realm-user,create a superuser in the file
realm>> to ensure you'll still be able to access your cluster.

To avoid accidentally restoring system indices, system data streams, and other
dot indices, append the `-.*` wildcard pattern to the `indices` value.

[source,console]
----
POST _snapshot/my_repository/my_snapshot_2099.05.06/_restore
{
"indices": "*,-.*",
"feature_states": [ "geoip" ]
}
----
// TEST[setup:setup-snapshots]
// TEST[s/^/DELETE my-index\nDELETE _data_stream\/logs-my_app-default\n/]
// TEST[s/_restore/_restore?wait_for_completion=true/]
// TEST[s/",/"/]
// TEST[s/"feature_states": \[ "geoip" \]//]

[discrete]
Expand Down
11 changes: 4 additions & 7 deletions docs/reference/snapshot-restore/take-snapshot.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,7 @@ PUT _slm/policy/nightly-snapshots
naming conflicts, the policy also appends a UUID to each snapshot name.
<3> <<snapshots-register-repository,Registered snapshot repository>> used to
store the policy's snapshots.
<4> Data streams and indices to include in the policy's snapshots. This
configuration includes all data streams and indices, including system
indices.
<4> Data streams and indices to include in the policy's snapshots.
<5> If `true`, the policy's snapshots include the cluster state. This also
includes all feature states by default. To only include specific feature
states, see <<back-up-specific-feature-state>>.
Expand Down Expand Up @@ -498,7 +496,7 @@ PUT _slm/policy/nightly-cluster-state-snapshots
<2> Excludes regular data streams and indices.

If you take dedicated snapshots of the cluster state, you'll need to exclude the
cluster state and system indices from your other snapshots. For example:
cluster state from your other snapshots. For example:

[source,console]
----
Expand All @@ -509,7 +507,7 @@ PUT _slm/policy/nightly-snapshots
"repository": "my_repository",
"config": {
"include_global_state": false, <1>
"indices": "*,-.*" <2>
"indices": "*" <2>
},
"retention": {
"expire_after": "30d",
Expand All @@ -521,8 +519,7 @@ PUT _slm/policy/nightly-snapshots

<1> Excludes the cluster state. This also excludes all feature states by
default.
<2> Includes all data streams and indices except system indices and other
indices that begin with a dot (`.`).
<2> Includes all regular data streams and indices.

[discrete]
[[create-snapshots-different-time-intervals]]
Expand Down