[Snapshot & Restore] Snapshot flow for system indices is broken in 8.0+

[jrodewig]

With elastic/elasticsearch#79670, you can no longer use the indices parameter of the ES create snapshot policy API or the ES restore snapshot API to back up or restore system indices in 8.0+. The creates a few issues for the corresponding snapshot UIs:

• The create snapshot policy wizard lets you select system indices. However, if you run the policy, the resulting snapshot doesn't contain the selected system indices. To avoid this, we should exclude system indices from the picker entirely. Note: This behavior may be changing with Disallow explicitly requesting system indices in snapshots elasticsearch#81235.

• Similarly, the restore snapshot wizard lets you select system indices to restore. However, a restore request that includes a system index returns an error. We should update the picker to exclude system indices.

• Since the snapshot UI doesn't support feature states, users can no longer selectively back up or restore system indices. You can only back up or restore all system indices as part of the cluster state. This can be an issue if a user only wants to restore specific system indices. For example, a user may want to restore their Kibana feature state but not override the security feature state that controls authentication.

Relates to #95128

[elasticmachine]

Pinging @elastic/kibana-stack-management (Team:Stack Management)

[sabarasaba]

For addressing the first point on hiding system indices when creating a policy a new ES issue has been created elastic/elasticsearch#82671.

For hiding system indices when restoring a snapshot we can leverage the feature_states array returned by the get snapshot api that we currently use. I created a PR for dealing with this #123365

[yuliacech]

With @sabarasaba's PR#123365 merged, system indices are no longer listed when restoring a snapshot and when elasticsearch#82671 will be implemented, system indices will not be displayed in the indices field when creating an SLM policy.

Currently, it's already not possible to include system indices via "indices" field (elasticsearch#79670 and elasticsearch#81235): when a wildcard is used, matching system indices will be ignored and when a system index is included by name, an error will be thrown when the policy is run. Users can only backup all system indices by enabling the toggle "include global state".

We need to add a combobox, when the toggle is disabled, that would list available features as options and would allow the user to select specific features for backup. A list of features is returned by the Get features API. The same combobox should be added to the "restore snapshot flow" as well (see #95128)

Also, the validation should allow the user to either backup only "normal" data streams and indices or only features (system indices). Currently, the "indices" field is mandatory in the UI (but not in the API).