[CCR] Follower info API doesn't return settings for paused indices

[jen-huang]

I have two paused indices, the follower info API does not return settings for them (Max read request operation count, etc):

GET /_all/_ccr/info

{
  "follower_indices" : [
    {
      "follower_index" : "copy-of-kibana_sample_data_logs",
      "remote_cluster" : "prod2",
      "leader_index" : "kibana_sample_data_logs",
      "status" : "paused"
    },
    {
      "follower_index" : "copy-of-kibana_sample_data_flights",
      "remote_cluster" : "prod2",
      "leader_index" : "kibana_sample_data_flights",
      "status" : "paused"
    }
  ]
}

From a UX perspective, it would be nice to:

1. Resume a follower index with its previous settings (currently, resuming a follower without additional parameters will make it resume following with default settings)
2. Display a paused follower's previous settings in its Edit form, so that the user can edit their previous values upon resuming

cc @martijnvg

[elasticmachine]

Pinging @elastic/es-distributed

[martijnvg]

The underlying reason why no setting currently cannot be returned, is that the follow settings are part of the shard follow tasks. If an index is paused then there are no shard follow tasks and therefor no settings can't be returned.

[sebelga]

@martijnvg So we won't have a solution for 6.7 right?

[martijnvg]

@jen-huang @sebelga Keeping the persistent tasks around as part of pausing index following, so that the follow info api can keep reporting the used index follow parameters, is quite impactful.

I started do make the change:
460a0d9

Pause follow marks the the shard follow node tasks as stopped. Resume follow then removes persistent tasks (which also removes the shard follow node tasks) and then starts new persistent tasks.

I don't think we should do that change before feature freeze. @jasontedor @dnhatn Do you agree?

[martijnvg]

We discussed this during the CCR meeting and we decided that that keeping the shard follow persistent tasks around after a pause follow api has been invoked is pretty invasive change so close to the feature freeze (in terms of size of the change and change in runtime behaviour). This is something that should be implemented in a 7.x release.

Besides keeping the shard follow tasks around after index following as been paused; there is also a need for having the the persistent tasks before the follower index has complete been initialized in order to track the file based restore that is being executed via the internal ccr repository.

[jen-huang]

Thanks for the update @martijnvg, we'll consider this a future enhancement for 7.x. The current UI/UX for 6.7 will stay as-is with warnings to the user about the ephemeral nature of settings.

[elasticmachine]

Pinging @elastic/es-ui (:UI)

[tlrx]

@martijnvg As an alternative, would it be possible to keep the initial parameters in the index metadata as index settings and reuse them when resuming the persistent task?

[tlrx]

Note to the reader: there is a bit of context also in #54996 that I just closed.

[martijnvg]

@tlrx I think that can work. However these index settings would need to be private and only updatable by us. Then the follower info api can fetch these settings and always return something even if the persistent task doesn't exist. We will need to keep these setting up to date if during a resume parameters change. As an alternative storage place for this, we could use IndexMetadata#customData. The upside is that the info is then always private to us and the follower info api can just fetch it for us from there.

The last idea I remember having around this issue, is that we should keep the persistent task around when pausing index following. I believe today the persistent task gets destroyed, but if add a pause state to the shard follow tasks then the parameters can always be fetched from the persistent task params. However I think this involves much more work then storing the last used settings as index settings or custom data.