Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CCR] Follower info API doesn't return settings for paused indices #37740

Open
jen-huang opened this issue Jan 23, 2019 · 10 comments
Open

[CCR] Follower info API doesn't return settings for paused indices #37740

jen-huang opened this issue Jan 23, 2019 · 10 comments
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >enhancement Team:Deployment Management Meta label for Management Experience - Deployment Management team Team:Distributed Meta label for distributed team

Comments

@jen-huang
Copy link

I have two paused indices, the follower info API does not return settings for them (Max read request operation count, etc):

GET /_all/_ccr/info

{
  "follower_indices" : [
    {
      "follower_index" : "copy-of-kibana_sample_data_logs",
      "remote_cluster" : "prod2",
      "leader_index" : "kibana_sample_data_logs",
      "status" : "paused"
    },
    {
      "follower_index" : "copy-of-kibana_sample_data_flights",
      "remote_cluster" : "prod2",
      "leader_index" : "kibana_sample_data_flights",
      "status" : "paused"
    }
  ]
}

From a UX perspective, it would be nice to:

  1. Resume a follower index with its previous settings (currently, resuming a follower without additional parameters will make it resume following with default settings)
  2. Display a paused follower's previous settings in its Edit form, so that the user can edit their previous values upon resuming

cc @martijnvg

@jen-huang jen-huang added the :Distributed/CCR Issues around the Cross Cluster State Replication features label Jan 23, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@martijnvg
Copy link
Member

The underlying reason why no setting currently cannot be returned, is that the follow settings are part of the shard follow tasks. If an index is paused then there are no shard follow tasks and therefor no settings can't be returned.

@sebelga
Copy link
Contributor

sebelga commented Jan 24, 2019

@martijnvg So we won't have a solution for 6.7 right?

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Jan 24, 2019
@martijnvg
Copy link
Member

@jen-huang @sebelga Keeping the persistent tasks around as part of pausing index following, so that the follow info api can keep reporting the used index follow parameters, is quite impactful.

I started do make the change:
460a0d9

Pause follow marks the the shard follow node tasks as stopped. Resume follow then removes persistent tasks (which also removes the shard follow node tasks) and then starts new persistent tasks.

I don't think we should do that change before feature freeze. @jasontedor @dnhatn Do you agree?

@martijnvg
Copy link
Member

We discussed this during the CCR meeting and we decided that that keeping the shard follow persistent tasks around after a pause follow api has been invoked is pretty invasive change so close to the feature freeze (in terms of size of the change and change in runtime behaviour). This is something that should be implemented in a 7.x release.

Besides keeping the shard follow tasks around after index following as been paused; there is also a need for having the the persistent tasks before the follower index has complete been initialized in order to track the file based restore that is being executed via the internal ccr repository.

@jen-huang
Copy link
Author

Thanks for the update @martijnvg, we'll consider this a future enhancement for 7.x. The current UI/UX for 6.7 will stay as-is with warnings to the user about the ephemeral nature of settings.

@cjcenizal cjcenizal added the :UI label Jan 30, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-ui (:UI)

@rjernst rjernst added Team:Distributed Meta label for distributed team Team:Deployment Management Meta label for Management Experience - Deployment Management team labels May 4, 2020
@cjcenizal cjcenizal removed the :ES-UI label Jun 9, 2020
@tlrx
Copy link
Member

tlrx commented Sep 8, 2021

@martijnvg As an alternative, would it be possible to keep the initial parameters in the index metadata as index settings and reuse them when resuming the persistent task?

@tlrx
Copy link
Member

tlrx commented Sep 8, 2021

Note to the reader: there is a bit of context also in #54996 that I just closed.

@martijnvg
Copy link
Member

@tlrx I think that can work. However these index settings would need to be private and only updatable by us. Then the follower info api can fetch these settings and always return something even if the persistent task doesn't exist. We will need to keep these setting up to date if during a resume parameters change. As an alternative storage place for this, we could use IndexMetadata#customData. The upside is that the info is then always private to us and the follower info api can just fetch it for us from there.

The last idea I remember having around this issue, is that we should keep the persistent task around when pausing index following. I believe today the persistent task gets destroyed, but if add a pause state to the shard follow tasks then the parameters can always be fetched from the persistent task params. However I think this involves much more work then storing the last used settings as index settings or custom data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >enhancement Team:Deployment Management Meta label for Management Experience - Deployment Management team Team:Distributed Meta label for distributed team
Projects
None yet
Development

No branches or pull requests

7 participants