Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CCR] Paused follower does not display in follower stats #37127

Closed
jen-huang opened this issue Jan 3, 2019 · 14 comments
Closed

[CCR] Paused follower does not display in follower stats #37127

jen-huang opened this issue Jan 3, 2019 · 14 comments
Assignees
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features

Comments

@jen-huang
Copy link

jen-huang commented Jan 3, 2019

I created a follower index, kibana_sample_data_logs.

Then I paused the follower index using:

POST /kibana_sample_data_logs/_ccr/pause_follow

Then I attempted to retrieve the stats of the follower index using both:

GET /kibana_sample_data_logs/_ccr/stats

and

GET /_ccr/stats

Neither one lists kibana_sample_data_logs as part of indices array. I resumed the follower using:

POST /kibana_sample_data_logs/_ccr/resume_follow
{}

and then the stats API showed the correct information.

For CCR UI, we need the stats API to:

  1. Show paused follower indices
  2. Show status for each follower index - whether it is paused or active. Could just be a boolean paused flag.

This will make it so that we can give the user the correct option in the UI (whether to resume or pause follower index based on its status).

cc @elastic/es-ui

@jen-huang jen-huang added the :Distributed/CCR Issues around the Cross Cluster State Replication features label Jan 3, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@martijnvg
Copy link
Member

So the reason that no stats are returned for paused follower indices, is that there are no shard follow tasks when index following is paused and the follow stats api is based on returning stats entries for shard follow tasks.

I think the best way for the UI is to figure out all the follower indices by checking whether the index setting index.xpack.ccr.following_index has been set to true via the cluster state api, then for each follower index check whether index following is active by checking the follow stats api. If there are no shard follow stats entries then index following is paused. Would that work?

@jen-huang
Copy link
Author

@martijnvg Thanks for the insight. While using cluster state api would work, the user could potentially have a large number of indices and we will have to look through each index's xpack.ccr.following_index setting. I think the CCR api is missing a key endpoint to "get list of all follower indices and their status". This will make our future UI enhancement work easier, and round out the CCR api UX experience as a whole.

Open for more discussion on this. Let me know if a solution might be feasible for 6.7, otherwise for 6.7 we will investigate the cluster state api route and/or adjust UI scope.

@martijnvg
Copy link
Member

@jen-huang I thought a bit more about this and I think we can change the follow stats api to also include follower indices that are paused. These indices would then we returned without any stats entries and would look like this:

{
   "indices": [
       {
            "index": "my_paused_index",
            "shards" : [] 
       },
       {
            "index": "my_active_index",
            "shards" : [
               {
                    ...
               },
               ....
            ] 
       }
   ]
}

Here the my_paused_index here has zero shard stats entries, because it is paused, but it is a follower index managed by ccr. Would this work for the UI? This change shouldn't require a lot of work on the ES side.

@jasontedor @dnhatn Do you think adding follower indices with empty shard entries is a good idea here in order to include paused follower indices in the stats api? This way the UI will then only need to check one api display stats for ccr follower indices (instead of the workaround mentioned above).

@jen-huang
Copy link
Author

jen-huang commented Jan 8, 2019

@martijnvg Yes, having zero shard stats would work for the UI 😄

@sebelga
Copy link
Contributor

sebelga commented Jan 9, 2019

@martijnvg Does having 0 shards means that it is paused?

What if ES would provide then a "status" field in that case

{
   "indices": [
       {
            "index": "my_paused_index",
            "status": "paused"
       },
       {
            "index": "my_active_index",
            "status": "running",
            "shards" : [
               {
                    ...
               },
               ....
            ] 
       }
   ]
}

@martijnvg
Copy link
Member

Does having 0 shards means that it is paused?

Yes, having 0 follow shard stats entries in the response means a follow index is paused.

What if ES would provide then a "status" field in that case

I think we could add a status field to make the the status of a follower index more clear.

@martijnvg martijnvg self-assigned this Jan 10, 2019
@jen-huang
Copy link
Author

Hi @martijnvg, @sebelga has started work on providing the follower index advanced settings UI at the time of creation: elastic/kibana#28267

The advanced settings include max_read_request_operation_count, max_outstanding_read_requests, etc - everything per create follower api documentation: https://www.elastic.co/guide/en/elasticsearch//reference/master/ccr-put-follow.html#_request_body_2

If the user has specified any of these settings, we would like to display that later on in the UI for their reference. However, we don't have any way to read out their settings after creation. Would it be possible to modify the stats api to return this information as well?

@martijnvg
Copy link
Member

martijnvg commented Jan 11, 2019

If the user has specified any of these settings, we would like to display that later on in the UI for their reference. However, we don't have any way to read out their settings after creation.

This should be visible in the cluster state under persistent_tasks if index following has started. For each follow shard there is an entry that contains the value for the advanced setting. So it is kind of duplicated and another place to look inside the cluster state, but it is there.

Would it be possible to modify the stats api to return this information as well?

I think it makes sense to add the used index follow parameters as part of the follow stats api. Then there is a single place all information about for ccr index following can be read.

@martijnvg
Copy link
Member

After thinking more about this, I think we should not include the follow index parameters / settings inside of the follow stats api. The follow stats api returns very dynamic information about the replication processes that ccr manages for each follow index shard; which is a different kind of information than the follow index parameters, what the status is of a follower index or which indices are follower indices.

What I think we should do instead, is add another api that returns that kind of information. I think we should add a follow info api. Which returns all the follower indices and for each follower index returns the status (active or paused) and index follow parameters. Then we can leave the follow stats api as is. The structure of the response of the follow info api would then look like this:

{
   "indices": {
        {
            "index": "follower_index1",
            "status": "active",
            "parameters": {
                 "remote_cluster": "eu_cluster",
                  "leader_index": "leader_index1",
                  "max_read_request_operation_count": 1024,
                  ...
             }
        },
        {
           "index": "follower_index2",
           ...
        },
        ...
    }
}

@yaronp68
Copy link

@martijnvg The separate list of follower indices API you mention will be very useful for the user IMO
In the GUI we will be able to use it for creating a list of followers with the dynamic information we need without hitting the stats API all the time

@sebelga
Copy link
Contributor

sebelga commented Jan 11, 2019

Agreed @martijnvg . A separate API would be better for listing follower indices. Cheers!

@jen-huang
Copy link
Author

@martijnvg ++ to that suggestion - a separate API would be perfect and fills a large gap.

@bmcconaghy
Copy link

++ to separate API. It would be useful for index management UI as well.

martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Jan 13, 2019
This api returns all follower indices and per follower index
the provided parameters at put follow / resume follow time and
whether index following is paused or active.

Closes elastic#37127
martijnvg added a commit that referenced this issue Jan 18, 2019
* Add ccr follow info api

This api returns all follower indices and per follower index
the provided parameters at put follow / resume follow time and
whether index following is paused or active.

Closes #37127

* iter

* [DOCS] Edits the get follower info API

* [DOCS] Fixes link to remote cluster

* [DOCS] Clarifies descriptions for configured parameters
martijnvg added a commit that referenced this issue Jan 18, 2019
This api returns all follower indices and per follower index
the provided parameters at put follow / resume follow time and
whether index following is paused or active.

Closes #37127
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features
Projects
None yet
Development

No branches or pull requests

6 participants