Snapshoted indices data (disk space & more)

[eirc]

Describe the feature:

What I really need is a way to tell how much disk space is used up by specific indices on snapshots.

Currently there is no way to get any statistics on the disk space used by snapshots except for directly querying the snapshot repository's filesystem. This is even more obscure on 5.0 where indices are not stored in folders with the index name but instead using an ID.

Two possible ideas on how to fix this:

• Provide a map of snapshot index IDs to index names on some API and let user go query the filesystem
• Keep the mapping internal and instead directly expose the disk space used (and possibly other data too) on some API essentially keeping users away from the snapshot repository filesystem

Both those approaches could be included for example in the _snapshot API by changing the indices array to contain hashes with data:

/_snapshot/{repository}/_all
{
  "snapshots" : [
    {
      "snapshot" : "mysnapshot-20170302141731",
      "uuid" : "qB7R3TIkQIKcKrDkPIlsLg",
      ...
      "indices" : [
        {
          "name": "nicelogs-2017.03.02"
          "id_on_disk": "jbD8iHU_Toy1kNRGXhS6sA",
          "size_on_disk_bytes": "1000000",
          "more_data_here": "???",
        },
        ...
      ],
      ...
    },
    ...

This is just an example, I know nothing about ES's conventions on naming things in the API

My question that prompted the feature request

[imotov]

This might be tricky. Files that belong to individual indices are shared between multiple snapshots, so it might be misleading to provide stats on per index per snapshot level. We might be able to provide overall per index stats for the entire repository. @abeyad what do you think?

[abeyad]

Index disk space per snapshot will not make much sense as @imotov mentioned because we share index data across snapshots that stays the same. So deleting a snapshot will not necessarily clear up all the disk space used by indexes in that particular snapshot, because another snapshot may contain the same index sharing much of the same data.

We might be able to provide overall per index stats for the entire repository.

Yes, this would be much more feasible to implement.

[eirc]

That's all true, it would make more sense to get global (repository) index stats. For my logging case this is exactly what I need to answer the question how much do past days' log retention costs in disk space. Finding what snapshots need to be deleted to fully delete a specific past day can already be done with the current API.

[tlrx]

We talked about this today with @ywelsch and this issue is similar to #18543, so I'm closing this.