field stats api should return results for fields in the mapping, but without indexed documents

[BigFunger]

Describe the feature:
Related to #17750

Since version 5.0, Kibana has had a dependency on the field stats api to determine which fields are aggregatable, and searchable. Because of this, the field stats API should return a subset of information that contains at least these propertied for fields in the mapping, but do not yet have documents indexed.

Example linked here:
https://gist.github.com/ycombinator/757cd9cd7303933abf3bc3b7e477c1fd#file-fieldstats-nodata-md

[clintongormley]

Fixed by #21821

[Bargs]

@clintongormley @jimczi I'm not sure #21821 fixes this. I just tried the following steps is a fresh build of ES master:

Create an index with a field that should be searchable and aggregatable based on its mapping:

PUT localhost:9200/nodata

{
    "mappings" : {
        "nodata" : {
            "properties" : {
                "nope" : { "type" : "integer" }
            }
        }
    }
}

GET http://localhost:9200/nodata/_field_stats?fields=*

{
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "indices": {
    "_all": {
      "fields": {}
    }
  }
}

Am I doing something wrong or is there a misunderstanding?

This issue is biting us in a few different ways right now:

elastic/kibana#9571
elastic/kibana#9028

[jimczi]

#21821 is about non-searchable fields so with index:no in the mapping. This means that a field will be missing from the field stats output only if it doesn't exist in any of the requested indices (as doc_values, indexed or stored field).
field_stats is currently not able to return any information on the field if there is no document with that field in the cluster. This is a known limitation and there is nothing we can do about it for the moment (which is why this issue is still open).

[Bargs]

Thanks for the clarification @jimczi, I was just confused by @clintongormley's comment above.

[jimczi]

I understand the need for Kibana to give hints to people using the UI. Like here is the list of fields that we think could work for this agg. Though it should be just an hint and user should still be able to set a custom field if they want. We also have issues with custom field mapper that could decide to not return any field_stats output. So yeah field_stats should not be trusted always ;)

[Bargs]

So yeah field_stats should not be trusted always ;)

Good to know! We'll try to come up with a solution in Kibana

[clintongormley]

Thanks for the clarification @jimczi, I was just confused by @clintongormley's comment above.

I was confused too :) Thanks @jimczi for the clarification

[spalger]

field_stats should not be trusted always

That's concerning....

it should be just an hint and user should still be able to set a custom field if they want.

What Kibana should/shouldn't do is carefully considered and we disagree with you here. We think it's a better user experience if Kibana and Elasticsearch work together to determine the exact list of options that the user has available. Not a list of things that might, probably, or could work.

In order to determine this we have pushed for prs like #17750 so the authority on the matter, elasticsearch, can tell us what it knows. When it says a field is not searchable or aggregatable there is nothing that should be more trustworthy.

[s1monw]

this feature make not much sense to me on field stats. Field stats can't be returned if there are no documents which implies that there are no stats. I also don't understand why the fields stats API returns mapping information at all? Why isn't this a dedicated API? Field stats is really about the inverted index and that's about it?

[spalger]

Maybe @clintongormley can remember where we discussed this, but we talked at length about making this into it's own API and compromised on this solution.

[clintongormley]

I'm happy to revisit the APIs that we have. There's get-mapping, get-field-mapping, and field-stats. I'd like to get rid of get-field-mapping in its current form as it doesn't quite do the job. Perhaps we need to replace get-field-mapping with the mapping aspects of field-stats, and then make field-stats JUST about the stats?

Also see #5368

[droberts195]

Coming up with some sort of solution to this (though not necessarily what the title of the issue says) is very important for the user experience of the machine learning UI.

The problem for ml is that when you create a new job we're in the situation of knowing there are going to be certain fields in the results eventually, but not having any results at the instant the job is created. We'd like to set up Kibana to be able to visualise our results as soon as they exist, but at present this is not possible because of the fact that Kibana is using the field stats API to populate its index pattern mappings and the field stats API doesn't return what Kibana needs for fields that don't exist in any indexed document.

cc @peteharverson @jgowdyelastic @sophiec20

[clintongormley]

@jimczi is working on a new field-capabilities API which will rely only on mapping, not on stats (or which docs are indexed). It will return field type and is_aggregatable and is_searchable only, with fields merged by type, index, or cluster.

[Hronom]

Is this available in Kibana 5.3.0 ?

[spalger]

@Hronom follow elastic/kibana#11014