Expose Lucene's FeatureField.

[jpountz]

Lucene has a new FeatureField which gives the ability to record numeric
features as term frequencies. Its main benefit is that it allows to boost
queries with the values of these features and efficiently skip non-competitive
documents at the same time using block-max WAND and indexed impacts.

[elasticmachine]

Pinging @elastic/es-search-aggs

[mayya-sharipova]

@jpountz Thanks Adrien, this is a very interesting and necessary feature. Excited to have in elasticsearch!

I am wondering if there an intention to index multiple values for a feature. With you current PR, if I try to index multiple values:

{ "index" : { "_index" : "findex", "_type" : "_doc", "_id" : "2" } }
{  "text" : "newspaper", "pagerank": [100.0, 200] }

I am getting the following in the explanation of query score (looks like multiple values got converted to the max float value):

"_explanation": {
	"value": 88.72284,
	"description": "Log function on the _feature field for the pagerank feature, computed as w * log(a + S) from:",
	"details": [
		{
			"value": 1.0,
			"description": "w, weight of this function",
			"details": []
		},
		{
			"value": 4.0,
			"description": "a, scaling factor",
			"details": []
		},
		{
			"value": 3.4028235E38,
			"description": "S, feature value",
			"details": []
		}
	]
}

[jpountz]

Thanks @mayya-sharipova, this is a very good observation, we should reject multi-valued fields explicitly!

[jimczi]

LGTM

[jimczi]

🥇

[mayya-sharipova]

@jpountz Thanks. Just one thing I want to clarify for myself. What does this phrase mean in the explanation below? What is w/2 here?

"k, pivot feature value that would give a score contribution equal to w/2"

"_explanation": {
	"value": 0.13026857,
	"description": "Saturation function on the _feature field for the url_length feature, computed as w * S / (S + k) from:",
	"details": [
		{
			"value": 1.0,
			"description": "w, weight of this function",
			"details": []
		},
		{
			"value": 0.33333334,
			"description": "k, pivot feature value that would give a score contribution equal to w/2",
			"details": []
		},
		{
			"value": 0.049926758,
			"description": "S, feature value",
			"details": []
		}
	]
}

[jpountz]

Thanks for testing @mayya-sharipova. It should be S/2 indeed, w doesn't make sense.

[jpountz]

Actually I read too quickly, the current explanation is correct: the score is computed as w * S / (S + k). So when S is equal to k, this becomes w * k / (k + k) = w / 2. I suspect it might be a bit confusing due to the fact that on the Lucene side we give users an explicit way to configure the boost (w) because from a Lucene perspective that's another query wrapper to use. However I don't think we need to expose it to Elasticsearch users since all queries already support configuring a boost, so if you would need to boost the impact of the feature query by 2, you could just do:

{
  "query": {
    "feature": {
      "field": "pagerank",
      "boost": 2
    }
  }
}

[mayya-sharipova]

@jpountz Thanks for the detailed explanation