Bug: Suggest request for a CustomSuggester fail with "ReduceSearchPhaseException" in Multi-DataNode scenarios.

[updixit]

**Elasticsearch version (**bin/elasticsearch --version): 1.7 and above

Plugins installed: []

JVM version (java -version): Jdk 1.8

OS version (uname -a if on a Unix-like system): Windows

Description of the problem including expected versus actual behavior:
Calling 'suggest' requests for a CustomSuggester works as expected when all shards are present on a single data Node. If there are >1 Data Node, the suggest fails with the following error in the reduce phase.
"Can't merge suggest result, this might be caused by suggest calls across multiple indices with different analysis chains. Suggest entries have different sizes actual [0] expected [1]);"

Reason:
Define a CustomSuggester which returns suggestions of 'CustomSuggestions' type(derived from Suggest.Suggestions class).
In the 'SearcPhaseController.java' when the suggest results from different shards are merged, the results from the coordinating DataNode are of type 'CustomSuggestion' but the results from all other data nodes are down-casted to Suggest.Suggestion type. This causes the reduce phase to fail.

Steps to reproduce:

1. Define CustomSuggestion class( extends Suggestion class). The 'Option' of the CustomSuggestion.Entry should have an additional attribute ( This can be any attribute. string, int float. eg: int 'randomAttribute'. Let's say this has a constant value =1).
2. Register the Custom Suggester in the Suggest Module.
3. Create a cluster with a single Data node.
   Issue request against the 'CustomSuggester'. All request succeed. The Options in the returned response contain the 'randomAttribute' as well.
4. Create a cluster with a >1 Data node.
   Issue request against the 'CustomSuggester'. Request fail with ReduceSearchPhaseException, while merging the results in the co-ordinating node.
5. On Debugging, you will notice that some results are of type "CustomSuggestion" and some are of type "Suggestion".

Questions
Why does this down casting happen?
How can I register the suggester so that the response is not downcasted to the Base Suggestion class?

Provide logs (if relevant):
2017-09-11 10:14:13,318][WARN ][transport.netty ] [localhost-1] Message not fully read (response) for [96] handler org.elasticsearch.search.action.SearchServiceTransportAction$6@35c82c9d, error [false], resetting
[2017-09-11 10:14:13,319][WARN ][transport.netty ] [localhost-1] Message not fully read (response) for [93] handler org.elasticsearch.search.action.SearchServiceTransportAction$6@5bcb6480, error [false], resetting
[2017-09-11 10:14:13,319][DEBUG][action.search.type ] [localhost-1] failed to reduce search
org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce]
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.onFailure(TransportSearchQueryThenFetchAction.java:159)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:41)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalStateException: Can't merge suggest result, this might be caused by suggest calls across multiple indices with different analysis chains. Suggest entries have different sizes actual [0] expected [1]
at org.elasticsearch.search.suggest.Suggest$Suggestion.reduce(Suggest.java:256)
at org.elasticsearch.search.suggest.Suggest.reduce(Suggest.java:185)
at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:396)
at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.doRun(TransportSearchQueryThenFetchAction.java:147)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
... 3 more

[tlrx]

Why does this down casting happen?

I think this is related to the way Suggestion are deserialized. In Suggest.readFrom() method, custom suggestion are instanciated back as Suggestion, while suggestion created on the same node remain CustomSuggestion. I think we should fix that.

[tlrx]

@updixit Just to be sure, did you reproduce it on a recent version like 5.6 or 6.0 betas?

[updixit]

I didn't have an opportunity to test this with ES5.6.
However, looking at the implementation of 'Suggest' in ES5.6, the logic of Suggest.readFrom() seems to be the same as in ES1.7, so it's very likely to repro in the latest versions as well.

[colings86]

@jimczi or @andyb-elastic could you take a look at this?

[andyb-elastic]

@colings86 sure thing

[andyb-elastic]

It definitely makes sense that there would be problems when plugin suggesters use their own types extending Suggest.Suggestion given how they're deserialized. We have a test for plugin suggesters that 1) doesn't extend those types and 2) only uses 1 data node, so it wouldn't catch it.

I wasn't able to reproduce this error by modifying those tests however, either on 1.7 or master

1.7: andyb-elastic@72de777
master: andyb-elastic@809394e

Are the changes in those commits roughly what's being described here or am I misunderstanding?

[andyb-elastic]

I'm able to reproduce this and working on a fix. Suggestion and its component classes need to be made serializable as NamedWriteables so there's some untangling to do

[jimczi]

cc @elastic/es-search-aggs

[andyb-elastic]

This was fixed in #30284 which will land in 7.0. See the custom-suggester example plugin for an illustration of how to provide a suggester in a plugin.