Cannot click beyond page 1000 of search results

[stefanabreitwieser]

This is pretty low-priority, but I was experimenting with some searching, and I noticed that it's impossible to move beyond page 1000 of the search results. If you try to access page 1001 or higher, it returns: Server Error (500).

[jraddaoui]

Hi @stefanabreitwieser,

Right! You're hitting Elasticsearch pagination limit, some notes from their docs:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-from-size.html (at the bottom).
• https://www.elastic.co/guide/en/elasticsearch/guide/current/pagination.html (deep paging note).
• https://www.elastic.co/guide/en/elasticsearch/guide/current/_fetch_phase.html (deep pagination note).

The options we have to avoid this issue are:

1. Increase the index.max_result_window setting value, which also increases the cost of the queries but allows to go over the 1000 page. The question would be, where do we set it?
2. Limit pagination to a fair amount of results and show a notification page when the limit is reached. Combined with some tweaks in the pager to avoid showing pages over the limit.

The other solutions mentioned in their docs (using the Scroll or Search After APIs) are not an option for us because they don't allow to go to an intermediate page and the Scroll API doesn't return aggregations.

Option 1 won't require development time but I'd suggest to go with option 2. With the alternative sort options we have and the filters we're adding in #91, if we tweak the pager it will be rare that someone goes to the 1000+ page.

[stefanabreitwieser]

Thanks Radda! Let's shelve this for now, but keep it as something to fix if we have leftover time at the end.