Settings API - Customize the hard limits for pagination and faceting

[gmourier]

🤖 API Diff

---

Summary

This pull request introduces two new settings API to customize the current hard limits that are blocking users to adjust the search engine behavior to their custom needs.

pagination settings API

Before v0.27, we didn't have a limit on the number of reachable documents for a query. A handful of users complained that this made their database scrappable or subject to DDoS attack, so we decided to introduce a limit of 1000 for v0.27, non-customizable.

Our hypothesis (Some users might need to customize it at a superior range) proved to be true, some users need to customize this limit.

The main argument is that some users need to make the search results scrappable by search engines for their SEO.

Some users also simply want to go further in the pagination despite the relevancy of the first pages.

No one has yet asked to customize it to a lower value, but we imagine this is possible, especially for those who want to reduce the “scrappability” of their dataset.

faceting settings API

Since v0.21, there is no more limit on the number of values returned per facet. This was the case before v0.21 but some users found the default value too low, so it was decided to remove this limit.

The fact that it is unlimited today poses performance problems for users who have many possible values for a facet.

Rather than blocking one or the other, we make this value customizable according to their needs.

Important

Increasing the default values of these properties can degrade performance and more easily expose an instance to DDoS attacks or scrapping by a malicious user.

The default values should be a good compromise between performance and usage for most users.

---

Changes

API Endpoints

• Introduce a pagination settings API resource. GET/PATCH/DELETE - /indexes/:indexName/settings/pagination
  • maxTotalHits property - Type: Integer - Default to 1000
• Introduce a faceting settings API resource. GET/PATCH/DELETE - /indexes/:indexName/settings/faceting
  • maxValuesPerFacet property - Type: Integer - Default to 100
• pagination and faceting objects are accessible from the global settings endpoints /indexes/:indexName/settings

Telemetry Datapoints

• pagination.max_total_hits and faceting.max_values_per_facet are as a property of the Settings Updated event.
• Pagination Updated event is introduced with the max_total_hits property.
• Faceting Updatedevent is introduced with the max_values_per_facet property

Out Of Scope

n/a

---

Attention To Reviewers

Introduced fields like maxTotalHits and maxValuesPerFacet are open to suggestion.

---

Misc

• Update OpenAPI specification file
• Update telemetry datapoints

[github-actions]

🤖 API change detected:

Added (6)

• DELETE /indexes/{indexUid}/settings/faceting
• DELETE /indexes/{indexUid}/settings/pagination
• GET /indexes/{indexUid}/settings/faceting
• GET /indexes/{indexUid}/settings/pagination
• PATCH /indexes/{indexUid}/settings/faceting
• PATCH /indexes/{indexUid}/settings/pagination

Modified (2)

• GET /indexes/{indexUid}/settings
  • Response modified: 200
    • Body attributes added: pagination, faceting
• PATCH /indexes/{indexUid}/settings
  • Body modified
    • Body attributes added: pagination, faceting

View documentation diff

Powered by Bump

[brunoocasali]

@gmourier give a look at the PATCH/PUT definition here I think it is not compatible with this one #152 😄 .

[bidoubiwa]

limited_to

What do you think of max_total_hits or total_hit_limit.

It goes in line with estimatedTotalHits

maxValuesPerFacet

Makes me think we can customize this value for every facet. Like genres: 20, color: 100

If this is not the case, what about something more generic like: facet_values_limit or max_facet_values

?

[dichotommy]

I have doubts about the name pagination.

Usually when we speak about pagination, we speak about how results are sent to and displayed on the front end—particularly when the total number of results is broken down into smaller pages.

This setting on the other hand affects only the total number of results returned by the engine. So it doesn't have anything to do with pagination as far as I can tell. Won't it create confusion?

Edit: Just wanted to say I totally understand if the pagination setting is being created because we plan to add additional pagination-related settings in the future, but I'm still not sure that this particular setting (search response maximum) should belong to that category.

[dichotommy]

limited_to
What do you think of max_total_hits or total_hit_limit.

Strong agree with max_total_hits 👍🏻

maxValuesPerFacet
Makes me think we can customize this value for every facet. Like genres: 20, color: 100

For me, the name maxValuesPerFacet is really clear. When I read maxValuesPerFacet: x, I think that the engine will show only the x most common facet values for each attribute defined in filterableAttributes.

[gmourier]

Thanks for your comments on the names! Do you have a better name than pagination knowing that the pain points about this limit have been expressed for this use case and that we will probably add more parameters about it? Unfortunately, I don't have a better suggestion to make.

[dichotommy]

Hi @gmourier , if you know that you will add more pagination-related settings in the future and wish to keep this one within that category, then I completely understand and support the naming decision.

On the other hand, I think this setting can be useful and interesting to people besides those who need it for pagination. One example that you've already brought up is security, to prevent scraping of the database. There may be other cases we haven't considered where users want to enforce a base limit on all search queries without using the parameter. My concern would be that users could miss this setting if it's in an object called pagination.

I think this setting would be easier to find as an int at the root level of the settings resource with a name like resultsLimit, maxResults, maxHits or similar. Just my two cents.

[bidoubiwa]

Poll time!

Limit the number of hits on which we can paginate

Little context, currently it would be inside a pagination object. For example:

{
   pagination: {
      limitedTo: 100
   }
}

This might change depending on another poll.

• limitedTo 👍
• maxTotalHits 🎉
• totalHitLimit 🚀
• other 👀

See previous comments for arguments.

[bidoubiwa]

Maximum values returned by each facet

Little context, the setting is inside a faceting object. For example:

{
   faceting: {
      maxValuesPerFacet: 100
   }
}

• maxValuesPerFacet 👍
• facetValuesLimit 🚀
• maxFacetValues 🎉
• other 👀

See previous comments for arguments.

[bidoubiwa]

Pagination object for nested limit on hits

Currently, the limitedTo representing the maximum number of search results on which a user can paginate.

Example:

{
   // other settings
   pagination: {
      limitedTo: 100
   }
}

The question is, do we want to call it pagination do we want to put the setting at the root:

• Keep pagination as the root object containing limitedTo (name can very based on this poll) 👍
• Put limitedTo at the root of the settings 🎉
• other 👀

See previous comments for arguments.