Data Types - Add Nested Fields Support

[gmourier]

The definitive specification is due at 2022-04-01

---

Summary

Lists the different data types supported by Meilisearch and adds a specific section for nested structures.

Since the fields are recursively flattened at indexing time internally, it makes it possible for a user to express a nested field location in any API parameter accepting document attributes using the . (dot-notation).

Example:

{
    "id": 0,
    "person": {
         "firstname": "Guillaume"
    }
}

Is internally represented

{
    "id": 0,
    "person.firstname": "Guillaume"
}

It can be expressed this way in any API parameter accepting document attributes.

e.g. "sortableAttributes": ["person.firstname"]

Much more complete examples are present in the specification.

[gmourier]

@Kerollmops and @irevoire, this is mostly taken from the documentation.

Can you double-check the given information from the specification POV before I ask other teams to review it?

Thanks!

[gmourier]

Additional questions

---

@meilisearch/docs-team I've added some examples on how to use dot-notation but I feel this is not the best place to write them. See 3.2.4.7. Dot-notation expression

Is it practical for you to have the information here?

Some API parameters where it could be described are not specified yet. (running late specification of already existing features, e.g. Indexes API, Documents API, Settings API)

---

@irevoire I've added an examples section that demonstrates that the dot-notation can be used to express all properties of an object. See 3.2.4.7. Dot-notation expression section.

e.g. ["person.*"] instead of ["person.firstname", "person.lastname"].

This notation could easily work on some parameters while not on others.

e.g. attributesToHighlight (works-well) vs searchableAttributes (impossible, it introduces equals field weight for a subset of document fields).

Is it possible to manage this notation for each parameter listed in 3.2.4.7.1. Concerned API parameters section and report errors in cases it can't be used? I can create a generic error message for this specific case.

[qdequele]

I've read all the parts about nested fields, including the primitive types. I didn't find anything to complain about. I just wondered about the last example because using person to match all the subfields is not explicit in my opinion but it removes a lot of user frustration, so I like it.

[gmourier]

I've read all the parts about nested fields, including the primitive types. I didn't find anything to complain about. I just wondered about the last example because using person to match all the subfields is not explicit in my opinion but it removes a lot of user frustration, so I like it.

Thanks @qdequele.

This part needs to be decided and precise.

e.g. ["person.*"] instead of ["person.firstname", "person.lastname"].

This notation could easily work on some parameters while not on others.

e.g. attributesToHighlight (works-well) vs searchableAttributes (difficult, it could be translated to having equal field weight for a subset of document fields).

We could first detect this expression on some parameters and block it with an error message, or simply ignore it and specify that each field must be expressly defined with the notation . for a first iteration.

I also like the fact that it allows going faster and that it could remove the frustration of having to explain everything in some cases.

[qdequele]

The problem with doing different behaviors on different routes or resources is that you lose predictability. So it would be nice to have the person.* syntax everywhere.

[gmourier]

What would be the order of importance of the person fields in this case according to you? e.g. searchableAttributes: ["fieldA", "fieldB", "person.*", "fieldD"]

In that case, I would imagine all person object properties to be defined with the same weight but it requires work that seems out-of-scope for the nested fields notation feature.

One solution for this parameter could be to have an "undefined" behavior in the same way as searchableAttributes: ["*"] which defines the order based on the position of the attributes received for the first time in a document. At least, there would be some consistency with the current * notation.

The .* (extended dot-notation) could be easily practicable on:

• attributesToRetrieve (/search, /documents)
• attributesToHighlight (/search)
• facetsDistribution (/search)
• attributesToCrop (/search)
• displayedAttributes (/settings)
• filterableAttributes (/settings)
• sortableAttributes (/settings)

While it does not seems practicable on:

• primaryKey (/index, /documents)
• filter (/search)
• sort (/search)
• rankingRules (/settings - custom ranking rules)

In these cases, using the .* (extended dot-notation) could lead to weird behaviors or even is impossible (e.g. primaryKey).

[gmourier]

Tech Meeting 2022-03-14

Attendees: @irevoire, @gmourier

Following the previous comments on this specification, we have evaluated the notion of wildcarded dot-notation. We were able to make a decision on whether or not to handle wildcards on certain parameters, however, one question remains and we would like to hear your opinions on our various solutions. See searchableAttributes case section.

Decisions

The .* (extended dot-notation) is accepted on those fields:

• attributesToRetrieve (/search, /documents)
• attributesToHighlight (/search)
• facetsDistribution (/search)
• attributesToCrop (/search)
• displayedAttributes (/settings)
• filterableAttributes (/settings)
• sortableAttributes (/settings)

The .* is not accepted and an error will be returned to guide the user for those fields:

• primaryKey (/index, /documents)
• filter (/search)
• sort (/search)
• rankingRules (/settings - custom ranking rules)

searchableAttributes case

Since Meilisearch cannot guarantee the order of arrival in the internal representation of the fields, users can't rely on properties position in their document to expect field weight to be applied following it.

We may want to deeply rework this further in the future.

The important thing to remember is that a user cannot rely on the * notation to determine the weight of fields for the Attribute ranking rule. This also extends to the .* notation regarding the nested object fields weights

Solution 1: Support the wildcard dot-notation as the current *. Vote with: 👍

{
     "searchableAttributes": ["a", "b.*", "c"]
}

Pros:

• The user is able to make nested fields searchable easily and doesn't need to specify them one by one.
• Consistent with the current "undefined" behavior regarding the field weight when * is specified.

Cons:

• Priority of fields are set from their position from the internal representation of the document property. The flattening algorithm may reorder it in some cases and lead it to be "undefined". b.2 could be positioned before b.1 and being prioritized over b.1.

Solution 2: Throw an error when .* is used in searchableAttributes. Vote with: 🚀

Pros:

• User is guided and can't have undefined behavior regarding the weight priority of nested fields.

Cons:

• The user will not be able to make nested fields searchable easily and will have to specify them one by one.

Solution 3: Accept .* at the end of searchableAttributes, throw an error when specified in the middle of searchableAttributes. Vote with: ❤️

{
     "searchableAttributes": ["a", "c", "b.*"]
}

Pros:

• The user is able to make nested fields searchable easily and doesn't need to specify them one by one.

Cons:

• The user is forced to use the .* notation at the end of the searchableAttributes list and thus can't favor a nested object by priority over others fields.

---

Nested notations

Given a nested object b, we could support both b.* and b to specify that all fields from b are listed. There seem to be two schools of thought on this, why not support both?

[curquiza]

Before voting, I need to know, maybe @irevoire can help

• is the wildcard "easy" to implement, I mean, what is the charge of work?
• also, what is the quickest one to implement?

[gmourier]

Regarding the decision #121 (comment)

The .* is not accepted and an error will be returned to guide the user for those fields:
primaryKey (/indexes, /documents)
filter (/search)
sort (/search)
rankingRules (/settings - custom ranking rules)

This statement is not true and Meilisearch can't throw an error.

This is due to the fact that several documents may not share the same schema. Also the fact of being able to specify settings before indexing documents participates in this decision change.

Expected Behavior

For example, one document might have a non-nested field called person while another has a person object containing properties. The chosen behavior is not to force the user's hand.

E.g.

If a user specifies "filter": "person = 'Guillaume'" at search time, the document that would have a nested object person would not be brought up by the filter.

If person is not defined as a filterableAttributes, it will throw an invalid_filter error. This case is managed as today in case a field does not exist.

[dichotommy]

I wasn't clear from reading the PR comments, but based on the last three lines of the spec I understand that wildcard dot-notation will not be included in v0.27.

Pretty clear spec, as always! 🧱 🧱 🧱