Duplicate IDs when indexing

[philippeauriach]

Describe the bug
Uploading several batches of the same content result in duplicates ids in meilisearch. Objects are coming from a Strapi service, and sent to a meilisearch instance (local docker latest image : 0.22.0) via the meilisearch nodejs package.

Full reproduction project available at : https://github.com/philippeauriach/meilisearch-dup-id

To Reproduce

• clone https://github.com/philippeauriach/meilisearch-dup-id
• configure your meilisearch instance endpoint in hooks/meilisearch/utils.js if needed
• yarn install
• yarn develop
• => you should have 27 products in your meilisearch instance (expected behavior)
• kill the process, run it again yarn develop
• => meilisearch now has 54 products (broken behavior, products are duplicated)

Expected behavior
Products should be replaced when having the same ID

Screenshots

MeiliSearch version: v0.22.0

Additional context
Strapi ?

[curquiza]

Thank you so much for the report!
@ManyTheFish is investigation on it 🙂

[ManyTheFish]

Hello @philippeauriach! Thanks for your bug report, this should be fixed in the next version of Meilisearch.

The bug occurs because a Document addition of 0 documents is made between other document addition.
To fix the bug in your application, you can change a bit your code in order to avoid empty document addition:

philippeauriach/meilisearch-dup-id#1

Thanks again for your issue! 👍

[khash]

FWIW, I've come across this issue with a pure Ruby client. Now I don't know if it is the same issue or not, but there is how you can recreate it:

(in this example index = client.index('acme'))

1. Add some docs index.add_documents([{id: "foo"}, {id: "bar"}])
2. Add an empty array index.add_documents([])

This will break the index in a way that while the API returns the documents correctly with /indexes/acme/documents, doing a GET on /indexes/acme/documents/foo returns a 404.

This means if you run the first line again (add docs), you will end up with duplicates.

This is caused by adding an empty array. Here is how you can reproduce it with only CURL on a clean setup:

$ curl -X POST http://localhost:7700/indexes/acme/documents --data '[{"id": "foo"}, {"id":"bar"}]'

$ curl http://localhost:7700/indexes/acme/documents/foo
{"id":"foo"}

$curl -X POST http://localhost:7700/indexes/acme/documents --data '[]'

$curl http://localhost:7700/indexes/acme/documents/foo
{"message":"Document with id foo not found.","errorCode":"document_not_found","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#document_not_found"}

All the while, curl http://localhost:7700/indexes/acme/documents returns the correct set of documents which makes things very confusing.

[curquiza]

Hello @khash! Your bug should not be present in the next version. I did not succeed to reproduce your issue on our branch that will be merged into main soon :)

[curquiza]

Closed by #1711

[Arsapol]

I face this problem in version 0.29.1 when I update index setting more than 1 time.

Before update : Every works fine
After update : Document data duplicate with the same id

[curquiza]

Hello @Arsapol

Is it related to this? #3021

Otherwise, can you open an issue with the detailed steps to reproduce?

[Kerollmops]

Indeed it looks related to #3021 and will be fixed in the next release of Meilisearch, i.e. v0.30.