Dumps API - Make dump creation an asynchronous task

[gmourier]

🤖 API Diff

Motivation

Today, the route to retrieve the status of a dump is not consistent with other APIs that retrieve the details of a resource and it does not share the same values for the status enum comparing to a task status.

Rather than reworking the dump API endpoints, we propose to make dumps an asynchronous task like the others.

This allows to have a history of dumps created for free and to unify the status directly as well as to provide a way to see the status of a dump creation from its associated task.

Changes

• Dump creation becomes an asynchronous task. The type associated is dumpCreation.
• The generated uid of a dump can be found in the details object of a task under the dumpUid field.
• Removes the 409 conflict dump_already_processing error when a dump is already processed since they can now be enqueued.
• POST on /dumps return a 202 Accepted representing a summarized task.
• The 202 Accepted (Summarized task response) and the related task object define the indexUid to null since dump are related to all indexes.
• dumpCreation tasks are only visible on GET /tasks and /task/task_uid since they are not related to a specific index.
• A dump creation is now a visible asynchronous task. The dump creation task takes priority over other enqueued tasks exactly as today. Its position in the list of tasks is not changed.
• /dumps/:dump_uid/status is removed.
• Remove dumps.get API Key action.

Misc

• Add forgotten task type for 202 Accepted response in the open-api.yml

🚨

• Tasks API - Rename uid to taskUid in the 202 - Accepted Summarized Task Response #144 (comment)

[github-actions]

🚨 Breaking API change detected:

Modified (28)

• DELETE /indexes/{indexUid}/documents
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/documents/{documentId}
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/displayed-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/distinct-attribute
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/filterable-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/ranking-rules
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/searchable-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/sortable-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/stop-words
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/synonyms
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• DELETE /indexes/{indexUid}/settings/typo-tolerance
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• GET /tasks
  • Response modified: 200
    • Body attribute modified: results
• GET /tasks/:taskUid
  • Response modified: 200
    • Body attribute modified: details
• POST /dumps
  • [Breaking] Response modified: 202
    • [Breaking] Body modified
• POST /indexes/{indexUid}/documents
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/documents/delete-batch
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/displayed-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/distinct-attribute
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/filterable-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/ranking-rules
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/searchable-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/sortable-attributes
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/stop-words
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/synonyms
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• POST /indexes/{indexUid}/settings/typo-tolerance
  • Response modified: 202
    • [Breaking] Body attribute modified: type
• PUT /indexes/{indexUid}/documents
  • Response modified: 202
    • [Breaking] Body attribute modified: type

Removed (1)

• [Breaking] GET /dumps/{dumpUid}/status

View documentation diff

Powered by Bump

[irevoire]

Everything looks clear and good to me 👍

If he has the time I would love to see a review from @nicolasvienot though because that's quite a big breaking for the cloud 😬

[bidoubiwa]

My only concern is that a user asking for a dump will potentially wait hours before having the dump if a lot of tasks are in front in the dump.

I don't know if this is problematic.

Apart for that, great spec!

[nicolasvienot]

The spec is clear, thanks @gmourier!

A dump creation is now an asynchronous task. This means that the dump creation task does not take priority over other tasks and that the dump will be created when the task store will not have any other processing task or when other enqueud tasks are not positioned before.

As @irevoire and @bidoubiwa noted, the fact that we can't prioritize a dump anymore and need to wait until all the previous tasks are processed is quite an issue as it means that we won't be able to trigger an update directly on the cloud solution.

What's your opinion on this @qdequele ?

[qdequele]

Hello @gmourier, after a discussion with @nicolasvienot about implementing the update feature (the ability to update a meilisearch version on the cloud manually) by the cloud team, we have some comments on this specification.

tl;dr:
We concluded that it would be much simpler, and the update would be much smoother for the user if the dump creation task is prioritized and not pushed at the end of the task queue.

Explanation:
When the cloud needs to update a meilisearch instance, the simplified process is the following.

• We modify the proxy to prevent users from sending new documents. We will basically send errors to every call made in POST/PUT/PATCH/DELETE on the /indexes/x/documents or /indexes/x/settings/*
• We trigger a new dump creation on the old engine.
• Wait until the dump creation task is finished.
• Download the dump and use it to launch a new server.
• Wait until the new server has indexed the dump and is ready to be used.
• Redirect the proxy traffic to the new server.

The goal is to update the server as fast as possible because the user will not be able to index new documents during the entire process. In a way, we don't want to wait until all tasks are finished because this time could be very long.

[gmourier]

Hey @meilisearch/cloud-team 👋

I updated the specification to mention that, like today, the dumps task will be prioritized before the other enqueued tasks. (the visual order of tasks won't be changed in the tasks list).

It will run as soon as the current processing task finishes, even if others have been enqueued before.

@Kerollmops it could also bypass the current processing task and set a global read transaction to fetch the current state of the B+ Tree of the different LMDBs (indexes) of a Meilisearch instance while a write transaction is in progress. What do you think about it? (This is for future consideration)

[bidoubiwa]

LGTM 👌