Dumps API - Make dump creation an asynchronous task (#139)

* wip * Make a dump creation a visible asynchronous task * Add precisions * Update open-api.yml * Add ommited type field for summarized task response * Add future possibilities * Apply suggestions from code review Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com> * Precise that indexUid can be null * Precise priorization of dumpCreation task over other task types * Keep taskUid for 202 response * remove dumps.get API key action Co-authored-by: cvermand <33010418+bidoubiwa@users.noreply.github.com>
meilisearch · Jul 7, 2022 · f01752e · f01752e
1 parent f2ba7e2
commit f01752e
Show file tree

Hide file tree

Showing 5 changed files with 52 additions and 159 deletions.
diff --git a/open-api.yaml b/open-api.yaml
@@ -341,6 +341,9 @@ components:
               description: Value for the primaryKey field encountered if any for indexCreation or indexUpdate task.
             settings:
               $ref: '#/components/schemas/settings'
+            dumpUid:
+              type: string
+              description: Identifier generated for the dump for dumpCreation task.
         error:
           description: Error object containing error details and context when a task has a failed status.
           $ref: '#/components/schemas/error'
@@ -541,37 +544,6 @@ components:
         - numberOfDocuments
         - isIndexing
         - fieldDistribution
-    dumps:
-      type: object
-      additionalProperties: false
-      examples:
-        - uid: 20210719-114144097
-          status: done
-          startedAt: '2021-07-19T14:31:16.920473Z'
-          finishedAt: '2021-07-19T14:32:16.920473Z'
-      title: dumps
-      description: ''
-      properties:
-        uid:
-          type: string
-          example: 20200929-114144097
-          description: ' Unique identifier for the dump'
-        status:
-          type: string
-          enum:
-            - in_progress
-            - failed
-            - done
-          example: in_progress
-          description: Status of the dump
-        startedAt:
-          $ref: '#/components/schemas/timestamp'
-        finishedAt:
-          $ref: '#/components/schemas/timestamp'
-      required:
-        - uid
-        - status
-        - startedAt
     error:
       title: error
       type: object
@@ -620,7 +592,6 @@ components:
               - settings.update
               - stats.get
               - dumps.create
-              - dumps.get
               - version
         indexes:
           type: array
@@ -887,13 +858,15 @@ components:
               - uid
               - indexUid
               - status
+              - type
               - enqueuedAt
           examples:
             Example:
               value:
                 taskUid: 0
                 indexUid: movies
                 status: enqueued
+                type: documentAddition
                 enqueuedAt: '2021-07-19T14:31:16.920473Z'
     '204':
       description: No Content
@@ -996,59 +969,17 @@ paths:
           content:
             application/json:
               schema:
-                $ref: '#/components/schemas/dumps'
+                $ref: '#/components/responses/202'
               examples:
-                Created:
+                '202':
                   value:
-                    uid: 20210719-114144097
-                    status: in_progress
-                    startedAt: '2021-07-19T14:31:16.920473Z'
-        '401':
-          $ref: '#/components/responses/401'
-  '/dumps/{dumpUid}/status':
-    get:
-      operationId: dumps.get
-      summary: Get dump status
-      description: |
-        Get the status of a dump creation process using the uid returned after calling the [dump creation route](https://docs.meilisearch.com/reference/api/dump.html#create-a-dump).
-
-        The returned status could be:
-
-        * `in_progress`: Dump creation is in progress.
-        * `failed`: An error occured during dump process, and the task was aborted.
-        * `done`: Dump creation is finished and was successful.
-      tags:
-        - Dumps
-      security:
-        - apiKey: []
-      responses:
-        '200':
-          description: Ok
-          content:
-            application/json:
-              schema:
-                $ref: '#/components/schemas/dumps'
+                    taskUid: 0
+                    indexUid: null
+                    status: enqueued
+                    type: dumpCreation
+                    enqueuedAt: '2021-07-19T14:31:16.920473Z'
         '401':
           $ref: '#/components/responses/401'
-        '404':
-          description: Not Found
-          content:
-            application/json:
-              schema:
-                $ref: '#/components/schemas/error'
-              examples:
-                Not Found:
-                  value:
-                    message: Dump `20201001-110357260` not found
-                    code: dump_not_found
-                    type: invalid_request
-                    link: 'https://docs.meilisearch.com/errors#dump_not_found'
-    parameters:
-      - schema:
-          type: string
-        name: dumpUid
-        in: path
-        required: true
   /health:
     get:
       operationId: health.check

diff --git a/text/0060-tasks-api.md b/text/0060-tasks-api.md
@@ -21,10 +21,10 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
 | field      | type    | description                                                                                                                                                                                                                   |
 |------------|---------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | uid        | integer | Unique sequential identifier                                                                                                                                                                                                  |
-| indexUid   | string  | Unique index identifier                                                                                                                                                                                                       |
+| indexUid   | string  | Unique index identifier. This field is `null` when the task type is `dumpCreation`.                                                                                                                                                                                                                                        |
 | batchUid   | integer | Identify in which batch a task has been grouped by auto-batching. It corresponds to the first task uid grouped within a batch. See [0096-auto-batching.md](0096-auto-batching.md)                                             |
 | status     | string  | Status of the task. Possible values are `enqueued`, `processing`, `succeeded`, `failed`                                                                                                                                       |
-| type       | string  | Type of the task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate`                                                                       |
+| type       | string  | Type of the task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate`, `dumpCreation`                                                       |
 | details    | object  | Details information for a task payload. See Task Details part.                                                                                                                                                                |
 | error      | object  | Error object containing error details and context when a task has a `failed` status. See [0061-error-format-and-definitions.md](0061-error-format-and-definitions.md)                                                         |
 | duration   | string  | Total elapsed time the engine was in processing state expressed as an `ISO-8601` duration format. Times below the second can be expressed with the `.` notation, e.g., `PT0.5S` to express `500ms`. Default is set to `null`. |
@@ -36,12 +36,12 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
 
 ##### Summarized `task` Object for `202 Accepted`
 
-| field      | type    | description                                                                      |
-|------------|---------|----------------------------------------------------------------------------------|
-| taskUid    | integer | Unique sequential identifier                                                     |
-| indexUid   | string  | Unique index identifier                                                          |
-| status     | string  | Status of the task. Value is `enqueued`                                          |
-| type       | string  | Type of the task.                                                                |
+| field      | type    | description                     |
+|------------|---------|---------------------------------|
+| taskUid    | integer | Unique sequential identifier           |
+| indexUid   | string  | Unique index identifier. This field is `null` when the task type is `dumpCreation`. |
+| status     | string  | Status of the task. Value is `enqueued` |
+| type       | string  | Type of the task. |
 | enqueuedAt | string  | Represent the date and time as `RFC 3339` format when the task has been enqueued |
 
 
@@ -68,6 +68,7 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
 | documentAdditionOrUpdate |
 | documentDeletion         |
 | settingsUpdate           |
+| dumpCreation |
 
 > 👍 Type values follow a `camelCase` naming convention.
 
@@ -119,6 +120,16 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
 | distinctAttribute    | `distrinctAttribute` payload string  |
 | displayedAttributes  | `displayedAttributes` payload array  |
 
+##### dumpCreation
+
+| name    | description  |
+| -----   | ------------ |
+| dumpUid | The generated uid of the dump |
+
+Since the creation of a dump is not a task associated with a particular index, it is only present on the `GET` - `/tasks` and `GET` - `tasks/:task_uid` endpoints.
+
+Fully qualified and summarized task objects related to a dump creation display a `null` `indexUid` field.
+
 #### 5. Examples
 
 e.g. A fully qualified `task` object in an `enqueued` state.
@@ -702,4 +713,4 @@ n/a
 - Use Hateoas capability to give direct access to a `task` resource.
 - Add dedicated task type names modifying a sub-setting. e.g. `SearchableAttributesUpdate`.
 - Add an archived state for old `tasks`.
-- Indicate the `API Key` identity that added a `task`.
+- Add the `API Key` identity that added a `task`.
diff --git a/text/0061-error-format-and-definitions.md b/text/0061-error-format-and-definitions.md
@@ -849,29 +849,6 @@ HTTP Code: `404 Not Found`
 
 ---
 
-## dump_already_processing
-
-`Synchronous`
-
-### Context
-
-This error occurs when the user tries to launch the creation of a new dump while a creation is already being processed.
-
-### Error Definition
-
-Http Code: `409 Conflict`
-
-```json
-{
-    "message": "A dump is already processing. You must wait until the current process is finished before requesting another dump.",
-    "code": "dump_already_processing",
-    "type": "invalid_request",
-    "link": "https://docs.meilisearch.com/errors#dump_already_processing"
-}
-```
-
----
-
 ## invalid_content_type
 
 `Synchronous`

diff --git a/text/0085-api-keys.md b/text/0085-api-keys.md
@@ -169,7 +169,6 @@ Only the master key allows managing the API keys.
 | settings.update | Provides access to `POST / DELETE` `/indexes/:authorizedIndexes/settings` and `/indexes/:authorizedIndexes/settings/*` routes. |
 | stats.get | Provides access to `GET` `/stats/`. **⚠️Non-authorized `indexes` are omitted from the response on `/stats`**. Also add access to `GET` `/indexes/:authorizedIndexes/stats`. |
 | dumps.create | Provides access to `POST` `/dumps` route. **As dumps are not scoped by indexes, a restriction on `indexes` does not affect this action.** |
-| dumps.get | Provides access to `GET` `/dumps/:dumpUid` route. **As dumps are not scoped by indexes, a restriction on `indexes` does not affect this action.** |
 | version | Provides access to `GET` `/version` route.
 
 ---

diff --git a/text/0105-dumps-api.md b/text/0105-dumps-api.md
@@ -1,7 +1,4 @@
-- Title: Dumps API
-- Start Date: 2022-01-12
-
-# Dumps
+# Dumps API and CLI
 
 ## 1. Summary
 
@@ -16,19 +13,18 @@ The dumps exist to upgrade a MeiliSearch instance from a previous version to a m
 ### 3.1. Summary Key Points
 
 - A dump creation can be scheduled from the MeiliSearch API using the `POST - /dumps` endpoint.
-- A dump creation status can be tracked using the `GET - /dumps/{uid}/status` endpoint.
+- A dump creation status can be tracked using the `GET - /tasks/:task_uid` endpoint.
 - MeiliSearch can only create one dump at a time.
 - By default, dumps are created in a folder named `dumps`, and can be found in the same directory as the MeiliSearch binary.
 - The `dumps` directory can be customized using the `--dumps-dir` configuration option. If the dump directory does not already exist when the dump creation process is called, MeiliSearch will create it.
-- If MeiliSearch is restarted after a dump creation, the dump's status will not appear on the `GET - /dumps/:uid/status` endpoint.
 - A `.dump` file can be imported using the `--import-dump` command-line flag.
 - The MeiliSearch server starts when the dump is fully imported and indexed.
 - By default, importing a dump when a database already exists (a non-empty data.ms folder in the same directory as the MeiliSearch binary) will stop the process and throw an error.
 - When using the command-line flag `--ignore-dump-if-db-exists=true`, MeiliSearch will use the existing database to start an instance instead of throwing an error. The dump will be ignored.
 - By default, trying to import a dump that does not exist, will stop the process and throw an error.
 - When using the command-line flag `--ignore-missing-dump`, MeiliSearch will continue its process and not throw an error.
 - When a dump is being imported, the http API is not available. Meilisearch can't receive read or write requests.
-- When a dump is being created, the task queue can receive other future operations to perform later but can't process any additional tasks during the dump creation.
+- `dumpCreation` task takes priority over enqueued `tasks`. This means that if a `dumpCreation` task is created, it will be directly processed when the current processing task finishes even if other tasks have been enqueued before.
 
 ---
 
@@ -41,45 +37,18 @@ Create a dump
 ##### 3.2.1.1. Body Payload
 N/A
 
-##### 3.2.1.2. Response
-
-`202 Accepted`
-
-```json
-{
-    "uid": "20220112-151751438",
-    "status": "in_progress",
-    "startedAt": "2022-01-12T15:17:51.438881Z"
-}
-```
-
-##### 3.2.1.3. Errors
-
-- 🔴 If Meilisearch is secured, accessing this route without the `Authorization` header returns a [missing_authorization_header](0061-error-format-and-definitions.md#missing_authorization_header) error.
-- 🔴 If Meilisearch is secured, accessing this route with a key that does not have permissions (i.e. other than the master-key) returns an [invalid_api_key](0061-error-format-and-definitions.md#invalid_api_key) error.
-- 🔴 Attempting to create a dump while a dump is already being created return an [dump_already_in_progress](0061-error-format-and-definitions.md#dump_already_in_progress) error.
+##### 3.2.1.2. Response Definition
 
-#### 3.2.2  GET `/dumps/:dump_uid/status`
+When the request is successful, Meilisearch returns the HTTP code `202 Accepted`. The response's content is the summarized representation of the received asynchronous task.
 
-Get a dump status
+See [Summarized `task` Object for `202 Accepted`](0060-tasks-api.md#summarized-task-object-for-202-accepted).
 
-##### 3.2.2.1. Response
+The uid of the generated dump can be found in the task details.
 
-`200 Success`
-
-```json
-{
-    "uid": "20220112-151751438",
-    "status": "in_progress",
-    "startedAt": "2022-01-12T15:17:51.438881Z"
-}
-```
-
-##### 3.2.2.2. Errors
+##### 3.2.1.3. Errors
 
 - 🔴 If Meilisearch is secured, accessing this route without the `Authorization` header returns a [missing_authorization_header](0061-error-format-and-definitions.md#missing_authorization_header) error.
 - 🔴 If Meilisearch is secured, accessing this route with a key that does not have permissions (i.e. other than the master-key) returns an [invalid_api_key](0061-error-format-and-definitions.md#invalid_api_key) error.
-- 🔴 Attempting to access a dump details that does not exist returns a [dump_not_found](0061-error-format-and-definitions.md#dump_not_found) error.
 
 ---
 
@@ -131,18 +100,24 @@ The following table describes which version of the dump correspond to which vers
 | `v4`         | `v0.25.0` and later                  | -                                  |
 
 What does "Highest compatibility dump version" means?
+
 For maintainance reasons, we cannot guarantee the compatibility from old dump versions to the newest ones.
 Concretely, if the user wants to upgrade from Meilisearch `v0.19.0` (dump `v1`) to `v0.26.0` (dump `v4`), migration should be done in two steps. First, import your `v0.19.0` dump into an instance running any version of Meilisearch between v0.21 and v0.24. Second, export another dump from this instance and import it to a final instance running with `v0.26.0`.
 
-## 2. Technical Aspects
+## 4. Technical Aspects
 
-### 2.1. Dump Creation
+### 4.1. Dump Creation
 
-When a dump is being created, the task queue can receive other future operations to perform later but can't process any additional tasks during the dump creation.
+When a dump is being created, the task queue can receive other future operations to perform.
 
-### 2.2. Importing a dump
+### 4.2. Importing a dump
 
 When a dump is being imported, the http API is not available. Meilisearch can't receive read or write requests.
 
-## 3. Future Possibilities
-- Make dump creation a `task`
+### 4.3. Dump Creation Task Priority
+
+Dump creation tasks have priority over other task types. If a `dumpCreation` task is enqueued, it will be directly processed when the current processing task finishes even if other tasks have been enqueued before.
+
+## 5. Future Possibilities
+
+- Give information about the import of a dump within the tasks.