Skip to content

Commit

Permalink
Merge #1752
Browse files Browse the repository at this point in the history 
1752: v0.28 dumps r=maryamsulemani97 a=dichotommy

Changes related to the dump API in v0.28. For a full description of these changes, see the [associated issue](#1703).

To do:

- [ ] Fix links (only if necessary for merge, as the links are not actually broken, just dependent on changes made in another PR)

Co-authored-by: Tommy Melvin <tommy@meilisearch.com>
Co-authored-by: Tommy <68053732+dichotommy@users.noreply.github.com>
Co-authored-by: Maryam <90181761+maryamsulemani97@users.noreply.github.com>
  • Loading branch information
@bors @dichotommy @maryamsulemani97
4 people authored Jun 29, 2022
2 parents 45b5f82 + 9713571 commit dc54d40
Show file tree
Hide file tree
Showing 7 changed files with 80 additions and 115 deletions.
8 changes: 0 additions & 8 deletions .code-samples.meilisearch.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -678,9 +678,6 @@ faceted_search_walkthrough_filter_1: |-
post_dump_1: |-
curl \
-X POST 'http://localhost:7700/dumps'
get_dump_status_1: |-
curl \
-X GET 'http://localhost:7700/dumps/20201101-110357260/status'
phrase_search_1: |-
curl \
-X POST 'http://localhost:7700/indexes/movies/search' \
Expand Down Expand Up @@ -930,11 +927,6 @@ updating_guide_create_dump: |-
curl \
-X POST 'http://localhost:7700/dumps' \
-H 'Authorization: Bearer API_KEY'
updating_guide_get_dump_status: |-
# replace {dump_uid} with the uid returned by the previous command
curl \
-X GET 'http://localhost:7700/dumps/{dump_uid}/status' \
-H 'Authorization: Bearer API_KEY'
updating_guide_get_settings_old: |-
# the -o option saves the output as a local file
curl \
Expand Down
2 changes: 0 additions & 2 deletions .vuepress/public/sample-template.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,6 @@ faceted_search_facets_distribution_1: |-
faceted_search_walkthrough_filter_1: |-
add_movies_json_1: |-
post_dump_1: |-
get_dump_status_1: |-
phrase_search_1: |-
sorting_guide_update_sortable_attributes_1: |-
sorting_guide_update_ranking_rules_1: |-
Expand Down Expand Up @@ -142,7 +141,6 @@ updating_guide_get_displayed_attributes_old: |-
updating_guide_reset_displayed_attributes_old: |-
updating_guide_reset_displayed_attributes_new: |-
updating_guide_create_dump: |-
updating_guide_get_dump_status: |-
updating_guide_get_settings_old: |-
updating_guide_retrieve_documents_old: |-
updating_guide_update_settings_old: |-
Expand Down
46 changes: 21 additions & 25 deletions learn/advanced/asynchronous_operations.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Currently, these are Meilisearch's asynchronous operations:
- Adding documents to an index
- Updating documents in an index
- Deleting documents from an index
- [Creating a dump](#dumps)
- Creating a dump

## Understanding tasks

Expand All @@ -27,32 +27,32 @@ All of Meilisearch's asynchronous operations belong to a category called "tasks"

The response from the [task API](/reference/api/tasks.md) will always include the following fields in the stated order:

| Field | Type | Description |
|--------------|---------|----------------------------------------------------------------------------------------------|
| `uid` | integer | The unique sequential identifier of the task |
| `indexUid` | string | The unique index identifier |
| `status` | string | The status of the task. Possible values are `enqueued`, `processing`, `succeeded`, `failed` |
| `type` | string | The type of task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate` |
| `details` | object | Detailed information on the task payload |
| `error` | object | Error details and context. Only present when a task has the `failed` status |
| `duration` | string | The total elapsed time the task spent in the `processing` state, in ISO 8601 format |
| `enqueuedAt` | string | The date and time when the task was first `enqueued`, in RFC 3339 format |
| `startedAt` | string | The date and time when the task began `processing`, in RFC 3339 format |
| `finishedAt` | string | The date and time when the task finished processing, whether `failed` or `succeeded`, in RFC 3339 format. |
| Field | Type | Description |
|--------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------|
| `uid` | integer | The unique sequential identifier of the task |
| `indexUid` | string | The unique index identifier (always `null` for dumps) |
| `status` | string | The status of the task. Possible values are `enqueued`, `processing`, `succeeded`, `failed` |
| `type` | string | The type of task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentPartial`, `documentDeletion`, `settingsUpdate`, `clearAll`, `dumpCreation` |
| `details` | object | Detailed information on the task payload |
| `error` | object | Error details and context. Only present when a task has the `failed` status |
| `duration` | string | The total elapsed time the task spent in the `processing` state, in ISO 8601 format |
| `enqueuedAt` | string | The date and time when the task was first `enqueued`, in RFC 3339 format |
| `startedAt` | string | The date and time when the task began `processing`, in RFC 3339 format |
| `finishedAt` | string | The date and time when the task finished processing, whether `failed` or `succeeded`, in RFC 3339 format. |

If a task fails due to an error, all error fields will be appended to the task response in an `error` object.

### Summarized task objects

All asynchronous operations return a summarized version of [the full `task` object](#task-api-response). It contains the following fields in the stated order:

| Field | Type | Description |
|--------------|---------|---------------------------------------------------------------------------------------|
| `taskUid` | integer | Unique sequential identifier |
| `indexUid` | string | Unique index identifier |
| `status` | string | Status of the task. Value is `enqueued` |
| `type` | string | Type of task |
| `enqueuedAt` | string | Represents the date and time in the RFC 3339 format when the task has been `enqueued` |
| Field | Type | Description |
|------------|---------|--------------------------------- |
| `uid` | integer | Unique sequential identifier |
| `indexUid` | string | Unique index identifier (always `null` for dumps) |
| `status` | string | Status of the task. Value is `enqueued` |
| `type` | string | Type of task |
| `enqueuedAt` | string | Represents the date and time in the RFC 3339 format when the task has been `enqueued` |

You can use this `taskUid` to get more details on [the status of the task](/reference/api/tasks.md#get-one-task).

Expand Down Expand Up @@ -139,11 +139,7 @@ Had the task failed, the response would have included an `error` object:
3. Once the task has completed processing, Meilisearch marks it as `succeeded`, if it was successful, or `failed`, if there was an error.
4. Tasks marked as `succeeded` or `failed` are not deleted and will remain visible in [the task list](/reference/api/tasks.md#get-tasks)

### Dumps

While dumps and tasks are both asynchronous operations, they use separate queues and behave differently. For instance, creating a new dump will freeze the task queue until the dump has been generated.

[You can read more about dumps in our dedicated guide.](/learn/advanced/dumps.md)
Tasks are processed in the order they were enqueued, with one exception: `dumpCreation`. Dumps are prioritized over all other tasks in the queue. Their task `uid` still reflects when they were enqueued relative to other tasks.

## Terminate Meilisearch while a task is being processed

Expand Down
61 changes: 29 additions & 32 deletions learn/advanced/dumps.md
View file
Original file line number Diff line number Diff line change
@@ -1,64 +1,61 @@
# Dumps

A dump is a compressed file containing an export of your Meilisearch instance. It contains all your indexes, documents, and settings, but in a raw unprocessed form. A dump isn't an exact copy of your database—it is closer to a blueprint that allows you to create an identical dataset. A dump can be imported when launching Meilisearch, but be advised that it may take some time to index all the documents within.
A dump is a compressed file containing an export of your Meilisearch instance. It contains all your indexes, documents, and settings, but in a raw unprocessed form. A dump isn't an exact copy of your database—it is closer to a blueprint that allows you to create an identical dataset.

A dump can be imported when launching Meilisearch, but be advised that it may take some time to index.

## Creating a dump

To create a dump of your dataset, you need to use the appropriate HTTP route: [`POST /dumps`](/reference/api/dump.md#create-a-dump). The dump creation process is an asynchronous task that takes time proportional to the size of your dataset.

<CodeSamples id="post_dump_1" />

The above code triggers a dump creation process. It also returns an object containing information about the dump:
The above code triggers a dump creation process. It also returns a [summarized task object](/learn/advanced/asynchronous_operations.md#summarized-task-objects) that you can use to check the status of your dump.

```
```json
{
"uid": "20200929-114144097",
"status": "in_progress",
"startedAt": "2020-09-29T11:41:44.392327Z"
"taskUid": 1,
"indexUid": null,
"status": "enqueued",
"type": "dumpCreation",
"enqueuedAt": "2022-06-21T16:10:29.217688Z"
}
```

You can use the returned `uid` (unique identifier indicating when the dump was triggered) to track its progress with the [get dump status route](/reference/api/dump.md#get-dump-status). The returned status could be:

- `in_progress`: Dump creation is in progress
- `failed`: An error occurred during dump process, and the task was aborted
- `done`: Dump creation is finished and was successful
In the below command, replace `1` with the `taskUid` returned by the previous command.

<CodeSamples id="get_dump_status_1" />
<CodeSamples id="get_task_1" />

The above code sample returns an object with the following details about the dump:
This command should return an object with detailed information about the dump operation:

```
```json
{
"uid": "20200929-114144097",
"status": "done",
"startedAt": "2020-09-29T11:41:44.392327Z",
"finishedAt": "2020-09-29T11:41:50.792147Z"
"uid": 1,
"indexUid": null,
"status": "succeeded",
"type": "dumpCreation",
"details": {
"dumpUid": "20220621-161029217"
},
"duration": "PT0.025872S",
"enqueuedAt": "2022-06-21T16:10:29.217688Z",
"startedAt": "2022-06-21T16:10:29.218297Z",
"finishedAt": "2022-06-21T16:10:29.244169Z"
}
```

After dump creation is finished, the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dumps-dir` configuration option](/learn/configuration/instance_options.md#dumps-destination). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**
After dump creation is finished—when `status` is `succeeded`—the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dumps-dir` configuration option](/learn/configuration/instance_options.md#dumps-destination). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**

If a dump file is visible in the file system, the dump process was successfully completed. **Meilisearch will never create a partial dump file**, even if you interrupt an instance while it is generating a dump.

::: note
Unlike [tasks](/learn/advanced/asynchronous_operations.md), dumps have no queue. **Meilisearch only processes one dump at a time.** If you attempt to create a dump while another dump is still processing, Meilisearch will throw an [error](/reference/api/error_codes.md#dump-already-processing).

Meilisearch does not process tasks during dump creation, but you can still add new requests to the task queue. This is also true for [snapshots](/learn/advanced/snapshots.md#snapshots).
:::

::: warning
If you restart Meilisearch after creating a dump, you will not be able to use the dumps endpoint to find out that dump's `status`. This has no effect on the dump file itself.
:::

## Importing a dump

When a dump is being imported, the API is not available to the task queue. As a result, no read or write operations can be performed until the importing process is complete.

Dumps in v0.20.0 and below are no longer compatible with the new versions. Before you start importing, check your [Meilisearch version](/reference/api/version.md#example) and proceed accordingly.
Dumps from v0.20.0 and below are no longer compatible with the new versions. Before you start importing, check your [Meilisearch version](/reference/api/version.md#example) and proceed accordingly.

::: note
We do not recommend using dumps from a new Meilisearch version to import an older version.
We do not recommend using dumps to migrate from a new Meilisearch version to an older one.

For example, you can import a dump from Meilisearch v0.21 into v0.22 without any problems. Importing a dump generated in v0.22 into a v0.21 instance, however, can lead to unexpected behavior.
:::
Expand All @@ -79,4 +76,4 @@ If you are using Meilisearch v0.20 or below, migration should be done in two ste

## Use cases

Dumps are used to restore your database after [updating Meilisearch](/learn/advanced/updating.md) or to copy your database to other Meilisearch instances without having to worry about their respective versions.
Dumps are used to restore your database after [updating Meilisearch](/learn/advanced/updating.md) or to copy your database to other Meilisearch instances without having to worry about their respective versions. For more on this subject, see a [comparison of snapshots and dumps](/learn/advanced/snapshots_vs_dumps.md).
29 changes: 18 additions & 11 deletions learn/advanced/updating.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -206,24 +206,31 @@ The server should return a response that looks like this:

```json
{
"uid": "20210212-151153878",
"status": "in_progress",
"startedAt": "2021-02-12T15:11:53.402327Z"
"taskUid": 1,
"indexUid": null,
"status": "enqueued",
"type": "dumpCreation",
"enqueuedAt": "2022-06-21T16:10:29.217688Z"
}
```

This process can take some time. Since dump creation is an [asynchronous operation](/learn/advanced/asynchronous_operations.md), you can use the returned `uid` to [track its status](/reference/api/dump.md#get-dump-status).
This command returns a `taskUid`. You can use this to [track the status](/reference/api/tasks.md#get-one-task) of your dump. Keep in mind that the process can take some time to complete.

<CodeSamples id="updating_guide_get_dump_status" />

Once the response to the previous command looks like this (`"status": "done"`), move on.
Once the `dumpCreation` task shows `"status": "succeeded"`, you're ready to move on.

```json
{
"uid": "20200929-114144097",
"status": "done",
"startedAt": "2020-09-29T11:41:44.392327Z",
"finishedAt": "2020-09-29T11:41:50.792147Z"
"uid": 1,
"indexUid": null,
"status": "succeeded",
"type": "dumpCreation",
"details": {
"dumpUid": "20220621-161029217"
},
"duration": "PT0.025872S",
"enqueuedAt": "2022-06-21T16:10:29.217688Z",
"startedAt": "2022-06-21T16:10:29.218297Z",
"finishedAt": "2022-06-21T16:10:29.244169Z"
}
```

Expand Down
45 changes: 12 additions & 33 deletions reference/api/dump.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,21 @@ Creating a dump is also referred to as exporting it, whereas launching Meilisear

During a [dump export](/reference/api/dump.md#create-a-dump), all indexes of the current instance are exported—together with their documents and settings—and saved as a single `.dump` file.

During a dump import, all indexes contained in the indicated `.dump` file are imported along with their associated documents and settings. Any existing index with the same uid as an index in the dump file will be overwritten.
During a dump import, all indexes contained in the indicated `.dump` file are imported along with their associated documents and settings. Any existing index with the same `uid` as an index in the dump file will be overwritten.

Dump imports must be performed when launching a Meilisearch instance [using the `import-dump` command-line option](/learn/configuration/instance_options.md#import-dump).

[Learn more about dumps](/learn/advanced/dumps.md).

## Create a dump

<RouteHighlighter method="POST" route="/dumps"/>

Triggers a dump creation process. Once the process is complete, a dump is created in the [dumps directory](/learn/configuration/instance_options.md#dumps-destination). If the dumps directory does not exist yet, it will be created.
Triggers a dump creation task. Once the process is complete, a dump is created in the [dumps directory](/learn/configuration/instance_options.md#dumps-destination). If the dumps directory does not exist yet, it will be created.

**Meilisearch only processes one dump at a time.** If you attempt to create a dump while another dump is still processing, Meilisearch will throw a [`dump_already_processing` error](/reference/api/error_codes.md#dump-already-processing).
Dump tasks take priority over all other tasks in the queue. This means that a newly created dump task will be processed as soon as the current task is finished.

The task queue will not process any further tasks during dump creation, but you can still add new requests to the queue.
[Learn more about asynchronous operations](/learn/advanced/asynchronous_operations.md).

### Example

Expand All @@ -28,35 +30,12 @@ The task queue will not process any further tasks during dump creation, but you

```json
{
"uid": "20200929-114144097",
"status": "in_progress",
"startedAt": "2020-09-29T11:41:44.392327Z"
"taskUid": 1,
"indexUid": null,
"status": "enqueued",
"type": "dumpCreation",
"enqueuedAt": "2022-06-21T16:10:29.217688Z"
}
```

## Get dump status

<RouteHighlighter method="GET" route="/dumps/{dump_uid}/status"/>

Get the status of a dump creation process using the uid returned after calling the [dump creation route](/reference/api/dump.md#create-a-dump). The dump `uid` is required.

The returned status could be:

- `in_progress`: Dump creation is in progress
- `failed`: An error occurred during dump process, and the task was aborted
- `done`: Dump creation is finished and was successful

### Example

<CodeSamples id="get_dump_status_1" />

#### Response: `200 Ok`

```json
{
"uid": "20200929-114144097",
"status": "done",
"startedAt": "2020-09-29T11:41:44.392327Z",
"finishedAt": "2020-09-29T11:41:50.792147Z"
}
```
You can use this `taskUid` to get more details on [the status of the task](/reference/api/tasks.md#get-one-task)
4 changes: 0 additions & 4 deletions reference/api/error_codes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,6 @@ The Content-Type header does not match the request body payload format or the fo

The Content-Type header was specified, but no request body was sent to the server or the request body is empty.

### `dump_already_processing`

Dump creation is already in progress. A new dump creation process can be triggered after the current one has been completed.

### `dump_not_found`

The requested dump could not be found.
Expand Down

0 comments on commit dc54d40

Please sign in to comment.