Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.28 dumps #1752

Merged
merged 13 commits into from
Jun 29, 2022
8 changes: 0 additions & 8 deletions .code-samples.meilisearch.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -672,9 +672,6 @@ faceted_search_walkthrough_filter_1: |-
post_dump_1: |-
curl \
-X POST 'http://localhost:7700/dumps'
get_dump_status_1: |-
curl \
-X GET 'http://localhost:7700/dumps/20201101-110357260/status'
phrase_search_1: |-
curl \
-X POST 'http://localhost:7700/indexes/movies/search' \
Expand Down Expand Up @@ -924,11 +921,6 @@ updating_guide_create_dump: |-
curl \
-X POST 'http://localhost:7700/dumps' \
-H 'Authorization: Bearer API_KEY'
updating_guide_get_dump_status: |-
# replace {dump_uid} with the uid returned by the previous command
curl \
-X GET 'http://localhost:7700/dumps/{dump_uid}/status' \
-H 'Authorization: Bearer API_KEY'
updating_guide_get_settings_old: |-
# the -o option saves the output as a local file
curl \
Expand Down
2 changes: 0 additions & 2 deletions .vuepress/public/sample-template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,6 @@ faceted_search_facets_distribution_1: |-
faceted_search_walkthrough_filter_1: |-
add_movies_json_1: |-
post_dump_1: |-
get_dump_status_1: |-
phrase_search_1: |-
sorting_guide_update_sortable_attributes_1: |-
sorting_guide_update_ranking_rules_1: |-
Expand Down Expand Up @@ -140,7 +139,6 @@ updating_guide_get_displayed_attributes_old: |-
updating_guide_reset_displayed_attributes_old: |-
updating_guide_reset_displayed_attributes_new: |-
updating_guide_create_dump: |-
updating_guide_get_dump_status: |-
updating_guide_get_settings_old: |-
updating_guide_retrieve_documents_old: |-
updating_guide_update_settings_old: |-
Expand Down
14 changes: 5 additions & 9 deletions learn/advanced/asynchronous_operations.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ Currently, these are Meilisearch's asynchronous operations:
- Adding documents to an index
- Updating documents in an index
- Deleting documents from an index
- [Creating a dump](#dumps)
- Creating a dump

## Understanding tasks

Expand All @@ -33,9 +33,9 @@ The response from the [task API](/reference/api/tasks.md) will always include th
| Field | Type | Description |
|--------------|---------|--------------------------------------------------------------------------------------------------------------------------------------------|
| `uid` | integer | The unique sequential identifier of the task |
| `indexUid` | string | The unique index identifier |
| `indexUid` | string | The unique index identifier (always `null` for dumps) |
| `status` | string | The status of the task. Possible values are `enqueued`, `processing`, `succeeded`, `failed` |
| `type` | string | The type of task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAddition`, `documentPartial`, `documentDeletion`, `settingsUpdate`, `clearAll` |
| `type` | string | The type of task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAddition`, `documentPartial`, `documentDeletion`, `settingsUpdate`, `clearAll`, `dumpCreation` |
| `details` | object | Detailed information on the task payload |
| `error` | object | Error details and context. Only present when a task has the `failed` status |
| `duration` | string | The total elapsed time the task spent in the `processing` state, in ISO 8601 format |
Expand All @@ -52,7 +52,7 @@ All asynchronous operations return a summarized version of the [`task` object](#
| Field | Type | Description |
|------------|---------|--------------------------------- |
| `uid` | integer | Unique sequential identifier |
| `indexUid` | string | Unique index identifier |
| `indexUid` | string | Unique index identifier (always `null` for dumps) |
| `status` | string | Status of the task. Value is `enqueued` |
| `type` | string | Type of task |
| `enqueuedAt` | string | Represents the date and time in the RFC 3339 format when the task has been `enqueued` |
Expand Down Expand Up @@ -142,11 +142,7 @@ Had the task failed, the response would have included an `error` object:
3. Once the task has completed processing, Meilisearch marks it as `succeeded`, if it was successful, or `failed`, if there was an error.
4. Tasks marked as `succeeded` or `failed` are not deleted and will remain visible in [the task list](/reference/api/tasks.md#get-all-tasks)

### Dumps

While dumps and tasks are both asynchronous operations, they use separate queues and behave differently. For instance, creating a new dump will freeze the task queue until the dump has been generated.

[You can read more about dumps in our dedicated guide.](/learn/advanced/dumps.md)
Tasks are processed in the order they were enqueued, with one exception: `dumpCreation`. Dumps are prioritized over all other tasks in the queue. Their task `uid` still reflects when they were enqueued relative to other tasks.

## Terminate Meilisearch while a task is being processed

Expand Down
61 changes: 29 additions & 32 deletions learn/advanced/dumps.md
Original file line number Diff line number Diff line change
@@ -1,64 +1,61 @@
# Dumps

A dump is a compressed file containing an export of your Meilisearch instance. It contains all your indexes, documents, and settings, but in a raw unprocessed form. A dump isn't an exact copy of your database—it is closer to a blueprint that allows you to create an identical dataset. A dump can be imported when launching Meilisearch, but be advised that it may take some time to index all the documents within.
A dump is a compressed file containing an export of your Meilisearch instance. It contains all your indexes, documents, and settings, but in a raw unprocessed form. A dump isn't an exact copy of your database—it is closer to a blueprint that allows you to create an identical dataset.

A dump can be imported when launching Meilisearch, but be advised that it may take some time to index.

## Creating a dump

To create a dump of your dataset, you need to use the appropriate HTTP route: [`POST /dumps`](/reference/api/dump.md#create-a-dump). The dump creation process is an asynchronous task that takes time proportional to the size of your dataset.

<CodeSamples id="post_dump_1" />

The above code triggers a dump creation process. It also returns an object containing information about the dump:
The above code triggers a dump creation process. It also returns a [summarized task object](/learn/advanced/asynchronous_operations.md#summarized-task-objects) that you can use to check the status of your dump.

```
```json
{
"uid": "20200929-114144097",
"status": "in_progress",
"startedAt": "2020-09-29T11:41:44.392327Z"
"taskUid": 1,
"indexUid": null,
"status": "enqueued",
"type": "dumpCreation",
"enqueuedAt": "2022-06-21T16:10:29.217688Z"
}
```

You can use the returned `uid` (unique identifier indicating when the dump was triggered) to track its progress with the [get dump status route](/reference/api/dump.md#get-dump-status). The returned status could be:

- `in_progress`: Dump creation is in progress
- `failed`: An error occurred during dump process, and the task was aborted
- `done`: Dump creation is finished and was successful
In the below command, replace `1` with the `taskUid` returned by the previous command.

<CodeSamples id="get_dump_status_1" />
<CodeSamples id="get_task_1" />

The above code sample returns an object with the following details about the dump:
This command should return an object with detailed information about the dump operation:

```
```json
{
"uid": "20200929-114144097",
"status": "done",
"startedAt": "2020-09-29T11:41:44.392327Z",
"finishedAt": "2020-09-29T11:41:50.792147Z"
"uid": 1,
"indexUid": null,
"status": "succeeded",
"type": "dumpCreation",
"details": {
"dumpUid": "20220621-161029217"
},
"duration": "PT0.025872S",
"enqueuedAt": "2022-06-21T16:10:29.217688Z",
"startedAt": "2022-06-21T16:10:29.218297Z",
"finishedAt": "2022-06-21T16:10:29.244169Z"
}
```

After dump creation is finished, the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dumps-dir` configuration option](/learn/configuration/instance_options.md#dumps-destination). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**
After dump creation is finished—when `status` is `succeeded`—the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dumps-dir` configuration option](/learn/configuration/instance_options.md#dumps-destination). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**

If a dump file is visible in the file system, the dump process was successfully completed. **Meilisearch will never create a partial dump file**, even if you interrupt an instance while it is generating a dump.

::: note
Unlike [tasks](/learn/advanced/asynchronous_operations.md), dumps have no queue. **Meilisearch only processes one dump at a time.** If you attempt to create a dump while another dump is still processing, Meilisearch will throw an [error](/reference/api/error_codes.md#dump-already-processing).

Meilisearch does not process tasks during dump creation, but you can still add new requests to the task queue. This is also true for [snapshots](/learn/advanced/snapshots.md#snapshots).
:::

::: warning
If you restart Meilisearch after creating a dump, you will not be able to use the dumps endpoint to find out that dump's `status`. This has no effect on the dump file itself.
:::

## Importing a dump

When a dump is being imported, the API is not available to the task queue. As a result, no read or write operations can be performed until the importing process is complete.

Dumps in v0.20.0 and below are no longer compatible with the new versions. Before you start importing, check your [Meilisearch version](/reference/api/version.md#example) and proceed accordingly.
Dumps from v0.20.0 and below are no longer compatible with the new versions. Before you start importing, check your [Meilisearch version](/reference/api/version.md#example) and proceed accordingly.

::: note
We do not recommend using dumps from a new Meilisearch version to import an older version.
We do not recommend using dumps to migrate from a new Meilisearch version to an older one.

For example, you can import a dump from Meilisearch v0.21 into v0.22 without any problems. Importing a dump generated in v0.22 into a v0.21 instance, however, can lead to unexpected behavior.
:::
Expand All @@ -79,4 +76,4 @@ If you are using Meilisearch v0.20 or below, migration should be done in two ste

## Use cases

Dumps are used to restore your database after [updating Meilisearch](/learn/advanced/updating.md) or to copy your database to other Meilisearch instances without having to worry about their respective versions.
Dumps are used to restore your database after [updating Meilisearch](/learn/advanced/updating.md) or to copy your database to other Meilisearch instances without having to worry about their respective versions. For more on this subject, see a [comparison of snapshots and dumps](/learn/advanced/snapshots_vs_dumps.md).
29 changes: 18 additions & 11 deletions learn/advanced/updating.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,24 +206,31 @@ The server should return a response that looks like this:

```json
{
"uid": "20210212-151153878",
"status": "in_progress",
"startedAt": "2021-02-12T15:11:53.402327Z"
"taskUid": 1,
"indexUid": null,
"status": "enqueued",
"type": "dumpCreation",
"enqueuedAt": "2022-06-21T16:10:29.217688Z"
}
```

This process can take some time. Since dump creation is an [asynchronous operation](/learn/advanced/asynchronous_operations.md), you can use the returned `uid` to [track its status](/reference/api/dump.md#get-dump-status).
This command returns a `taskUid`. You can use this to [track the status](/reference/api/tasks.md#get-one-task) of your dump. Keep in mind that the process can take some time to complete.

<CodeSamples id="updating_guide_get_dump_status" />

Once the response to the previous command looks like this (`"status": "done"`), move on.
Once the `dumpCreation` task shows `"status": "succeeded"`, you're ready to move on.

```json
{
"uid": "20200929-114144097",
"status": "done",
"startedAt": "2020-09-29T11:41:44.392327Z",
"finishedAt": "2020-09-29T11:41:50.792147Z"
"uid": 1,
"indexUid": null,
"status": "succeeded",
"type": "dumpCreation",
"details": {
"dumpUid": "20220621-161029217"
},
"duration": "PT0.025872S",
"enqueuedAt": "2022-06-21T16:10:29.217688Z",
"startedAt": "2022-06-21T16:10:29.218297Z",
"finishedAt": "2022-06-21T16:10:29.244169Z"
}
```

Expand Down
45 changes: 12 additions & 33 deletions reference/api/dump.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,21 @@ Creating a dump is also referred to as exporting it, whereas launching Meilisear

During a [dump export](/reference/api/dump.md#create-a-dump), all indexes of the current instance are exported—together with their documents and settings—and saved as a single `.dump` file.

During a dump import, all indexes contained in the indicated `.dump` file are imported along with their associated documents and settings. Any existing index with the same uid as an index in the dump file will be overwritten.
During a dump import, all indexes contained in the indicated `.dump` file are imported along with their associated documents and settings. Any existing index with the same `uid` as an index in the dump file will be overwritten.

Dump imports must be performed when launching a Meilisearch instance [using the `import-dump` command-line option](/learn/configuration/instance_options.md#import-dump).

[Learn more about dumps](/learn/advanced/dumps.md).

## Create a dump

<RouteHighlighter method="POST" route="/dumps"/>

Triggers a dump creation process. Once the process is complete, a dump is created in the [dumps directory](/learn/configuration/instance_options.md#dumps-destination). If the dumps directory does not exist yet, it will be created.
Triggers a dump creation task. Once the process is complete, a dump is created in the [dumps directory](/learn/configuration/instance_options.md#dumps-destination). If the dumps directory does not exist yet, it will be created.

**Meilisearch only processes one dump at a time.** If you attempt to create a dump while another dump is still processing, Meilisearch will throw a [`dump_already_processing` error](/reference/api/error_codes.md#dump-already-processing).
Dump tasks take priority over all other tasks in the queue. This means that a newly created dump task will be processed as soon as the current task is finished.

The task queue will not process any further tasks during dump creation, but you can still add new requests to the queue.
[Learn more about asynchronous operations](/learn/advanced/asynchronous_operations.md).

### Example

Expand All @@ -28,35 +30,12 @@ The task queue will not process any further tasks during dump creation, but you

```json
{
"uid": "20200929-114144097",
"status": "in_progress",
"startedAt": "2020-09-29T11:41:44.392327Z"
"taskUid": 1,
"indexUid": null,
dichotommy marked this conversation as resolved.
Show resolved Hide resolved
"status": "enqueued",
"type": "dumpCreation",
"enqueuedAt": "2022-06-21T16:10:29.217688Z"
}
```

## Get dump status

<RouteHighlighter method="GET" route="/dumps/{dump_uid}/status"/>

Get the status of a dump creation process using the uid returned after calling the [dump creation route](/reference/api/dump.md#create-a-dump). The dump `uid` is required.

The returned status could be:

- `in_progress`: Dump creation is in progress
- `failed`: An error occurred during dump process, and the task was aborted
- `done`: Dump creation is finished and was successful

### Example

<CodeSamples id="get_dump_status_1" />

#### Response: `200 Ok`

```json
{
"uid": "20200929-114144097",
"status": "done",
"startedAt": "2020-09-29T11:41:44.392327Z",
"finishedAt": "2020-09-29T11:41:50.792147Z"
}
```
You can use this `taskUid` to get more details on [the status of the task](/reference/api/tasks.md#get-one-task)
4 changes: 0 additions & 4 deletions reference/api/error_codes.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,6 @@ The Content-Type header does not match the request body payload format or the fo

The Content-Type header was specified, but no request body was sent to the server or the request body is empty.

### `dump_already_processing`

Dump creation is already in progress. A new dump creation process can be triggered after the current one has been completed.

### `dump_not_found`

The requested dump could not be found.
Expand Down