Skip to content
This repository has been archived by the owner on Mar 21, 2024. It is now read-only.

Swap Indexes API #192

Merged
merged 17 commits into from
Nov 29, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 56 additions & 1 deletion open-api.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -690,6 +690,20 @@ components:
- indexes
- expiresAt
description: API keys are stored and managed by the master key holder and the default admin key holder. These are the keys used by the technical teams to interact with Meilisearch at the level of the client/server code. API keys can have restrictions on which methods can be accessed via an actions list; they can also expiresAt a specific date time and be restricted to a specific set of indexes.
swapOperation:
type: object
properties:
indexes:
type: array
description: The two indexUids to swap in the given operation.
required:
- indexes
swapIndexes:
type: array
description: Array of swap operations that will be processed atomically.
required: true
items:
$ref: '#/components/schemas/swapOperation'
parameters:
indexUid:
name: indexUid
Expand Down Expand Up @@ -1009,10 +1023,18 @@ components:
summary: 202 Accepted dumpCreation response example
value:
taskUid: 0
indexUid: movies
indexUid: null
status: enqueued
type: dumpCreation
enqueuedAt: '2021-01-01T09:39:00.000000Z'
202_indexSwap:
summary: 202 Accepted indexSwap response example
value:
taskUid: 0
indexUid: null
status: enqueued
type: indexSwap
enqueuedAt: '2021-01-01T09:39:00.000000Z'
tags:
- name: Indexes
description: |
Expand Down Expand Up @@ -3410,4 +3432,37 @@ paths:
- apiKey: []
parameters:
- $ref: '#/components/parameters/taskUid'
'/swap-indexes':
post:
operationId: indexes.swap
summary: Swap Indexes
description: Deploy a new version of an index without any downtime for clients by swapping documents, settings, and task history between two indexes. Specifying several swap operations that will be processed in an atomic way is possible.
tags:
- Indexes
security:
- apiKey: []
parameters:
- $ref: '#/components/parameters/Content-Type'
request:
requestBody:
content:
application/json:
schema:
$ref: '#/components/schemas/swapIndexes'
examples:
Example:
- indexes: ["indexA", "indexA_new"]
- indexes: ["indexB", "indexB_new"]
responses:
'202':
description: Accepted
content:
application/json:
schema:
$ref: '#/components/responses/202'
examples:
'202':
$ref: '#/components/examples/202_indexSwap'
'401':
$ref: '#/components/responses/401'
security: []
11 changes: 11 additions & 0 deletions text/0034-telemetry-policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
| Documents Deleted | Aggregated event on all received requests via the `DELETE` - `/indexes/:indexUid/documents`, `DELETE` - `/indexes/:indexUid/documents/:documentId`, `POST` - `indexes/:indexUid/documents/delete-batch` routes during one hour or until a batch size reaches `500Kb`. |
| Index Created | Occurs when an index is created via `POST` - `/indexes`. |
| Index Updated | Occurs when an index is updated via `PUT` - `/indexes/:indexUid`. |
| Indexes Swapped | Occurs when indexes are swapped via `POST` - `/swap-indexes`. |
| Settings Updated | Occurs when the settings are updated via `POST` - `/indexes/:indexUid/settings`. |
| SearchableAttributes Updated | Occurs when searchable attributes are updated via `POST` - `/indexes/:indexUid/settings/searchable-attributes`. |
| RankingRules Updated | Occurs when ranking rules are updated via `POST` - `/indexes/:indexUid/settings/ranking-rules`. |
Expand Down Expand Up @@ -160,6 +161,7 @@ The collected data is sent to [Segment](https://segment.com/). Segment is a plat
| `filtered_by_type` | `true` if `GET /tasks` endpoint is filtered by `type`, otherwise `false` | false | `Tasks Seen` |
| `filtered_by_status` | `true` if `GET /tasks` endpoint is filtered by `status`, otherwise `false` | false | `Tasks Seen` |
| `per_index_uid` | `true` if an uid is used to fetch an index stat resource, otherwise `false` | false | `Stats Seen` |
| `swap_operation_number` | The number of swap operation given in `POST /swap-indexes` API call | 2 | `Indexes Swapped` |
| `matching_strategy.most_used_strategy` | Most used word matching strategy among all search requests in this batch | `last` | `Documents Searched POST`, `Documents Searched GET` |
| `per_document_id` | `true` if `DELETE /indexes/:indexUid/documents/:documentUid` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
| `clear_all` | `true` if `DELETE /indexes/:indexUid/documents` endpoint was used in this batch, otherwise `false` | false | `Documents Deleted` |
Expand Down Expand Up @@ -313,6 +315,15 @@ This property allows us to gather essential information to better understand on

---

## `Indexes Swapped`

| Property name | Description | Example |
|---------------|-------------|---------|
| user_agent | Represents the user-agent encountered for this API call. | `["Meilisearch Ruby (v2.1)", "Ruby (3.0)"]` |
| swap_operation_number | The number of swap operation given for this API call. | `2` |

---

## `Documents Added`

> The Documents Added event is sent once an hour or when a batch reaches the maximum size of `500kb`. The event's properties are averaged over all requests on `POST` - `/indexes/:indexUid/documents`.
Expand Down
11 changes: 9 additions & 2 deletions text/0060-tasks-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
| uid | integer | Unique sequential identifier |
| indexUid | string | Unique index identifier. This field is `null` when the task type is `dumpCreation`. |
| status | string | Status of the task. Possible values are `enqueued`, `processing`, `succeeded`, `failed` |
| type | string | Type of the task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate`, `dumpCreation` |
| type | string | Type of the task. Possible values are `indexCreation`, `indexUpdate`, `indexDeletion`, `indexSwap`, `documentAdditionOrUpdate`, `documentDeletion`, `settingsUpdate`, `dumpCreation` |
| details | object | Details information for a task payload. See Task Details part. |
| error | object | Error object containing error details and context when a task has a `failed` status. See [0061-error-format-and-definitions.md](0061-error-format-and-definitions.md) |
| duration | string | Total elapsed time the engine was in processing state expressed as an `ISO-8601` duration format. Times below the second can be expressed with the `.` notation, e.g., `PT0.5S` to express `500ms`. Default is set to `null`. |
Expand Down Expand Up @@ -64,10 +64,11 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
| indexCreation |
| indexUpdate |
| indexDeletion |
| indexSwap |
| documentAdditionOrUpdate |
| documentDeletion |
| settingsUpdate |
| dumpCreation |
| dumpCreation |

> 👍 Type values follow a `camelCase` naming convention.

Expand Down Expand Up @@ -106,6 +107,12 @@ As writing is asynchronous for most of Meilisearch's operations, this API makes
|------------------|--------------------------------------------------------------------------------------|
| deletedDocuments | Number of deleted documents. Should be all documents contained in the deleted index. |

##### indexSwap

| name | description |
|------------------|--------------------------------------------------------------------------------------|
| swaps | Object containing the payload originating the `indexSwap` task creation |

##### settingsUpdate

| name | description |
Expand Down
58 changes: 56 additions & 2 deletions text/0061-error-format-and-definitions.md
Original file line number Diff line number Diff line change
Expand Up @@ -827,15 +827,30 @@ This error code is generic. It should not be used. Instead, a clear and precise

## index_not_found

`Synchronous`
`Asynchronous` / `Synchronous`
gmourier marked this conversation as resolved.
Show resolved Hide resolved

### Context

This error happens when a requested index can't be found.

### Error Definition

HTTP Code: `404 Not Found`
HTTP Code: `404 Not Found` when

#### Variant: Multiples indexUids can't be found

```json
{
"message": "Indexes `:indexUids` not found.",
"code": "index_not_found",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#index_not_found"
}
```

- The `:indexUids` is inferred when the message is generated, values are separated by `,`.

#### Variant: An index can't be found

```json
{
Expand All @@ -850,6 +865,45 @@ HTTP Code: `404 Not Found`

---

## duplicate_index_found

`Synchronous`

### Context

This error happens when the same indexUid is used twice in the `POST`- `swap-indexes` payload.

### Error Definition


#### Variant: A single indexUid is found twice in the payload

```json
{
"message": "Indexes must be declared only once during a swap. `:indexUid` was specified several times.",
"code": "duplicate_index_found",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#duplicate_index_found"
}
```

- The `:indexUid` is inferred when the message is generated.

#### Variant: Several indexUids are found twice in the payload

```json
{
"message": "Indexes must be declared only once during a swap. `:indexUids` were specified several times.",
"code": "duplicate_index_found",
"type": "invalid_request",
"link": "https://docs.meilisearch.com/errors#duplicate_index_found"
}
```

- The `:indexUids` is inferred when the message is generated, values are separated by `,`.

---

## document_not_found

`Synchronous`
Expand Down
1 change: 1 addition & 0 deletions text/0085-api-keys.md
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,7 @@ Create an API key.
| indexes.get | Provides access to `GET` `/indexes` and `/indexes/:authorizedIndexes`. **⚠️Non-authorized `indexes` are omitted from the response on `/indexes`**. |
| indexes.update | Provides access to `PUT` `/indexes/:authorizedIndexes`. |
| indexes.delete | Provides access to `DELETE` `/indexes/:authorizedIndexes`. |
| indexes.swap | Provides access to `POST` `/swap-indexes`. See [Swap Indexes API](0191-swap-indexes-api.md) specification. |
| tasks.get | Provides access to `GET` `/tasks`. **⚠️Non-authorized `indexes` are omitted from the response on `/tasks`**. Also add access to `GET` `/indexes/:authorizedIndexes/tasks` routes. |
| settings.get | Provides access to `GET` `/indexes/:authorizedIndexes/settings` and `/indexes/:authorizedIndexes/settings/*` routes. |
| settings.update | Provides access to `POST / DELETE` `/indexes/:authorizedIndexes/settings` and `/indexes/:authorizedIndexes/settings/*` routes. |
Expand Down
124 changes: 124 additions & 0 deletions text/0191-swap-indexes-api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Swap Indexes API

## 1. Summary

The swap indexes API allows to atomically deploy several new versions of indexes without any downtime for the search clients.

## 2. Motivation

It's critical to deploy a new version of an index without any downtimes to the search clients. This capability improves the development experience by allowing Meilisearch to better fit into their workflow.

## 3. Functional Specification

### 3.1. 0 downtime deployment workflow

A 0 downtime deployment looks like this:

1. Search clients search on `indexA`.
2. The developer builds a new index `indexB` representing the new index version to deploy to the search clients.
3. When `indexB` is built and ready to be deployed, the developer sends an indexes swap request to Meilisearch for `indexA` and `indexB`.
4. `indexB` documents, settings and tasks are swapped with `indexA`.
5. Search clients search on the updated `indexA` without experiencing any downtime.

### 3.2. Deploying Multiple New Indexes Versions

The swap API supports multiple swap operations in an atomic fashion.

This means that for a search experience built using multiple indexes, Meilisearch is able to deploy all changes at once and thus clients will access the new version of all indexes at once without any downtime.

There is no need to deploy each new version of indexes one by one.

### 3.3. Enqueued Tasks After A Swap Operation Creation

Tasks enqueued after an `indexSwap` task creation date do not have their `indexUid` modified when the `indexSwap` will succeed. That is, if they are enqueued on `indexA`, they will run on the new version of `indexA`.

### 3.4. API Endpoints Definition

#### 3.4.1. `POST` - `/swap-indexes`

Send one or many indexes swap operation at once.

##### 3.4.1.1. Payload definition

The payload body expects an array of JSON objects representing swap operations.

```json
[
{
"indexes": ["indexA", "indexA_new"]
},
{
"indexes": ["indexB", "indexB_new"]
}
]
```

💡 In the given example, two swap operations will occur at the same time and atomically.

`indexA` data will be swapped with `indexA_new` data while `indexB` data will be swapped with `indexB_new` data.

> Sending `[]` is considered valid. No swap transactions will be performed.

###### Swap Object Definition

| Field | Type | Required |
|--------------------------------------------------|---------------------------------------|----------|
| [indexes](#33111-indexes) | Array of string representing indexUids| True |

###### 3.4.1.1.1. `indexes`

- Type: Array of string
- Required: True

Determines which two indexes should exchange their data for their given swap object.

##### 3.4.1.2. Response Definition

When the request is in a successful state, Meilisearch returns the HTTP code `202 Accepted`. The response's content is the summarized representation of the received asynchronous task.

```json
{
"taskUid": 1,
"indexUid": null,
"status": "enqueued",
"type": "indexSwap",
"enqueuedAt": "2021-08-12T10:00:00.000000Z"
}
```

> An `indexSwap` task is considered a global task; thus `indexUid` is null.

See [Summarized `task` Object for `202 Accepted`](0060-tasks-api.md#summarized-task-object-for-202-accepted).

##### 3.4.1.3. Errors

###### 3.4.1.3.1. Synchronous Errors

- 🔴 If the instance is secured by a master key, accessing this route without the `Authorization` header returns a [missing_authorization_header](0061-error-format-and-definitions.md#missing_authorization_header) error.
- 🔴 If the instance is secured by a master key, accessing this route with a key that does not have permissions (missing `indexes.swap` action or having a value for the `indexes` field of a swap operation not being defined in the API Key `indexes` array) (i.e. other than the master key) returns an [invalid_api_key](0061-error-format-and-definitions.md#invalid_api_key) error.
- 🔴 Omitting Content-Type header returns a [missing_content_type](0061-error-format-and-definitions.md#missing_content_type) error.
- 🔴 Sending an empty Content-Type returns an [invalid_content_type](0061-error-format-and-definitions.md#invalid_content_type) error.
- 🔴 Sending a different Content-Type than `application/json` returns an [invalid_content_type](0061-error-format-and-definitions.md#invalid_content_type) error.
- 🔴 Sending an empty payload returns a [missing_payload](0061-error-format-and-definitions.md#missing_payload) error.
- 🔴 Sending an invalid JSON payload returns a [malformed_payload](0061-error-format-and-definitions.md#malformed_payload) error.
- 🔴 Sending an `indexes` array not containing **exactly** 2 indexUids for a swap operation object returns a [bad_request](0061-error-format-and-definitions.md#bad_request) error.
- 🔴 Sending an indexUid more than once in the request payload returns a [duplicate_index_found](0061-error-format-and-definitions.md#duplicate_index_found) error.

###### 3.4.1.3.2. Asynchronous Errors

- 🔴 Sending indexUids that do not exist within the `indexes` field of a swap operation returns an [index_not_found](0061-error-format-and-definitions.md#index_not_found) error.

## 4. Technical Details

### 4.1. Swapping Data

When indexes are swapped their data is exchanged. It concerns:

- The documents
- The settings
- The tasks history
- An index swap between index_a and index_b will also replace every mention of index_a by index_b (and vice-versa) in the task history. Enqueued tasks are left unmodified.

## 5. Future Possibilities

- Introduce a way to delete one of the swapped indexes when the swap operation occurs.