Skip to content

Commit

Permalink
Merge #2098
Browse files Browse the repository at this point in the history
2098: v1.0 r=guimachiavelli a=maryamsulemani97

Staging branch for v1.0.
Closes #2092, #2087, #2086, #2085, #2082, #2079, #2078, #2077, #2075, #2073, #2072, #2069, #2068, #2067, #2066, #2065

Co-authored-by: maryamsulemani97 <maryam@meilisearch.com>
Co-authored-by: gui machiavelli <hey@guimachiavelli.com>
Co-authored-by: Maryam <90181761+maryamsulemani97@users.noreply.github.com>
Co-authored-by: gui machiavelli <gui@meilisearch.com>
  • Loading branch information
5 people authored Feb 6, 2023
2 parents cce211c + 0bb031d commit 81d59b4
Show file tree
Hide file tree
Showing 29 changed files with 554 additions and 300 deletions.
2 changes: 2 additions & 0 deletions .vale/styles/Vocab/word_list/accept.txt
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,5 @@ AMIs
[Cc]ancelation
Brotli
zlib
glibc
autogenerate
5 changes: 3 additions & 2 deletions .vuepress/config.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
const ogprefix = 'og: http://ogp.me/ns#'
module.exports = {
title: 'Meilisearch Documentation v0.30',
title: 'Meilisearch Documentation v1.0',
description: 'Open source Instant Search Engine',
theme: 'default-prefers-color-scheme',
themeConfig: {
Expand Down Expand Up @@ -105,6 +105,7 @@ module.exports = {
path: '/learn/getting_started/quick_start.html',
collapsable: false,
children: [
'/learn/getting_started/supported_os',
'/learn/getting_started/quick_start',
{
title: 'Meilisearch 101',
Expand Down Expand Up @@ -341,7 +342,7 @@ module.exports = {
pattern: '**/*.md',
strictExt: true,
ignoreFilePattern: ['document_structure'],
ignoreHashPattern: ['actions-2', 'primary-key-2', 'query-parameters-2', 'query-parameters-3'],
ignoreHashPattern: ['actions-2', 'primary-key-2', 'query-parameters-2', 'query-parameters-3', 'body-2', 'body-3', 'phrase-search-2'],
exitLevel: 'warn',
},
],
Expand Down
9 changes: 5 additions & 4 deletions .vuepress/public/postman/meilisearch-collection.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"info": {
"_postman_id": "a4f9078b-c91f-4399-9a98-5e4adc5099a2",
"name": "Meilisearch v0.30",
"_postman_id": "bcc2fb5b-d130-4a7b-9acf-c481248b2b9d",
"name": "Meilisearch v1.0",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
Expand Down Expand Up @@ -481,7 +481,8 @@
"{{url}}"
],
"path": [
"tasks"
"tasks",
"cancel"
],
"query": [
{
Expand Down Expand Up @@ -2035,7 +2036,7 @@
"bearer": [
{
"key": "token",
"value": "masterKey",
"value": "aSampleMasterKey",
"type": "string"
}
]
Expand Down
8 changes: 2 additions & 6 deletions learn/advanced/dumps.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Creating a dump is also referred to as exporting it, whereas launching Meilisear

## Creating a dump

To create a dump of your dataset, to use the [create a dump endpoint](/reference/api/dump.md#create-a-dump):
To create a dump of your dataset, use the [create a dump endpoint](/reference/api/dump.md#create-a-dump):

<CodeSamples id="post_dump_1" />

Expand Down Expand Up @@ -50,7 +50,7 @@ This command should return an object with detailed information about the dump op

The dump creation process is an asynchronous task that takes time proportional to the size of your dataset. All indexes of the current instance are exported along with their documents and settings and saved as a single `.dump` file.

After dump creation is finished—when `status` is `succeeded`—the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dumps-dir` configuration option](/learn/configuration/instance_options.md#dumps-directory). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**
After dump creation is finished—when `status` is `succeeded`—the dump file is added to the dump directory. By default, this folder is named `dumps` and can be found in the same directory as your Meilisearch binary. You can customize [this using the `--dump-dir` configuration option](/learn/configuration/instance_options.md#dump-directory). **If the dump directory does not already exist when the dump creation process is called, Meilisearch will create it.**

If a dump file is visible in the file system, the dump process was successfully completed. **Meilisearch will never create a partial dump file**, even if you interrupt an instance while it is generating a dump.

Expand Down Expand Up @@ -80,10 +80,6 @@ As the data contained in the dump needs to be indexed, the process will take som
./meilisearch --import-dump /dumps/20200813-042312213.dump
```

### Importing a dump for v0.20 or below

If you are using Meilisearch v0.20 or below, migration should be done in two steps. First, import your v0.20 dump into an instance running any version of Meilisearch between v0.21 and v0.25. Second, export another dump from this instance and import it to a final instance running your targeted version.

## Use cases

Dumps are used to restore your database after [updating Meilisearch](/learn/advanced/updating.md) or to copy your database to other Meilisearch instances without having to worry about their respective versions. For more on this subject, see a [comparison of snapshots and dumps](/learn/advanced/snapshots_vs_dumps.md).
10 changes: 5 additions & 5 deletions learn/advanced/geosearch.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,7 +95,7 @@ Our restaurant dataset looks like this once we add geopositioning data:
```

::: warning
Trying to index a dataset with one or more documents containing badly formatted `_geo` values will cause Meilisearch to throw an [`invalid_geo_field`](/reference/errors/error_codes.md#invalid-geo-field) error. In this case, the update will fail and no documents will be added or modified.
Trying to index a dataset with one or more documents containing badly formatted `_geo` values will cause Meilisearch to throw an [`invalid_document_geo_field`](/reference/errors/error_codes.md#invalid-document-geo-field) error. In this case, the update will fail and no documents will be added or modified.
:::

### Using `_geo` with CSV
Expand Down Expand Up @@ -133,12 +133,12 @@ First, ensure your documents contain valid geolocation data and that you have ad
_geoRadius(lat, lng, distance_in_meters)
```

`lat` and `lng` must be floating point numbers indicating a geographic position. `distance_in_meters` must be an integer indicating the radius covered by the `_geoRadius` filter. If any of these three parameters are invalid or missing, Meilisearch will return an [`invalid_filter`](/reference/errors/error_codes.md#invalid-filter) error.
`lat` and `lng` must be floating point numbers indicating a geographic position. `distance_in_meters` must be an integer indicating the radius covered by the `_geoRadius` filter. If any of these three parameters are invalid or missing, Meilisearch will return an [`invalid_search_filter`](/reference/errors/error_codes.md#invalid-search-filter) error.

[You can read more about using `filter` in our dedicated guide.](/learn/advanced/filtering_and_faceted_search.md#using-filters)

::: warning
`_geo`, `_geoDistance`, and `_geoPoint` are not valid filter rules. Trying to use any of them with the `filter` search parameter will result in an [`invalid_filter`](/reference/errors/error_codes.md#invalid-filter) error.
`_geo`, `_geoDistance`, and `_geoPoint` are not valid filter rules. Trying to use any of them with the `filter` search parameter will result in an [`invalid_search_filter`](/reference/errors/error_codes.md#invalid-search-filter) error.
:::

### Examples
Expand Down Expand Up @@ -222,12 +222,12 @@ _geoPoint(0.0, 0.0):asc

`_geoPoint` requires two floating point numbers indicating a location's latitude and longitude. You must also specify whether the sort should be ascending (`asc`) or descending (`desc`). Ascending sort will prioritize results closer to the specified location, while descending sort will put the most distant results first.

If either `lat` or `lng` is invalid or missing, Meilisearch will return an [`invalid_sort`](/reference/errors/error_codes.md#invalid-sort) error. An error will also be thrown if you fail to indicate a sorting order.
If either `lat` or `lng` is invalid or missing, Meilisearch will return an [`invalid_search_sort`](/reference/errors/error_codes.md#invalid-search-sort) error. An error will also be thrown if you fail to indicate a sorting order.

[You can read more about sorting in our dedicated guide.](/learn/advanced/sorting.md#sorting-results-at-search-time)

::: warning
`_geo`, `_geoDistance`, and `_geoRadius` are not valid `sort` values. Trying to use any of them with the `sort` search parameter will result in an [`invalid_sort`](/reference/errors/error_codes.md#invalid-sort) error.
`_geo`, `_geoDistance`, and `_geoRadius` are not valid `sort` values. Trying to use any of them with the `sort` search parameter will result in an [`invalid_search_sort`](/reference/errors/error_codes.md#invalid-search-sort) error.
:::

### Examples
Expand Down
14 changes: 13 additions & 1 deletion learn/advanced/known_limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

Currently, Meilisearch has a number of known limitations. Some of these limitations are the result of intentional design trade-offs, while others can be attributed to [LMDB](/learn/advanced/storage.md), the key-value store that Meilisearch uses under the hood.

This guide covers hard limits that cannot be altered. Meilisearch also has some default limits that _can_ be changed, such as a [default payload limit of 100MB](/learn/configuration/instance_options.md#payload-limit-size), a [default database size limit of 100GiB](/learn/configuration/instance_options.md#max-index-size), and a [default search limit of 20 hits](/reference/api/search.md#limit).
This guide covers hard limits that cannot be altered. Meilisearch also has some default limits that _can_ be changed, such as a [default payload limit of 100MB](/learn/configuration/instance_options.md#payload-limit-size) and a [default search limit of 20 hits](/reference/api/search.md#limit).

## Maximum number of query words

Expand Down Expand Up @@ -112,3 +112,15 @@ user = 1 OR user = 2 […] OR user = 1500 OR user = 1501 […] OR user = 2000 OR
**Limitation:** Meilisearch might throw an internal error when indexing large batches of documents.

**Explanation:** Indexing a large batch of documents, such as a JSON file over 3.5GB in size, can result in Meilisearch opening too many file descriptors. Depending on your machine, this might reach your system's default resource usage limits and trigger an internal error. Use [`ulimit`](https://www.ibm.com/docs/en/aix/7.1?topic=u-ulimit-command) or a similar tool to increase resource consumption limits before running Meilisearch. For example, call `ulimit -Sn 3000` in a UNIX environment to raise the number of allowed open file descriptors to 3000.

## Maximum database size

**Limitation:** The maximum size of an index is 500GiB, and the maximum size of the task database is 10GiB.

**Explanation:** Meilisearch allocates all the virtual memory it requires upfront. The maximum database size ensures instances can contain several large indexes without reaching operating system limits on the amount of virtual memory available to a single process.

## Maximum number of indexes in an instance

**Limitation:** A single Meilisearch instance can safely contain around 180 indexes in Linux and macOS environments, and around 15 indexes in Windows environments.

**Explanation:** Operating systems restrict the amount of virtual memory available to a single process. Since Meilisearch allocates the maximum index size for each index upfront, this effectively creates a limit on how many indexes an instance can contain. The values provided are indicative: because of the particularities of each specific setup, the actual maximum number of indexes per instance will vary from machine to machine. A Linux install might contain 200 indexes without issue, or return allocation failures at 181 indexes depending on the runtime environment.
6 changes: 3 additions & 3 deletions learn/advanced/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,17 +24,17 @@ Now snapshots are created in `mySnapShots/` directory.

The first snapshot is created on launching Meilisearch. After that, snapshots are created routinely on a set interval until you deactivate snapshots by ending the Meilisearch instance. By default, one snapshot is taken every 24 hours.

The amount of time between each new snapshot can be modified with [`--snapshot-interval-sec`](/learn/configuration/instance_options.md#snapshot-interval):
The interval between each new snapshot can be modified by providing an integer to [`--schedule-snapshot`](/learn/configuration/instance_options.md#schedule-snapshot-creation):

```bash
meilisearch --schedule-snapshot --snapshot-interval-sec 3600
meilisearch --schedule-snapshot=3600
```

After running the above code, a snapshot is created every hour (3600 seconds).

During snapshot creation, old snapshots are **automatically overwritten**. This means only the most recent snapshot should be present in the folder at any given time.

[[More about snapshots flags and environment variables]](/learn/configuration/instance_options.md#schedule-snapshot-creation)
[More about snapshots flags and environment variables](/learn/configuration/instance_options.md#schedule-snapshot-creation)

## Starting from a snapshot

Expand Down
2 changes: 1 addition & 1 deletion learn/advanced/sorting.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ By default, Meilisearch focuses on ordering results according to their relevancy
This can be useful in many situations, such as when a user wants to see the cheapest products available in a webshop.

::: tip
Sorting at search time can be particularly effective when combined with [placeholder searches](/learn/what_is_meilisearch/features.md#placeholder-search).
Sorting at search time can be particularly effective when combined with [placeholder searches](/reference/api/search.md#placeholder-search).
:::

## Configuring Meilisearch for sorting at search time
Expand Down
4 changes: 1 addition & 3 deletions learn/advanced/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,7 @@ The choice of LMDB comes with certain pros and cons, especially regarding databa

#### Database size

When deleting documents from a Meilisearch index, you may notice disk space usage remains the same. This happens because LMDB internally marks that space as free, but does not make it available for the operating system at large. This design choice leads to better performance, as there is no need for periodic compaction operations. As a result, disk space occupied by LMDB (and thus by Meilisearch) tends to increase over time.

It is not currently possible to calculate the precise maximum amount of space a Meilisearch instance can occupy. However, disk space usage is dictated in part by index and task database size. This means you can have a general idea of the upper size boundaries by taking into account the number of indexes in your instance together with the values configured for `--max-index-size` and `--max-task-db`.
When deleting documents from a Meilisearch index, you may notice disk space usage remains the same. This happens because LMDB internally marks that space as free, but does not make it available for the operating system at large. This design choice leads to better performance, as there is no need for periodic compaction operations. As a result, disk space occupied by LMDB (and thus by Meilisearch) tends to increase over time. It is not possible to calculate the precise maximum amount of space a Meilisearch instance can occupy.

#### Memory usage

Expand Down
10 changes: 3 additions & 7 deletions learn/advanced/updating.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@ This guide only works for versions v0.15 and above. If you are using an older ve

This section contains instructions for upgrading from specific versions. Most versions don't require version-specific steps and you should be able to upgrade directly. If the version you are upgrading from isn't listed here, no additional steps are required.

- If you are updating **from v0.20 or below**, please ensure all updates finish processing before creating the dump. `enqueued` updates will not be exported and may result in data loss.

- If you are using **v0.24 or below**, use the `X-Meili-API-Key: API_KEY` authorization header:

<CodeSamples id="updating_guide_check_version_old_authorization_header" />
Expand Down Expand Up @@ -79,7 +81,7 @@ Once the status is `processed`, you're good to go. Repeat this process for all i

### Create the dump

Before creating your dump, make sure that your [dump directory](/learn/configuration/instance_options.md#dumps-directory) is somewhere accessible. By default, dumps are created in a folder called `dumps` at the root of your Meilisearch directory.
Before creating your dump, make sure that your [dump directory](/learn/configuration/instance_options.md#dump-directory) is somewhere accessible. By default, dumps are created in a folder called `dumps` at the root of your Meilisearch directory.

**Cloud platforms** like DigitalOcean, AWS, and GCP are configured to store dumps in the `/var/opt/meilisearch/dumps` directory.

Expand Down Expand Up @@ -258,12 +260,6 @@ For **cloud platforms**, move the new Meilisearch binary to the `/usr/bin` direc
mv meilisearch /usr/bin/meilisearch
```

::: warning
If you are using Meilisearch v0.20 or below, migration should be done in two steps. First, import your dump into an instance running any version of Meilisearch from v0.21 to v0.24, inclusive. Second, export another dump from this instance and import it to a final instance running your targeted version.

Once Meilisearch v1 is released, this two-step process won't be necessary as v1 will be compatible with dumps from all previous versions.
:::

## Step 3: Import data

### Launch Meilisearch and import the dump
Expand Down
Loading

0 comments on commit 81d59b4

Please sign in to comment.