Skip to content

Commit

Permalink
Improve docs for cloud backup/restore (#31599)
Browse files Browse the repository at this point in the history
GitOrigin-RevId: 5b68c872cf2deaa5f73b5b0b9e627bfafeb352bd
  • Loading branch information
nipunn1313 authored and Convex, Inc. committed Nov 15, 2024
1 parent cc666c0 commit 8343b1e
Show file tree
Hide file tree
Showing 7 changed files with 59 additions and 57 deletions.
4 changes: 2 additions & 2 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,7 +132,7 @@ npx convex import <path>.zip
```

See description and use-cases:
[snapshot import](/docs/database/import-export/import.mdx).
[data import](/docs/database/import-export/import.mdx).

### Export data to a file

Expand All @@ -143,7 +143,7 @@ npx convex export --include-file-storage --path <path>
```

See description and use-cases:
[snapshot export](/docs/database/import-export/export.mdx).
[data export](/docs/database/import-export/export.mdx).

### Display data from tables

Expand Down
2 changes: 1 addition & 1 deletion docs/dashboard/deployments/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ shows the values configured in your `auth.config.js` for user

The
[backup & restore page](https://dashboard.convex.dev/deployment/settings/backups)
lets you [backup](/docs/database/backup-restore.mdx) the data stored in your
lets you [backup](/database/backup-restore.mdx) the data stored in your
deployment's database and file storage. On this page, you can schedule periodic
backups.

Expand Down
6 changes: 2 additions & 4 deletions docs/database/import-export/export.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Exporting Data Snapshots"
sidebar_label: "Snapshot Export"
title: "Data Export"
sidebar_label: "Data Export"
description: "Export your data out of Convex"
sidebar_position: 168
---
Expand All @@ -14,5 +14,3 @@ Alternatively, you can export the same data with the
```sh
npx convex export --path ~/Downloads
```

<BetaAdmonition feature="Snapshot export" verb="is" />
20 changes: 10 additions & 10 deletions docs/database/import-export/import-export.mdx
Original file line number Diff line number Diff line change
@@ -1,26 +1,26 @@
---
title: "Import & Export"
title: "Data Import & Export"
sidebar_position: 90
---

If you're bootstrapping your app from existing data, Convex provides three ways
to get the data in:

- One-shot import into a single table via the
[CLI](/docs/database/import-export/import.mdx#single-table-import).
- One-shot import of all tables via the
[CLI](/docs/database/import-export/import.mdx#import-data-from-a-snapshot-zip-file).
- Import from csv/json into a single table via the
[CLI](/database/import-export/import.mdx#single-table-import).
- Restore from a backup via the [dashboard](/database/backup-restore) or
[CLI](/database/import-export/import.mdx#restore-data-from-a-backup-zip-file).
- [Streaming import](/docs/production/integrations/streaming-import-export.md)
from any existing database via Fivetran or Airbyte destination connectors.
from any existing database via Airbyte destination connector.

You can export data from Convex in two ways.

- Snapshot export via the [dashboard](/docs/database/import-export/export.mdx).
Great for running a quick analysis.
- [Streaming export](/docs/production/integrations/streaming-import-export.md)
- Download a backup as a zip from the [dashboard](/database/backup-restore).
- Set up
[streaming export](/docs/production/integrations/streaming-import-export.md)
to any external database via Fivetran or Airbyte. Great for connecting to a
custom BI setup (eg [Snowflake](https://www.snowflake.com/),
[Databricks](https://www.databricks.com), or
[BigQuery](https://cloud.google.com/bigquery)):

<BetaAdmonition feature="Import & Export" verb="is" />
<BetaAdmonition feature="Data Import & Export" verb="is" />
76 changes: 39 additions & 37 deletions docs/database/import-export/import.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Importing Data Snapshots"
sidebar_label: "Snapshot Import"
title: "Data Import"
sidebar_label: "Data Import"
description: "Import data into Convex"
sidebar_position: 169
---
Expand All @@ -11,7 +11,7 @@ You can import data into Convex from a local file using the command line.
npx convex import
```

<BetaAdmonition feature="Snapshot import" verb="is" />
<BetaAdmonition feature="Data import" verb="is" />

Use `--help` to see all options. The most common flows are described here.

Expand Down Expand Up @@ -39,49 +39,52 @@ The default is to import into your dev deployment. Use `--prod` to import to
your production deployment or `--preview-name` to import into a preview
deployment.

## Import data from a snapshot ZIP file
## Restore data from a backup ZIP file

```sh
npx convex import <path>.zip
```

Import a ZIP file into a Convex deployment, where the ZIP file has been
downloaded from [Snapshot Export](/docs/database/import-export/export.mdx) in
the dashboard. Documents will retain their `_id` and `_creationTime` fields so
references between tables are maintained.
Import from a [Backup](/database/backup-restore) into a Convex deployment, where
the backup is a ZIP file that has been downloaded on the dashboard. Documents
will retain their `_id` and `_creationTime` fields so references between tables
are maintained.

Imports where tables have existing data will fail by default, but you can
specify `--replace` to replace existing data in tables mentioned in the ZIP
file.

## Use cases

1. Restore a deployment from backup. Download a snapshot export when tables are
in a good state, and restore from this backup if needed.
1. Seed dev deployments with sample data.

```sh
npx convex import --prod --replace backup_snapshot.zip
# full backup - exported from prod or another dev deployment.
npx convex import seed_data.zip

# Import single table from jsonl/csv
npx convex import --table <table name> data.jsonl
```

2. Seed dev deployments with sample data, exported from prod or another dev
deployment.
2. Restore a deployment from a [backup](/database/backup-restore)
programmatically. Download a backup, and restore from this backup if needed.

```sh
npx convex import seed_snapshot.zip
npx convex import --prod --replace backup.zip
```

3. Seed preview deployments with sample data, exported from prod, dev, or
another preview deployment. Example for Vercel, seeding data from
`seed_snapshot.zip` committed in the root of the repo.
`seed_data.zip` committed in the root of the repo.

```sh
npx convex deploy --cmd 'npm run build' &&
if [ "$VERCEL_ENV" == "preview" ]; then
npx convex import --preview-name "$VERCEL_GIT_COMMIT_REF" seed_snapshot.zip;
npx convex import --preview-name "$VERCEL_GIT_COMMIT_REF" seed_data.zip;
fi
```

4. Clear a table efficiently with an empty snapshot import.
4. Clear a table efficiently with an empty import.

```sh
touch empty_file.jsonl
Expand All @@ -90,47 +93,46 @@ npx convex import --replace --table <tableNameToClear> empty_file.jsonl

## Features

- Snapshot import is the only way to create documents with pre-existing `_id`
and `_creationTime` fields.
- Data import is the only way to create documents with pre-existing `_id` and
`_creationTime` fields.
- The `_id` field must match Convex's ID format.
- If `_id` or `_creationTime` are not provided, new values are chosen during
import.
- Snapshot import creates and replaces tables atomically (except when using
- Data import creates and replaces tables atomically (except when using
`--append`).
- Queries and mutations will not view intermediate states where partial data
is imported.
- Indexes and schemas will work on the new data without needing time for
re-backfilling or re-validating.
- Snapshot import only affects tables that are mentioned in the import, either
by `--table` or as entries in the ZIP file.
- Data import only affects tables that are mentioned in the import, either by
`--table` or as entries in the ZIP file.
- While JSON and JSONLines can import arbitrary JSON values, ZIP imports can
additionally import other Convex values: Int64, Bytes, etc. Types are
preserved in the ZIP file through the `generated_schema.jsonl` file.
- Snapshot import of ZIP files that include [file storage](/file-storage) import
the files and preserve [`_storage`](/docs/database/advanced/system-tables.mdx)
- Data import of ZIP files that include [file storage](/file-storage) import the
files and preserve [`_storage`](/docs/database/advanced/system-tables.mdx)
documents, including their `_id`, `_creationTime`, and `contentType` fields.

## Warnings

- [Streaming Export](/docs/production/integrations/streaming-import-export.md)
(Fivetran or Airbyte) does not understand snapshot imports, similar to table
deletion and creation and some schema changes. We recommend resetting
streaming export sync after a snapshot import.
- Avoid changing the ZIP file between downloading it from Snapshot Export and
(Fivetran or Airbyte) does not handle data imports or backup restorations,
similar to table deletion and creation and some schema changes. We recommend
resetting streaming export sync after a restore or a data import.
- Avoid changing the ZIP file between downloading it from Data Export and
importing it with `npx convex import`. Some manual changes of the ZIP file may
be possible, but remain undocumented. Please share your use case and check
with the Convex team in [Discord](https://convex.dev/community).
- Snapshot import is not always supported when importing into a deployment that
was created before Convex version 1.7.
- The import may work, especially when importing a ZIP snapshot from a
- Data import is not always supported when importing into a deployment that was
created before Convex version 1.7.
- The import may work, especially when importing a ZIP backup from a
deployment created around the same time as the target deployment. As a
special case, snapshot import can always restore a deployment from its own
backup.
special case, you can always restore from backups from its own deployment.
- Reach out in [Discord](https://convex.dev/community) if you encounter
issues, as there may be a workaround.

Snapshot import uses database bandwidth to write all documents, and file
bandwidth if the export includes file storage. You can observe this bandwidth in
the [usage dashboard](https://dashboard.convex.dev/team/settings/usage) as
function name `_cli/import` and associated cost in the
Data import uses database bandwidth to write all documents, and file bandwidth
if the export includes file storage. You can observe this bandwidth in the
[usage dashboard](https://dashboard.convex.dev/team/settings/usage) as function
name `_cli/import` and associated cost in the
[limits docs](/production/state/limits#database).
6 changes: 4 additions & 2 deletions docs/production.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,10 @@ Teams developing on Convex usually follow this workflow:
5. All team members can make backend changes and test them out with their
individual dev deployments. When a change is ready the team member opens a
pull-request (or commits to a shared branch).
- [Snapshot import](/docs/database/import-export/import.mdx) can be used to
populate a dev deployment with a data snapshot from prod deployment.
- [Backup / Restore](//database/backup-restore) can be used to populate a dev
deployment with data from a prod deployment.
- [Data import](/database/import-export/import.mdx) can be used to populate a
dev deployment with synthetic seed data.
- Members of a team with the [Pro plan](https://www.convex.dev/plans) can get
separate
[preview deployments](/docs/production/hosting/preview-deployments.mdx) to
Expand Down
2 changes: 1 addition & 1 deletion docs/testing/convex-backend.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ tests against the real backend has these advantages:
- Your tests will run against the same code as your Convex production (as long
you keep the local backend up-to-date).
- Limits on argument, data, query sizes are enforced.
- You can bootstrap a large test dataset from a snapshot import.
- You can bootstrap a large test dataset from a data import.
- You can test your client code in combination with your backend logic.

## Limitations
Expand Down

0 comments on commit 8343b1e

Please sign in to comment.