Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Superduper renaming superduper #2387

Merged
merged 14 commits into from
Aug 15, 2024
Merged
2 changes: 1 addition & 1 deletion docs/content/ai_integrations/transformers.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ For a fully worked out training/ fine-tuning use-case refer to the [use-cases se

### LLM fine-tuning

superduper provides a convenient fine-tuning method based on the [trl](https://huggingface.co/docs/trl/index) framework to help you train data in the database.
`superduper` provides a convenient fine-tuning method based on the [trl](https://huggingface.co/docs/trl/index) framework to help you train data in the database.

### Supported Features

Expand Down
2 changes: 1 addition & 1 deletion docs/content/api/base/superduper.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ superduper(item: Optional[Any] = None,
| item | A database or model |
| kwargs | Additional keyword arguments to pass to the component |

Superduper API to automatically wrap an object to a db or a component.
`superduper` API to automatically wrap an object to a db or a component.

Attempts to automatically wrap an item in a superduper.ioponent by
using duck typing to recognize it.
Expand Down
4 changes: 2 additions & 2 deletions docs/content/api/components/component.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,8 @@ Component(self,
| uuid | UUID of the leaf. |
| artifacts | A dictionary of artifacts paths and `DataType` objects |

Base class for all components in superduper.
Base class for all components in `superduper`.

Class to represent superduper serializable entities
Class to represent `superduper` serializable entities
that can be saved into a database.

2 changes: 1 addition & 1 deletion docs/content/apply_api/checkpoint.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# `Checkpoint`

- Save intermediate results of training via superduper
- Save intermediate results of training via `superduper`
- Load a different point of the training process by specifying `Checkpoint` explicitly
- Particularly useful with deep-learning models

Expand Down
2 changes: 1 addition & 1 deletion docs/content/apply_api/cron_job.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# `Cron Job`

- Iterate computations, queries and actions on a crontab
- Can be deployed on superduper Enterprise
- Can be deployed on Superduper Enterprise

***Usage pattern***

Expand Down
6 changes: 3 additions & 3 deletions docs/content/apply_api/model.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# `Model`

- Wrap a standard AI model with functionality necessary for superduper
- Wrap a standard AI model with functionality necessary for `superduper`
- Configure validation and training of a model on database data

***Dependencies***
Expand Down Expand Up @@ -28,7 +28,7 @@ by adding the parameters `datatype=...` or `output_schema=...`.

## Implementations

Here are a few superduper native implementations:
Here are a few `superduper` native implementations:

**`ObjectModel`**

Expand All @@ -47,7 +47,7 @@ db.apply(m)

**`QueryModel`**

Use a superduper query to extract data from `db`
Use a `superduper` query to extract data from `db`

```python
from superduper.components.model import QueryModel
Expand Down
8 changes: 4 additions & 4 deletions docs/content/apply_api/plugin.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- Supports a plugin system that dynamically loads Python modules and packages at runtime.
- Supports functioning as subcomponents, providing dependency support for custom models and other local code.
- Capable of applying to a database and storing in the artifact_store, exporting as a superduper format package for sharing with others.
- Capable of applying to a database and storing in the artifact_store, exporting as a `superduper` format package for sharing with others.
- Supports automatic installation of dependencies listed in the requirements.txt file under the plugin.

***Usage pattern***
Expand Down Expand Up @@ -86,15 +86,15 @@ db.apply(model)

Initialization and installation

- During plugin initialization, superduper loads the component’s Python module or package into `sys.modules`, allowing subsequent use of native import statements for plugin utilization.
- During plugin initialization, `superduper` loads the component’s Python module or package into `sys.modules`, allowing subsequent use of native import statements for plugin utilization.
- If the plugin package includes a `requirements.txt`, dependencies are installed prior to loading the Python code.
- The plugin is installed only once per process; if it detects that the same plugin has already been installed in the current runtime, the installation is skipped.

Usage

- When exported locally, the plugin component saves all necessary dependency files for the plugins into the superduper package, allowing for sharing to different locations.
- When exported locally, the plugin component saves all necessary dependency files for the plugins into the `superduper` package, allowing for sharing to different locations.
- When executing `db.apply(plugin)`, the necessary Python dependency environment files for the plugin are saved in the artifact_store. During `db.load("plugin", "plugin_identifier")`, these files are downloaded to the local `~/.superduper/plugin/` directory, followed by the initialization and installation of the plugin.
- As a sub-component, superduper’s encode and decode logic ensures that the plugin is loaded prior to the parent component to maintain dependency integrity.
- As a sub-component, Superduper’s encode and decode logic ensures that the plugin is loaded prior to the parent component to maintain dependency integrity.


***See also***
Expand Down
2 changes: 1 addition & 1 deletion docs/content/apply_api/template.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
- `Template` is leveraged by `Application`.
- Snapshot allows users to know that their validation comparisons are apples-to-apples
- A `Template` is useful for sharing, migrating and distributing AI components
- A `Template` may be applied to any superduper deployment
- A `Template` may be applied to any Superduper deployment

***Usage pattern***

Expand Down
2 changes: 1 addition & 1 deletion docs/content/apply_api/trigger.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- Listen for update, inserts and deletes
- Take a specific action contigent on these changes
- Can be deployed on superduper Enterprise
- Can be deployed on Superduper Enterprise

***Usage pattern***

Expand Down
6 changes: 3 additions & 3 deletions docs/content/core_api/apply.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ db.apply(component)
A `Model` is a wrapper around a standard ML/ AI model. It may contain additional functionality, such as
pre- and post-processing, and encoding/ decoding data into/ from the correct type required by the database.

`db.apply(model)` tells superduper to store the model and its metadata in the system.
`db.apply(model)` tells `superduper` to store the model and its metadata in the system.

If additional configurations, such as training parameters, are added to the `Model` then the `db.apply` command
will also train the component on data in superduper.
Expand All @@ -35,13 +35,13 @@ Read more about `Model` [here](../apply_api/model).

### `Listener`

A `Listener` wraps a `Model`. The `db.apply(listener)` tells superduper to "listen" for incoming data and to compute outputs on those data, saving them back in superduper.
A `Listener` wraps a `Model`. The `db.apply(listener)` tells `superduper` to "listen" for incoming data and to compute outputs on those data, saving them back in `superduper`.

Read more about `Listener` [here](../apply_api/listener).

### `VectorIndex`

A `VectorIndex` wraps one or two `Listener` components, and tells superduper that the outputs computed, should
A `VectorIndex` wraps one or two `Listener` components, and tells `superduper` that the outputs computed, should
be made searchable via vector-search queries.

Read more about `VectorIndex` [here](../apply_api/vector_index).
Expand Down
6 changes: 3 additions & 3 deletions docs/content/core_api/connect.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Connect

The standard way to connect to superduper is via the `superduper` decorator:
The standard way to connect to Superduper is via the `superduper` decorator:

## Development mode

Expand All @@ -23,7 +23,7 @@ For a semi-exhaustive list of possible connections see [here](../reusable_snippe

### Fine grained configuration

superduper chooses default `artifact_store` (file blob storage) and `metadata_store` (AI metadata) values for your connection. These defaults may be overridden directly:
`superduper` chooses default `artifact_store` (file blob storage) and `metadata_store` (AI metadata) values for your connection. These defaults may be overridden directly:

```python
db = superduper(
Expand Down Expand Up @@ -66,4 +66,4 @@ metadata_store: <metadata-store-uri>
## Next steps

`db` is now your connection to your data, models, and model meta-data.
Now that you have established this connection you are ready to build, deploy and manage AI with superduper.
Now that you have established this connection you are ready to build, deploy and manage AI with Superduper.
4 changes: 2 additions & 2 deletions docs/content/data_integrations/intro.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data integrations

superduper integrates with 3 types of data-backend:
Superduper integrates with 3 types of data-backend:

- NoSQL
- [MongoDB Community Edition](https://www.mongodb.com/try/download/community)
Expand All @@ -17,7 +17,7 @@ superduper integrates with 3 types of data-backend:
- [Pandas](https://pandas.pydata.org/docs/)

Although these data-backends provide very different functionality,
with superduper they are accessible via a uniform API.
with Superduper they are accessible via a uniform API.

However, developers should bear in mind that there are a few
differences between MongoDB and SQL data-backends.
Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/basic_insertion.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Basic insertion

superduper supports inserting data wrapped as dictionaries in Python.
Superduper supports inserting data wrapped as dictionaries in Python.
These dictionaries may contain basic JSON-compatible data, but also
other data-types to be handled with `DataType` components. All data inserts are wrapped with the `Document` wrapper:

Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/data_encodings_and_schemas.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# (Optional) Setting up tables and encodings

`superduper` has flexible support for data-types. In both MongoDB and SQL databases,
Superduper has flexible support for data-types. In both MongoDB and SQL databases,
one can uses `superduper.DataType` to define one's own data-types.

If no-datatypes are provided, `superduper` [uses fallbacks](./auto_data_types.md) to encode and decode data.
Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/inserting_data.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Data inserts

superduper allows developers to insert data from a variety of sources,
Superduper allows developers to insert data from a variety of sources,
encoding and decoding objects, such as images and videos, not usually handled
explicitly by the `db.databackend`.

Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/mongodb_queries.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MongoDB select queries

superduper supports the `pymongo` query API to build select queries.
Superduper supports the `pymongo` query API to build select queries.
There is one slight difference however, since queries built with `pymongo`'s formalism
are executed lazily:

Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/native_vector_search.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export SUPERDUPER_CLUSTER_VECTOR_SEARCH_TYPE=native
If `superduper` detects this configuration, it uses the inbuilt mechanism
of your `db.databackend` to perform the vector-comparison.

Currently `superduper` supports the native implementation of these databases:
Currently Superduper supports the native implementation of these databases:

- MongoDB Atlas

Expand Down
4 changes: 2 additions & 2 deletions docs/content/execute_api/overview.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
# Execute API

superduper implements 2 main classes of `db.databackend`:
Superduper implements 2 main classes of `db.databackend`:

- [MongoDB](../data_integrations/mongodb)
- [SQL backends](../data_integrations/sql)

Correspondingly, superduper currently has 2 flavours of query API:
Correspondingly, Superduper currently has 2 flavours of query API:

- [`pymongo`](https://pymongo.readthedocs.io/en/stable/)
- [`ibis`](https://ibis-project.org/)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ sidebar_position: 15

# Working with external data sources

superduper supports data added from external data-sources.
When doing this, `superduper` supports:
Superduper supports data added from external data-sources.
When doing this, Superduper supports:

- web URLs
- URIs of objects in `s3` buckets
Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/sidecar_index_vector_search.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Sidecar vector-comparison integration

For databases which don't have their own vector-search implementation, `superduper` offers
For databases which don't have their own vector-search implementation, Superduper offers
2 integrations:

- In memory vector-search
Expand Down
4 changes: 2 additions & 2 deletions docs/content/execute_api/sql_queries.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# SQL select queries

In order to support as many data-backends as possible, superduper supports the `ibis` query API to build SQL queries.
In order to support as many data-backends as possible, Superduper supports the `ibis` query API to build SQL queries.

With `superduper` one would write:
With Superduper one would write:

```python
t = db['my_table']
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Some applications require large data-blobs and objects, which are either larger
- large audio
- videos

In order to handle such data, superduper provides a few options when
In order to handle such data, Superduper provides a few options when
creating a `DataType` via the `encodable` parameter.

## Artifact store reference with `encodable='artifact'`
Expand Down Expand Up @@ -75,7 +75,7 @@ You may verify that with this command:
iconv -f ISO-8859-1 -t UTF-8 artifacts/866cf8526595d3620d6045172fb16d1efefac4b1
```

The superduper query reloads the data and passes it to the query result,
The Superduper query reloads the data and passes it to the query result,
without any user intervention.

## Just-in-time loading with `encodable='lazy_artifact'`:
Expand Down
2 changes: 1 addition & 1 deletion docs/content/execute_api/vector_search.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Vector-search

superduper aims to provide first-class support for
Superduper aims to provide first-class support for
vector-search, including embedding computation in preparation
and run-time, as well as executing the fast vector-comparison
and returning results in a way compatible with standard database
Expand Down
2 changes: 1 addition & 1 deletion docs/content/fundamentals/data_transmision.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ In AI, typical types of data are:
- **...bespoke in house data**

Most databases don't support any data other than numbers and text.
superduper enables the use of these more interesting data-types using the `Document` wrapper.
Superduper enables the use of these more interesting data-types using the `Document` wrapper.

### `Document`

Expand Down
8 changes: 4 additions & 4 deletions docs/content/fundamentals/datalayer_overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@ sidebar_position: 3

# Datalayer

The `Datalayer` is the principle point of entry in `superduper` for:
The `Datalayer` is the principle point of entry in Superduper for:

- Communicating with the database
- Instructing models and other components to work together with the database
- Accessing and storing meta-data about your `superduper` models and data
- Accessing and storing meta-data about your Superduper models and data

Technically, the `Datalayer` "wires together" several important backends involved in the AI workflow:

- Querying the database via the **databackend**
- Storing and retrieving serialized model-weights and other artifacts from the **artifact store**
- Storing and retrieval important meta-data, from the **meta-data store** and information about models and other components which are to be installed with `superduper`
- Storing and retrieval important meta-data, from the **meta-data store** and information about models and other components which are to be installed with Superduper
- Performing computations over the data in the **databackend** using the models saved in the **artifact store**

```python
Expand Down Expand Up @@ -196,4 +196,4 @@ Validate your components (mostly models)

### `db.predict`

Infer predictions from models hosted by `superduper`. Read more about this and about models [here](../apply_api/model.md).
Infer predictions from models hosted by Superduper. Read more about this and about models [here](../apply_api/model.md).
4 changes: 2 additions & 2 deletions docs/content/fundamentals/design.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Architecture

Here is a schematic of the `superduper` design.
Here is a schematic of the Superduper design.

![](/img/light.png)

### Explanation

1. `superduper` expects data and components to be added/ updated from a range of client-side mechanisms: **scripts**, **apps**, **notebooks** or **third-party database clients** (possibly non-python).
1. Superduper expects data and components to be added/ updated from a range of client-side mechanisms: **scripts**, **apps**, **notebooks** or **third-party database clients** (possibly non-python).

1. Users and programs can add **components** (**models**, data **encoders**, **vector-indexes** and more) from the client-side. These large items are stored in the **artifact-store** and are tracked via the **meta-data** store.

Expand Down
2 changes: 1 addition & 1 deletion docs/content/fundamentals/overview.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Fundamentals

In this section, we try and guide developers through the principles and algorithms underlying `superduper`.
In this section, we try and guide developers through the principles and algorithms underlying Superduper.

For a more "how-to" approach refer first to the [tutorials](../tutorials/intro.md), [use-cases](../../use_cases), [reusable snippets](../reusable_snippets/), and to [core api](../core_api/) usage.
4 changes: 2 additions & 2 deletions docs/content/fundamentals/vector_search_algorithm.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,12 @@ sidebar_position: 7

# Vector-search

superduper allows users to implement vector-search in their database by either
Superduper allows users to implement vector-search in their database by either
using in-database functionality, or via a sidecar implementation with `lance` and `FastAPI`.

## Philosophy

In `superduper`, from a user point-of-view vector-search isn't a completely different beast than other ways of
In Superduper, from a user point-of-view vector-search isn't a completely different beast than other ways of
using the system:

- The vector-preparation is exactly the same as preparing outputs with any model,
Expand Down
2 changes: 1 addition & 1 deletion docs/content/get_started/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ tags:

# Configure

superduper provides a range of configurable options for setting
Superduper provides a range of configurable options for setting
up your environment:

Configurations can either be injected:
Expand Down
Loading