From 84fa2563a341a9d74ed33f0afa35540c99f46f8b Mon Sep 17 00:00:00 2001 From: JieguangZhou Date: Thu, 26 Sep 2024 16:39:25 +0800 Subject: [PATCH] Updated CONTRIBUTING.md --- .github/workflows/ci_plugins.yaml | 2 +- .github/workflows/release_plugins.yaml | 2 +- CHANGELOG.md | 1 + CONTRIBUTING.md | 172 +++++++++++++++++++++---- 4 files changed, 153 insertions(+), 24 deletions(-) diff --git a/.github/workflows/ci_plugins.yaml b/.github/workflows/ci_plugins.yaml index 51716c0e6..cb25200d5 100644 --- a/.github/workflows/ci_plugins.yaml +++ b/.github/workflows/ci_plugins.yaml @@ -1,4 +1,4 @@ -name: CI +name: Plugin Testing on: pull_request: diff --git a/.github/workflows/release_plugins.yaml b/.github/workflows/release_plugins.yaml index 672ecbfe3..31730f143 100644 --- a/.github/workflows/release_plugins.yaml +++ b/.github/workflows/release_plugins.yaml @@ -1,4 +1,4 @@ -name: Release in Pypi/DockerHub +name: Release plugins on: push: branches: diff --git a/CHANGELOG.md b/CHANGELOG.md index 6c36deec1..2306a786b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Fix the conflict of the same identifier during encoding. - Fix the issue where MongoDB did not create the output table. - Fix the bug in the CI where plugins are skipping tests. +- Updated CONTRIBUTING.md #### New Features & Functionality diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 86499dce0..121276aa0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -43,20 +43,23 @@ Install the dependencies: ```shell # Install pip-tools and latest version of pip -python3 -m pip install --upgrade pip-tools +pip install --upgrade pip-tools -# Install the superduper project in editable mode along with the developer tools -python3 -m pip install -e '.' -python3 -m pip install -r deploy/installations/testenv_requirements.txt -make install_devkit +# Install the SuperDuper project in editable mode, along with the necessary developer tools. +pip install -e '.[test]' ``` -(Optional) build the docker development environment: - +Install the required plugins for your development environment. ```shell -make build_sandbox +# The mongodb plugin is required for the tests (nosql) +pip install -e 'plugins/mongodb[test]' +# The ibis and sqlalchemy plugins are required for the tests (sql) +pip install -e 'plugins/ibis[test]' +pip install -e 'plugins/sqlalchemy[test]' ``` +You can install any additional plugins needed for your development environment. + ## Branch workflow We follow something called a "fork and pull request" workflow for collaborating on our project. See [here](https://gist.github.com/Chaser324/ce0505fbed06b947d962) for a great overview on what some of these mysterious terms mean! @@ -74,30 +77,128 @@ make unit_testing ### Extension integration tests -These tests that package integrations, such as `sklearn` or `openai` -work properly. +We use specific use cases to test the entire system. ```shell -make ext_testing +make usecase_testing ``` -### Databackend integration tests +### Plugin tests -These tests check that data-backend integrations such as MongoDB or SQLite -work as expected. +We maintain a set of plugins that are tested independently.. If you change the plugin code, you can run the tests for that plugin. ```shell -make databackend_testing +export PYTHONPATH=./ +# Install the plugin you want to test +pip install -e 'plugins/[test]' +# Run the tests +pytest plugins//plugin_test ``` -### Smoke tests of cluster mode -These tests check that cluster mode works as expected (`ray`, `vector-search`, `cdc`, `rest`): +## Lint and type-check the code -```shell -make smoke_testing +We use `black` for code formatting, `run` for linting, and `mypy` for type-checking. + +You can run the following commands to check the code: + +``` +make lint-and-type-check +``` + +If you want to format the code, you can run the following command: +``` +make fix-and-check +``` + +## Contributing to new plugins and templates + +The `superduper` project is designed to be extensible and customizable. You can create new plugins and templates to extend the functionality of the project. + +### Create a new plugin + +Plugins are Python packages that extend the functionality of the SuperDuper project. +More details about the plugins can be found in the documentation: + +- [Data plugins](https://docs.superduper.io/docs/data_plugins/intro) +- [AI plugins](https://docs.superduper.io/docs/category/ai-plugins) + +If you want to create a new plugin, you can follow the steps below: + +1. Copy `plugins/template` to `plugins/` +2. Modify the name of the plugin in the following names: + - Plugin name in `pyproject.toml` + - Package name `superduper_` +3. Write the code for the plugin +4. Create the tests directory `plugin_test/` and write the tests +5. Modify the `__version__` in the `superduper_/__init__.py` file. We use this version for releasing the plugin. + +We follow x.y.z versioning for the plugins, where the x.y version matches the superduper x.y version. + +We increment the z version for new releases and increment the x.y version for major releases. + +### Create a new template + +Templates are reusable components that facilitate the quick and easy building of AI applications. +More details about the templates can be found in the documentation: + +- [Templates](https://docs.superduper.io/docs/category/templates) + +If you want to create a new template, you can follow the steps below: + +1. Create a new directory in `templates/` +2. Create a `build.ipynb` file with the code to build the template + 1. Present the build process in a Jupyter notebook + 2. Package all the components into the application + 3. Build a template from the application + 4. Export the template using `template.export('.')`, and then you can get `component.json` in the directory + +## Contributing to the documentation +We maintain the documentation in the [superduper-docs](https://github.com/superduper-io/superduper-docs) repository. + + +Please go to the repository and create a pull request with the changes you want to make. + +### Fork and clone the repository + +``` +git clone git@github.com:/superduper-docs.git +cd superduper-docs ``` +### Updating the documentation + +For most document updates and additions, you can directly modify the files under `superduper-docs/content`. + +But there are some special cases: + +- Plugin documentation +- Template documentation + +**Creating of updating documentation for a plugin** + +After you create or update a plugin, you need to update the documentation. + +1. Update the `README.md` file in the plugin directory. +2. Copy the file to the `superduper-docs/content/plugins` directory and change the file name to `.md`. + +**Creating of updating documentation for a template** + +After you create or update a template, you need to update the documentation. + +We can use the `to_docusaurus_markdown.py` script to convert the Jupyter notebook to the markdown file. + +``` +python3 to_docusaurus_markdown.py /templates//build.ipynb +``` + +Then a new markdown file `/templates//build.md`. + +You can copy the file to the `superduper-docs/content/templates` directory and change the file name to `.md`. + + + + ## Create an issue If you have an unsolvable problem or find a bug with the code, we @@ -113,10 +214,37 @@ Creating a useful issue, is itself a useful skill. Think about following these p - To flag the issue to the team, escalate this in the Slack channels - Tag relevant people who have worked on that issue, or know the feature +## CI workflow + +We have two ci workflows that run on the pull requests: + +- Code Testing: Unittests, Extension Integration Tests. The code testing is run on every pull request. +- Plugin Testing: Plugin tests. The plugin testing only runs on the pull requests that change the plugin code. + +Additionally, we have a plugin release workflow that runs on the main branch. The plugin release workflow will release the plugins to the PyPI. + +### Code Testing + +1. Lint and type-check +2. Unit Testing, will run the unittests with mongodb and sqlite +3. Usecase Testing, will run the usecases with mongodb and sqlite + +### Plugin Testing + +The plugin use matrix testing to test the plugins which are changed in the pull request. + +1. Lint and type-check +2. Run `plugins//plugin_test` +3. Run the base testing(Optional): If the config file `plugins//plugin_test/config.yaml` exists, the unittests and usecases will be run with the plugin. + +### Release plugins on PyPI + +The workflow detects commit messages like `[PLUGINS] Bump Version [plugin_1 | plugin_2]` and releases the corresponding plugins to PyPI. + ## Getting Help 🙋 -[![Slack](https://img.shields.io/badge/Slack-superduperdb-8A2BE2?logo=slack)](https://join.slack.com/t/superduper-public/shared_invite/zt-1yodhtx8y-KxzECued5QBtT6JFnsSNrQ) -[![Issues](https://img.shields.io/badge/Issues-superduperdb-8A2BE2?logo=github)](https://github.com/superduper-io/superduper-stealth/issues) -[![Wiki](https://img.shields.io/badge/Project%20Wiki-superduperdb-8A2BE2?logo=github)](https://github.com/superduper-io/superduper-stealth/wiki) +[![Slack](https://img.shields.io/badge/Slack-superduper-8A2BE2?logo=slack)](https://join.slack.com/t/superduper-public/shared_invite/zt-1yodhtx8y-KxzECued5QBtT6JFnsSNrQ) +[![Issues](https://img.shields.io/badge/Issues-superduper-8A2BE2?logo=github)](https://github.com/superduper-io/superduper/issues) +[![Wiki](https://img.shields.io/badge/Project%20Wiki-superduper-8A2BE2?logo=github)](https://github.com/superduper-io/superduper/wiki) If you have any problems please contact a maintainer or community volunteer. The GitHub issues or the Slack channel are a great place to start. We look forward to seeing you there! :purple_heart: