Updated CONTRIBUTING.md

.github/workflows/ci_plugins.yaml

@@ -1,4 +1,4 @@
-name: CI
+name: Plugin Testing
 
 on:
   pull_request:

.github/workflows/release_plugins.yaml

@@ -1,4 +1,4 @@
-name: Release in Pypi/DockerHub
+name: Release plugins
 on:
   push:
     branches:

CHANGELOG.md

@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fix the conflict of the same identifier during encoding.
 - Fix the issue where MongoDB did not create the output table.
 - Fix the bug in the CI where plugins are skipping tests.
+- Updated CONTRIBUTING.md
 
 #### New Features & Functionality
 

CONTRIBUTING.md

@@ -43,20 +43,23 @@ Install the dependencies:
 
 ```shell
 # Install pip-tools and latest version of pip
-python3 -m pip install --upgrade pip-tools
+pip install --upgrade pip-tools
 
-# Install the superduper project in editable mode along with the developer tools
-python3 -m pip install -e '.'
-python3 -m pip install -r deploy/installations/testenv_requirements.txt
-make install_devkit
+# Install the SuperDuper project in editable mode, along with the necessary developer tools.
+pip install -e '.[test]'
 ```
 
-(Optional) build the docker development environment:
-
+Install the required plugins for your development environment.
 ```shell
-make build_sandbox
+# The mongodb plugin is required for the tests (nosql)
+pip install -e 'plugins/mongodb[test]'
+# The ibis and sqlalchemy plugins are required for the tests (sql)
+pip install -e 'plugins/ibis[test]'
+pip install -e 'plugins/sqlalchemy[test]'
 ```
 
+You can install any additional plugins needed for your development environment.
+
 ## Branch workflow
 
 We follow something called a "fork and pull request" workflow for collaborating on our project. See [here](https://gist.github.com/Chaser324/ce0505fbed06b947d962) for a great overview on what some of these mysterious terms mean! 
@@ -74,30 +77,128 @@ make unit_testing
 
 ### Extension integration tests
 
-These tests that package integrations, such as `sklearn` or `openai`
-work properly.
+We use specific use cases to test the entire system.
 
 ```shell
-make ext_testing
+make usecase_testing
 ```
 
-### Databackend integration tests
+### Plugin tests
 
-These tests check that data-backend integrations such as MongoDB or SQLite 
-work as expected.
+We maintain a set of plugins that are tested independently.. If you change the plugin code, you can run the tests for that plugin.
 
 ```shell
-make databackend_testing
+export PYTHONPATH=./
+# Install the plugin you want to test
+pip install -e 'plugins/<PLUGIN_NAME>[test]'
+# Run the tests
+pytest plugins/<PLUGIN_NAME>/plugin_test
 ```
 
-### Smoke tests of cluster mode
 
-These tests check that cluster mode works as expected (`ray`, `vector-search`, `cdc`, `rest`):
+## Lint and type-check the code
 
-```shell
-make smoke_testing
+We use `black` for code formatting, `run` for linting, and `mypy` for type-checking.
+
+You can run the following commands to check the code:
+
+```
+make lint-and-type-check
+```
+
+If you want to format the code, you can run the following command:
+```
+make fix-and-check
+```
+
+## Contributing to new plugins and templates
+
+The `superduper` project is designed to be extensible and customizable. You can create new plugins and templates to extend the functionality of the project.
+
+### Create a new plugin
+
+Plugins are Python packages that extend the functionality of the SuperDuper project.
+More details about the plugins can be found in the documentation:
+
+- [Data plugins](https://docs.superduper.io/docs/data_plugins/intro)
+- [AI plugins](https://docs.superduper.io/docs/category/ai-plugins)
+
+If you want to create a new plugin, you can follow the steps below:
+
+1. Copy `plugins/template` to `plugins/<PLUGIN_NAME>`
+2. Modify the name of the plugin in the following names:
+   - Plugin name in `pyproject.toml`
+   - Package name `superduper_<PLUGIN_NAME>`
+3. Write the code for the plugin
+4. Create the tests directory `plugin_test/` and write the tests
+5. Modify the `__version__` in the `superduper_<PLUGIN_NAME>/__init__.py` file. We use this version for releasing the plugin.
+
+We follow x.y.z versioning for the plugins, where the x.y version matches the superduper x.y version.
+
+We increment the z version for new releases and increment the x.y version for major releases.
+
+### Create a new template
+
+Templates are reusable components that facilitate the quick and easy building of AI applications.
+More details about the templates can be found in the documentation:
+
+- [Templates](https://docs.superduper.io/docs/category/templates)
+
+If you want to create a new template, you can follow the steps below:
+
+1. Create a new directory in `templates/<TEMPLATE_NAME>`
+2. Create a `build.ipynb` file with the code to build the template
+   1. Present the build process in a Jupyter notebook
+   2. Package all the components into the application
+   3. Build a template from the application
+   4. Export the template using `template.export('.')`, and then you can get `component.json` in the directory
+
+## Contributing to the documentation
+We maintain the documentation in the [superduper-docs](https://github.com/superduper-io/superduper-docs) repository.
+
+
+Please go to the repository and create a pull request with the changes you want to make.
+
+### Fork and clone the repository
+
+```
+git clone git@github.com:<FORKED_NAME>/superduper-docs.git
+cd superduper-docs
 ```
 
+### Updating the documentation
+
+For most document updates and additions, you can directly modify the files under `superduper-docs/content`.
+
+But there are some special cases:
+
+- Plugin documentation
+- Template documentation
+
+**Creating of updating documentation for a plugin**
+
+After you create or update a plugin, you need to update the documentation.
+
+1. Update the `README.md` file in the plugin directory.
+2. Copy the file to the `superduper-docs/content/plugins` directory and change the file name to `<PLUGIN_NAME>.md`.
+
+**Creating of updating documentation for a template**
+
+After you create or update a template, you need to update the documentation.
+
+We can use the `to_docusaurus_markdown.py` script to convert the Jupyter notebook to the markdown file.
+
+```
+python3 to_docusaurus_markdown.py <Your superduper project path>/templates/<TEMPLATE_NAME>/build.ipynb
+```
+
+Then a new markdown file `<Your superduper project path>/templates/<TEMPLATE_NAME>/build.md`.
+
+You can copy the file to the `superduper-docs/content/templates` directory and change the file name to `<TEMPLATE_NAME>.md`.
+
+
+
+
 ## Create an issue
 
 If you have an unsolvable problem or find a bug with the code, we
@@ -113,10 +214,37 @@ Creating a useful issue, is itself a useful skill. Think about following these p
 - To flag the issue to the team, escalate this in the Slack channels
 - Tag relevant people who have worked on that issue, or know the feature
 
+## CI workflow
+
+We have two ci workflows that run on the pull requests:
+
+- Code Testing: Unittests, Extension Integration Tests. The code testing is run on every pull request.
+- Plugin Testing: Plugin tests. The plugin testing only runs on the pull requests that change the plugin code.
+
+Additionally, we have a plugin release workflow that runs on the main branch. The plugin release workflow will release the plugins to the PyPI.
+
+### Code Testing
+
+1. Lint and type-check
+2. Unit Testing, will run the unittests with mongodb and sqlite
+3. Usecase Testing, will run the usecases with mongodb and sqlite
+
+### Plugin Testing
+
+The plugin use matrix testing to test the plugins which are changed in the pull request.
+
+1. Lint and type-check
+2. Run `plugins/<PLUGIN_NAME>/plugin_test`
+3. Run the base testing(Optional): If the config file `plugins/<PLUGIN_NAME>/plugin_test/config.yaml` exists, the unittests and usecases will be run with the plugin.
+
+### Release plugins on PyPI
+
+The workflow detects commit messages like `[PLUGINS] Bump Version [plugin_1 | plugin_2]` and releases the corresponding plugins to PyPI.
+
 ## Getting Help 🙋
 
-[![Slack](https://img.shields.io/badge/Slack-superduperdb-8A2BE2?logo=slack)](https://join.slack.com/t/superduper-public/shared_invite/zt-1yodhtx8y-KxzECued5QBtT6JFnsSNrQ)
-[![Issues](https://img.shields.io/badge/Issues-superduperdb-8A2BE2?logo=github)](https://github.com/superduper-io/superduper-stealth/issues)
-[![Wiki](https://img.shields.io/badge/Project%20Wiki-superduperdb-8A2BE2?logo=github)](https://github.com/superduper-io/superduper-stealth/wiki)
+[![Slack](https://img.shields.io/badge/Slack-superduper-8A2BE2?logo=slack)](https://join.slack.com/t/superduper-public/shared_invite/zt-1yodhtx8y-KxzECued5QBtT6JFnsSNrQ)
+[![Issues](https://img.shields.io/badge/Issues-superduper-8A2BE2?logo=github)](https://github.com/superduper-io/superduper/issues)
+[![Wiki](https://img.shields.io/badge/Project%20Wiki-superduper-8A2BE2?logo=github)](https://github.com/superduper-io/superduper/wiki)
 
 If you have any problems please contact a maintainer or community volunteer. The GitHub issues or the Slack channel are a great place to start. We look forward to seeing you there! :purple_heart: