-
Notifications
You must be signed in to change notification settings - Fork 59
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* docs: change to sphinx docs Signed-off-by: Keming <kemingy94@gmail.com> * fix pages ci Signed-off-by: Keming <kemingy94@gmail.com> * fix rst Signed-off-by: Keming <kemingy94@gmail.com> * fix ci for pages Signed-off-by: Keming <kemingy94@gmail.com> * Update mosec/worker.py Co-authored-by: zclzc <38581401+lkevinzc@users.noreply.github.com> Signed-off-by: Keming <kemingy94@gmail.com> * rm design doc Signed-off-by: Keming <kemingy94@gmail.com> --------- Signed-off-by: Keming <kemingy94@gmail.com> Co-authored-by: zclzc <38581401+lkevinzc@users.noreply.github.com>
- Loading branch information
Showing
37 changed files
with
528 additions
and
313 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,13 +1,15 @@ | ||
# Contributing to `Mosec` | ||
|
||
Before contributing to this repository, please first discuss the change you wish to make via issue, email, or any other method with the owners of this repository before making a change. | ||
|
||
### Pull Request Process | ||
## Pull Request Process | ||
|
||
1. After you have forked this repository, you could use `make install` for *the first time* to install the local development dependencies; afterward, you may use `make dev` to build the library when you have made any code changes. | ||
2. Before committing your changes, you can use `make format && make lint` to ensure the codes follow our style standards. | ||
3. Please add corresponding tests to your change if that's related to new feature or API, and ensure `make test` can pass. | ||
4. Submit your pull request. | ||
|
||
### Contacts | ||
## Contacts | ||
|
||
- Keming <kemingy94@gmail.com> | ||
- zclzc <lkevinzc@gmail.com> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# Minimal makefile for Sphinx documentation | ||
# | ||
|
||
# You can set these variables from the command line, and also | ||
# from the environment for the first two. | ||
SPHINXOPTS ?= | ||
SPHINXBUILD ?= sphinx-build | ||
SOURCEDIR = source | ||
BUILDDIR = build | ||
|
||
# Put it first so that "make" without argument is like "make help". | ||
help: | ||
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) | ||
|
||
.PHONY: help Makefile | ||
|
||
# Catch-all target: route all unknown targets to Sphinx using the new | ||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). | ||
%: Makefile | ||
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
@ECHO OFF | ||
|
||
pushd %~dp0 | ||
|
||
REM Command file for Sphinx documentation | ||
|
||
if "%SPHINXBUILD%" == "" ( | ||
set SPHINXBUILD=sphinx-build | ||
) | ||
set SOURCEDIR=source | ||
set BUILDDIR=build | ||
|
||
%SPHINXBUILD% >NUL 2>NUL | ||
if errorlevel 9009 ( | ||
echo. | ||
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx | ||
echo.installed, then set the SPHINXBUILD environment variable to point | ||
echo.to the full path of the 'sphinx-build' executable. Alternatively you | ||
echo.may add the Sphinx directory to PATH. | ||
echo. | ||
echo.If you don't have Sphinx installed, grab it from | ||
echo.https://www.sphinx-doc.org/ | ||
exit /b 1 | ||
) | ||
|
||
if "%1" == "" goto help | ||
|
||
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% | ||
goto end | ||
|
||
:help | ||
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% | ||
|
||
:end | ||
popd |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,65 @@ | ||
# Configuration file for the Sphinx documentation builder. | ||
# | ||
# For the full list of built-in configuration values, see the documentation: | ||
# https://www.sphinx-doc.org/en/master/usage/configuration.html | ||
|
||
import os | ||
import sys | ||
|
||
sys.path.insert(0, os.path.abspath("../..")) | ||
|
||
# -- Project information ----------------------------------------------------- | ||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information | ||
|
||
project = "mosec" | ||
copyright = "2023, mosec maintainers" | ||
author = "mosec maintainers" | ||
release = "latest" | ||
|
||
# -- General configuration --------------------------------------------------- | ||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration | ||
|
||
extensions = [ | ||
"sphinx.ext.viewcode", | ||
"sphinx.ext.autodoc", | ||
"sphinx.ext.githubpages", | ||
"sphinx.ext.napoleon", | ||
"myst_parser", | ||
"sphinx_copybutton", | ||
"sphinxcontrib.programoutput", | ||
"sphinx_autodoc_typehints", | ||
"sphinxext.opengraph", | ||
] | ||
|
||
templates_path = ["_templates"] | ||
exclude_patterns = [] | ||
source_suffix = [".rst", ".md"] | ||
master_doc = "index" | ||
language = "en" | ||
|
||
# Extension configuration | ||
myst_heading_anchors = 3 | ||
autodoc_member_order = "bysource" | ||
# https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html | ||
napoleon_attr_annotations = True | ||
napoleon_include_init_with_doc = True | ||
napoleon_use_admonition_for_references = True | ||
# https://sphinxext-opengraph.readthedocs.io/en/latest/ | ||
ogp_site_url = "https://mosecorg.github.io/mosec/" | ||
ogp_image = "https://user-images.githubusercontent.com/38581401/134487662-49733d45-2ba0-4c19-aa07-1f43fd35c453.png" | ||
|
||
# -- Options for HTML output ------------------------------------------------- | ||
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output | ||
|
||
html_theme = "furo" | ||
html_logo = "https://user-images.githubusercontent.com/38581401/134487662-49733d45-2ba0-4c19-aa07-1f43fd35c453.png" | ||
html_static_path = ["_static"] | ||
|
||
# Theme | ||
html_theme_options = { | ||
"sidebar_hide_name": True, | ||
"navigation_with_keys": True, | ||
"source_repository": "https://github.com/mosecorg/mosec/", | ||
"source_branch": "main", | ||
"source_directory": "docs/", | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
```{include} ../../../CONTRIBUTING.md | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# Development | ||
|
||
```{toctree} | ||
--- | ||
hidden: | ||
--- | ||
contributing | ||
``` | ||
|
||
- {doc}`contributing` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,19 +1,25 @@ | ||
# Echo Example | ||
|
||
An echo server is usually the very first server you wanna implement to get familiar with the framework. | ||
|
||
This server sleeps for a given period and return. It is a simple illustration of how **multi-stage workload** is implemented. It also shows how to write a simple **validation** for input data. | ||
|
||
The default JSON protocol will be used since the (de)serialization methods are not overridden in this demo. In particular, the input `data` of `Preprocess`'s `forward` is a dictionary decoded by JSON from the request body's bytes; and the output dictionary of `Postprocess`'s `forward` will be JSON-encoded as a mirrored process. | ||
|
||
#### **`echo.py`** | ||
## **`echo.py`** | ||
|
||
```python | ||
--8<-- "examples/echo.py" | ||
```{include} ../../../examples/echo.py | ||
:code: python | ||
``` | ||
|
||
#### Start | ||
## Start | ||
|
||
python echo.py | ||
```shell | ||
python echo.py | ||
``` | ||
|
||
#### Test | ||
## Test | ||
|
||
http :8000/inference time=1.5 | ||
```shell | ||
http :8000/inference time=1.5 | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,19 +1,25 @@ | ||
# Customized GPU Allocation | ||
|
||
This is an example demonstrating how to give different worker processes customized environment variables to control things like GPU device allocation, etc. | ||
|
||
Assume your machine has 4 GPUs, and you hope to deploy your model to all of them to handle inference requests in parallel, maximizing your service's throughput. With MOSEC, we provide parallel workers with customized environment variables to satisfy the needs. | ||
|
||
As shown in the codes below, we can define our inference worker together with a list of environment variable dictionaries, each of which will be passed to the corresponding worker process. For example, if we set `CUDA_VISIBLE_DEVICES` to `0-3`, (the same copy of) our model will be deployed on 4 different GPUs and be queried in parallel, largely improving the system's throughput. You could verify this either from the server logs or the client response. | ||
|
||
#### **`custom_env.py`** | ||
## **`custom_env.py`** | ||
|
||
```python | ||
--8<-- "examples/custom_env.py" | ||
```{include} ../../../examples/custom_env.py | ||
:code: python | ||
``` | ||
|
||
#### Start | ||
## Start | ||
|
||
python custom_env.py | ||
```shell | ||
python custom_env.py | ||
``` | ||
|
||
#### Test | ||
## Test | ||
|
||
curl -X POST http://127.0.0.1:8000/inference -d '{"dummy": 0}' | ||
```shell | ||
http :8000/inference dummy=0 | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,11 +1,31 @@ | ||
# Examples | ||
|
||
```{toctree} | ||
--- | ||
hidden: | ||
--- | ||
echo | ||
env | ||
ipc | ||
jax | ||
metric | ||
pytorch | ||
stable_diffusion | ||
``` | ||
|
||
We provide examples across different ML frameworks and for various tasks in this section. | ||
|
||
## Get started | ||
## Requirements | ||
|
||
All the examples in this section are self-contained and tested. Feel free to grab one and run: | ||
|
||
python model_server.py | ||
```shell | ||
python model_server.py | ||
``` | ||
|
||
To test the server, we use [`httpie`](https://github.com/httpie/httpie) and [`httpx`](https://github.com/encode/httpx) by default. You can have other choices but if you want to install them: | ||
|
||
pip install httpie httpx | ||
```shell | ||
pip install httpie httpx | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.