-
Notifications
You must be signed in to change notification settings - Fork 260
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add dummy task type to test backend plugin Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add docs page Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask models Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add function to convert resources Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add tests to `dask` task Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Remove namespace Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update setup.py Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add dask to `plugin/README.md` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Add README.md for `dask` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Top level export of `JopPodSpec` and `DaskCluster` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update docs for images Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update README.md Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update models after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update task after `flyteidl` change Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Raise error when less than 1 worker Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update flyteidl to >= 1.3.2 Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc requirements Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Update doc-requirements.txt Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Re-lock dependencies on linux Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> * Update dask API docs Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Fix documentation links Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Default optional model constructor arguments to `None` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Refactor `convert_resources_to_resource_model` to `core.resources` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Use `convert_resources_to_resource_model` in `core.node` Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> * Incorporate review feedback Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Signed-off-by: Bernhard Stadlbauer <bernhard@pachama.com> Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
- Loading branch information
1 parent
581a5c6
commit 531db3c
Showing
19 changed files
with
906 additions
and
26 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
.. _dask: | ||
|
||
################################################### | ||
Dask API reference | ||
################################################### | ||
|
||
.. tags:: Integration, DistributedComputing, KubernetesOperator | ||
|
||
.. automodule:: flytekitplugins.dask | ||
:no-members: | ||
:no-inherited-members: | ||
:no-special-members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# Flytekit Dask Plugin | ||
|
||
Flyte can execute `dask` jobs natively on a Kubernetes Cluster, which manages the virtual `dask` cluster's lifecycle | ||
(spin-up and tear down). It leverages the open-source Kubernetes Dask Operator and can be enabled without signing up | ||
for any service. This is like running a transient (ephemeral) `dask` cluster - a type of cluster spun up for a specific | ||
task and torn down after completion. This helps in making sure that the Python environment is the same on the job-runner | ||
(driver), scheduler and the workers. | ||
|
||
To install the plugin, run the following command: | ||
|
||
```bash | ||
pip install flytekitplugins-dask | ||
``` | ||
|
||
To configure Dask in the Flyte deployment's backed, follow | ||
[step 1](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_dask/index.html#step-1-deploy-the-dask-plugin-in-the-flyte-backend) | ||
and | ||
[step 2](https://docs.flyte.org/projects/cookbook/en/latest/auto/auto/integrations/kubernetes/k8s_dask/index.html#step-2-environment-setup) | ||
|
||
An [example](https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_dask/index.html) | ||
can be found in the documentation. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
""" | ||
.. currentmodule:: flytekitplugins.dask | ||
This package contains the Python related side of the Dask Plugin | ||
.. autosummary:: | ||
:template: custom.rst | ||
:toctree: generated/ | ||
Dask | ||
Scheduler | ||
WorkerGroup | ||
""" | ||
|
||
from flytekitplugins.dask.task import Dask, Scheduler, WorkerGroup |
Oops, something went wrong.