doc: Add custom runtime with PVC example #495

zhlsunshine · 2024-03-19T10:02:23Z

Motivation

I'm very happy to quickly install kserve modelmesh by following quickstart, but I encountered the problem when I want to write a python-based custom ServingRuntime for model mnist-svm.joblib which is also used in many guides and docs, and I also opened an issue for it. However, I can hardly find the complete process which is available online, including the community. So I pieced together some information to make it's easier for user to do it.

Modifications

Add a REAMD for describing the complete process
Add a folder for custom ServingRuntime, including the python code, Dockerfile and required library configuration file.

Result

The process for writing a python-based custom ServingRuntime is completed, hope it can be a easy start for fresh user.

rafvasq

Thank you @zhlsunshine! I think having a complete end-to-end example for this will be very helpful. I've done a first-pass and reviewed some edits that I think will benefit the doc.

I also think that since this is an E2E example along with ready-to-use code, it can be added to /docs/examples instead, maybe in its own directory called /docs/examples/python-custom-runtime.

.../python-based-custom-runtime-example/Python-Based-Custom-Runtime-with-Model-Stored-on-PVC.md

zhlsunshine · 2024-03-20T01:34:18Z

Hi @rafvasq, thanks for reviewing and comments, I moved this e2e example into /docs/examples/python-custom-runtime as you suggested, and change the doc according to your comments.

rafvasq

Thanks @zhlsunshine, this looks great. Just the small nit, run make run fmt to get past the Lint check, and rebase.

docs/examples/python-custom-runtime/Python-Based-Custom-Runtime-with-Model-Stored-on-PVC.md

zhlsunshine · 2024-03-21T04:13:30Z

Thanks @zhlsunshine, this looks great. Just the small nit, run make run fmt to get past the Lint check, and rebase.

Hi @rafvasq rebasing done!

#### Motivation Triton introduced [support for more model frameworks last year](https://developer.nvidia.com/blog/real-time-serving-for-xgboost-scikit-learn-randomforest-lightgbm-and-more/) and can support xgboost, lightgbm, and more. This PR adds examples and docs to advertise this. #### Modifications - Add newly supported models to Triton runtime config, setting `autoSelect: false`. - Add an example ISVC config for Triton-served XGBoost model. - Update example-models doc to reflect example models added in kserve/modelmesh-minio-examples#7 - Update model-formats README to reflect framework support and framework-specific docs to show example ISVC using Triton. - Add FVTs for lightgbm and xgboost deployment on Triton runtime #### Result Closes kserve#185 --------- Signed-off-by: Rafael Vasquez <raf.vasquez@ibm.com> Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com> Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

…ording to comments Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

This is to move away from CVE-20204-24786: infinite loop vulnerability. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com> Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

zhlsunshine · 2024-03-22T03:40:37Z

Hi @rafvasq, I also completed the golangci-lint process, submitted the formatted docs.

rafvasq

Thanks @zhlsunshine!

oss-prow-bot · 2024-03-22T14:04:13Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rafvasq, zhlsunshine

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rafvasq]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

oss-prow-bot · 2024-03-22T14:04:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rafvasq, zhlsunshine

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rafvasq]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rafvasq · 2024-03-22T14:04:27Z

/lgtm

oss-prow-bot bot requested review from ckadner, njhill and rafvasq March 19, 2024 10:02

zhlsunshine force-pushed the cust-mlmodel branch from 1085963 to 613e31c Compare March 19, 2024 10:03

rafvasq linked an issue Mar 19, 2024 that may be closed by this pull request

The Python-Based Custom Runtime with MLServer can not support to deploy a model stored on a Persistent Volume Claim #494

Closed

rafvasq requested changes Mar 19, 2024

View reviewed changes

zhlsunshine force-pushed the cust-mlmodel branch from 08f4e21 to e4d8169 Compare March 20, 2024 01:26

rafvasq self-requested a review March 20, 2024 18:08

rafvasq requested changes Mar 20, 2024

View reviewed changes

docs/examples/python-custom-runtime/Python-Based-Custom-Runtime-with-Model-Stored-on-PVC.md Outdated Show resolved Hide resolved

zhlsunshine force-pushed the cust-mlmodel branch from 3c233a2 to 786c913 Compare March 21, 2024 01:06

rafvasq and others added 7 commits March 22, 2024 11:27

add the python-based custom servingruntime with PVC

049788c

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

fix some typos

018aa57

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

move the e2e example to docs/examples/ and change the description acc…

1bbd402

…ording to comments Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

remove the quote in content

03e82be

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

chore: update google.golang.org/protobuf to v1.33.0 (kserve#496)

072626b

This is to move away from CVE-20204-24786: infinite loop vulnerability. Signed-off-by: Edgar Hernández <23639005+israel-hdez@users.noreply.github.com> Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

update the chagne by golangci-lint

1a8cc6c

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>

zhlsunshine force-pushed the cust-mlmodel branch from dd9fdab to 1a8cc6c Compare March 22, 2024 03:39

Merge branch 'main' into cust-mlmodel

9445558

rafvasq approved these changes Mar 22, 2024

View reviewed changes

oss-prow-bot bot added the approved label Mar 22, 2024

oss-prow-bot bot assigned rafvasq Mar 22, 2024

oss-prow-bot bot added the lgtm label Mar 22, 2024

rafvasq changed the title ~~The Python-Based Custom Runtime with Model Stored on Persistent Volume Claim~~ doc: Add custom runtime with PVC example Mar 22, 2024

rafvasq merged commit 70cdc3e into kserve:main Mar 22, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: Add custom runtime with PVC example #495

doc: Add custom runtime with PVC example #495

zhlsunshine commented Mar 19, 2024

rafvasq left a comment

zhlsunshine commented Mar 20, 2024

rafvasq left a comment •

edited

Loading

zhlsunshine commented Mar 21, 2024 •

edited

Loading

zhlsunshine commented Mar 22, 2024

rafvasq left a comment

oss-prow-bot bot commented Mar 22, 2024

oss-prow-bot bot commented Mar 22, 2024

rafvasq commented Mar 22, 2024

doc: Add custom runtime with PVC example #495

doc: Add custom runtime with PVC example #495

Conversation

zhlsunshine commented Mar 19, 2024

Motivation

Modifications

Result

rafvasq left a comment

Choose a reason for hiding this comment

zhlsunshine commented Mar 20, 2024

rafvasq left a comment • edited Loading

Choose a reason for hiding this comment

zhlsunshine commented Mar 21, 2024 • edited Loading

zhlsunshine commented Mar 22, 2024

rafvasq left a comment

Choose a reason for hiding this comment

oss-prow-bot bot commented Mar 22, 2024

oss-prow-bot bot commented Mar 22, 2024

rafvasq commented Mar 22, 2024

rafvasq left a comment •

edited

Loading

zhlsunshine commented Mar 21, 2024 •

edited

Loading