Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Python-Based Custom Runtime with MLServer can not support to deploy a model stored on a Persistent Volume Claim #494

Closed
zhlsunshine opened this issue Mar 13, 2024 · 0 comments · Fixed by #495
Assignees
Labels
question Further information is requested

Comments

@zhlsunshine
Copy link

zhlsunshine commented Mar 13, 2024

Describe the bug
I store the trained model (I use mnist-svm.joblib in my case) on a PVC, and I have some extra logics to handle the trained model after it is loaded. Therefore, I need to write a custom ServingRuntime to handle it.

To Reproduce
It works well when I follow the doc: Deploy a model stored on a Persistent Volume Claim, I can see the model file mnist-svm.joblib and model-settings.json under the folder of /models/_mlserver_models/, showing as below:
image

However, I want to write a custom ServingRuntime, so I go on to follow the doc: Python-Based Custom Runtime with MLServer, then create a new ServingRuntime and create an InferenceService for this ServingRuntime. After all these actions, everything is okay, except the inference service can not be True due to "NOT_FOUND" error in inference service, showing below:
image

Based on the comparison, I think there should be something wrong with the Python-Based Custom Runtime with MLServer when using Persistent Volume Claim to store a trained model.

Expected behavior

Hope that there is an explicit demo to show how to support the use case of python-based custom runtime with MLServer with a model stored on a Persistent Volume Claim. Thanks a lot if it's possible!

@zhlsunshine zhlsunshine added the bug Something isn't working label Mar 13, 2024
@rafvasq rafvasq self-assigned this Mar 18, 2024
@rafvasq rafvasq linked a pull request Mar 19, 2024 that will close this issue
@rafvasq rafvasq added question Further information is requested and removed bug Something isn't working labels Mar 19, 2024
rafvasq pushed a commit that referenced this issue Mar 22, 2024
#### Motivation

I'm very happy to quickly install `kserve modelmesh` by following
[quickstart](https://github.com/kserve/modelmesh-serving/blob/main/docs/quickstart.md),
but I encountered the problem when I want to write a python-based custom
`ServingRuntime` for model `mnist-svm.joblib` which is also used in many
guides and docs, and I also opened an [issue
](#494) for it.
However, I can hardly find the complete process which is available
online, including the community. So I pieced together some information
to make it's easier for user to do it.

#### Modifications

1. Add a REAMD for describing the complete process
2. Add a folder for custom ServingRuntime, including the python code,
Dockerfile and required library configuration file.

#### Result

The process for writing a python-based custom `ServingRuntime` is
completed, hope it can be a easy start for fresh user.

---------

Signed-off-by: zhlsunshine <huailong.zhang@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants