document for huggingface(vllm) servingruntime for multi-node #402
Conversation
✅ Deploy Preview for elastic-nobel-0aef7a ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Signed-off-by: jooho lee <jlee@redhat.com>
Signed-off-by: jooho lee <jlee@redhat.com>
Signed-off-by: jooho lee <jlee@redhat.com>
| timeoutSeconds: 5 | ||
| initialDelaySeconds: 10 | ||
| .. | ||
| ~~~ |
There was a problem hiding this comment.
maybe add a note on how and where to do it?
If using custom runtime or isvc.
There was a problem hiding this comment.
this needs to be set in servingruntime. like huggingfaceserver-multinode
I add this simply to the doc.
|
|
||
| ### Key Validations | ||
|
|
||
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize`. |
There was a problem hiding this comment.
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize`. | |
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` respectively. |
| ### Key Validations | ||
|
|
||
| - `TENSOR_PARALLEL_SIZE` and `PIPELINE_PARALLEL_SIZE` cannot be set via environment variables. They must be configured through `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize`. | ||
| - In a ServingRuntime designed for multi-node, both `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` must be set. |
There was a problem hiding this comment.
there is no default expected default values for pipelineParallelSize (as it should be > 2)
There was a problem hiding this comment.
line number 16 explains the minimum value per field
| - In a ServingRuntime designed for multi-node, both `workerSpec.tensorParallelSize` and `workerSpec.pipelineParallelSize` must be set. | ||
| - The minimum value for `workerSpec.tensorParallelSize` is **1**, and the minimum value for `workerSpec.pipelineParallelSize` is **2**. | ||
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. | ||
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. |
There was a problem hiding this comment.
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. | |
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. |
There was a problem hiding this comment.
make sure it is nested from the previous bullet regarding GPUs.
There was a problem hiding this comment.
it was implemented but after discussion we decided to add a validation check for it. User have to set this.
| - The minimum value for `workerSpec.tensorParallelSize` is **1**, and the minimum value for `workerSpec.pipelineParallelSize` is **2**. | ||
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. | ||
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. | ||
| - The Autoscaler must be configured as `external`. |
There was a problem hiding this comment.
if no other autoscaler is supported, why not default to it independently of what the user defines?
There was a problem hiding this comment.
I don't think there is a way to set default autoscaler in ServingRuntime.
| - Currently, four GPU types are allowed: `nvidia.com/gpu` (*default*), `intel.com/gpu`, `amd.com/gpu`, and `habana.ai/gaudi`. | ||
| - You can specify the GPU type via InferenceService, but if it differs from what is set in the ServingRuntime, both GPU types will be assigned to the resource. Then it can cause issues. | ||
| - The Autoscaler must be configured as `external`. | ||
| - The only supported storage protocol for StorageURI is `PVC`. |
There was a problem hiding this comment.
isn't modelCar already supported by KServe?
There was a problem hiding this comment.
The first phase only supports PVC. modelcar might be included in the next phase.
|
|
||
|
|
||
| 2. **`workerSpec.pipelineParallelSize`** | ||
| This setting determines how many nodes are involved in the deployment. This variable represents the total number of nodes, including both the head and worker nodes. |
There was a problem hiding this comment.
I think it's worth mentioning that all nodes must have GPU available.
There was a problem hiding this comment.
Not all nodes need to have GPUs, as affinity can be used to select GPU-enabled nodes. Additionally, NFD (Node Feature Discovery) can add GPU labels to nodes, allowing the openshift to choose nodes with GPUs automatically.
|
|
||
| ### 2. Download the Model to the PVC | ||
|
|
||
| To download the model, export your Hugging Face token (`HF_TEST_TOKEN`) as an environment variable. Replace `%token%` with your actual token. |
There was a problem hiding this comment.
It has no special meaning.
|
|
||
| ### 2. Download the Model to the PVC | ||
|
|
||
| To download the model, export your Hugging Face token (`HF_TEST_TOKEN`) as an environment variable. Replace `%token%` with your actual token. |
There was a problem hiding this comment.
is the %string% the pattern for place holders?
I've seen on other places {{string}}
There was a problem hiding this comment.
correct, it is for placeholders.
The pattern is what I usually use but not something special. The most important thing is that the user understands what the string means. I think it is understandable for users
| !!! success "Expected Output" | ||
| ```{ .bash .no-copy } | ||
| NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE | ||
| huggingface-llama3 http://huggingface-llama3.default.example.com 12m |
There was a problem hiding this comment.
It does not matter to me. And also, actually 405b model take 12 mins only for loading the model.
Signed-off-by: jooho lee <jlee@redhat.com>
Signed-off-by: jooho lee <jlee@redhat.com>
"Fixes #issue-number" or "Add description of the problem this PR solves"
Proposed Changes
This PR add a new documentation for setting up multi-node/multi-GPU inference using the Hugging Face LLM Serving Runtime. It includes detailed instructions on prerequisites, key configurations, model inference, and sample requests for OpenAI completions and chat endpoints. This documentation aims to enhance user understanding and streamline the deployment process, ensuring a smooth experience for developers looking to leverage Hugging Face's capabilities in a Kubernetes environment
This documentation is valid only after kserve/kserve#3972 is merged.