Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support using bucket URLs without using a cache profile #343

Open
nstogner opened this issue Dec 7, 2024 · 2 comments
Open

Support using bucket URLs without using a cache profile #343

nstogner opened this issue Dec 7, 2024 · 2 comments

Comments

@nstogner
Copy link
Contributor

nstogner commented Dec 7, 2024

Currently bucket URLs (i.e. s3://, gcs://) are only supported in conjunction with .spec.cacheProfile. It would be nice to be able to load from those locations without needed to setup a cache profile.

Option A: Wait for vllm-project/vllm#10192 to land.
Option B: Load to disk/mem on the node, then point the serving engine at that mount path.

@nstogner
Copy link
Contributor Author

nstogner commented Dec 7, 2024

Option B might be useful for engines other than vLLM. It is probably worthwhile checking to see if those engines already support loading directly from bucket URLs at this point.

@nstogner
Copy link
Contributor Author

nstogner commented Dec 7, 2024

Complication with opt B is that nodes might not have sufficient storage/mem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant