[rust] Avoid panic in error case #3133
Conversation
| if (!Files.isRegularFile(config)) { | ||
| throw new FileNotFoundException("config.json file not found"); | ||
| } | ||
| Path file = modelDir.resolve("model.safetensors"); |
There was a problem hiding this comment.
Do we need to handle sharded checkpoints here?
There was a problem hiding this comment.
We only support safetensors, we don't support for LLM for now.
There was a problem hiding this comment.
Is it always the case for text-embedding models that they are provided in a single checkpoint? It seems from huggingface models that is overwhelmingly true, but there are some big text-embedding models in the single of billions parameter range (5b-7b) that are typically sharded but can still fit on single gpu. Example https://huggingface.co/Alibaba-NLP/gte-Qwen1.5-7B-instruct, https://huggingface.co/intfloat/e5-mistral-7b-instruct
It's not super important for now, but at some point we'll probably need to handle sharded checkpoints even for embedding models since the user is free to specify how many shards/max file size of checkpoints
Description
Brief description of what this PR is about