Better error messages when failing to load models with sentencepiece based tokenizers

[Harsha-Nori]

Just surfacing several discussions on an issue to track. Today, guidance throws a cryptic error if you try to install transformers and guidance without sentencepiece installed for models with sentencepiece-based tokenizers due to the path not being taken. We had some exception handling logic earlier, but it isn't robust enough to reliably recommend this to new users anymore.

pip install guidance transformers

from guidance import models, gen
lm = models.Transformers("microsoft/Phi-3-mini-4k-instruct", trust_remote_code=True)

File ~/miniconda3/envs/guidance/lib/python3.12/site-packages/guidance/models/transformers/_transformers.py:104, in TransformersTokenizer.__init__(self, model, transformers_tokenizer, chat_template, ignore_bos_token, **kwargs)
    102 if hasattr(transformers_tokenizer, "convert_tokens_to_string"):
    103     token_str = transformers_tokenizer.convert_tokens_to_string([token])
--> 104     roundtrip_id = transformers_tokenizer.encode(token_str)[0]
    105     if roundtrip_id == i:
    106         byte_coded = token_str.encode()

IndexError: list index out of range

[Harsha-Nori]

Oops, nevermind, captured well already here :): #958

[hudson-ai]

I'm suddenly getting this exception in CI despite having "fixed" it before. Seems a recent change impacted this. I'll open a PR so we can discuss how to best handle errors here.

[Saibo-creator]

I encountered the same error, and it turned out that sentencepiece wasn't installed. Installing sentencepiece resolved the problem.