Add more summarization models

[enjalot]

We could easily add more models to the list of chat models used for summarization:
https://github.com/enjalot/latent-scope/blob/main/latentscope/models/chat_models.json

There are plenty of small open source LLMs on HuggingFace that could be added to the list as long as the transformers provider is compatible.

We should also add more Chat API models, currently only OpenAI and Mistral are implemented in providers but Cohere and Together should be easy to add.

Comments on this issue for which models to add would be a helpful start. Especially if there are fine-tuned small models focused on summarization (my experience with out-of-the box 1B and 7B is that their quality isn't great).

[enjalot]

I've updated in 0.2.3 to add Llama 3 8b instruct and removed some older transformer chat models that didn't perform well. It seems to do at least as well as gpt-3.5 did.

Adding more transformer models should still be done, but it's a little more complicated as some of them seem to have different prompt formats.

I still want to add Claude and other API providers.

[enjalot]

Ideally we use a similar HF hub search that was recently enabled for the embedding models. the only issue would be automatically sorting the chat templating for summarizing. I'm sure there is a library for this we could use

[enjalot]

Perhaps we use https://github.com/dottxt-ai/outlines
to simplify prompt template across models and get more reliable structured output (currently doing some hacks to get just the label back from the model)

[enjalot]

I think we can search for "conversational" + "text-generation-interface" to get useful instruct models:
https://huggingface.co/models?other=conversational,text-generation-inference&sort=downloads

[meuleman]

#66 adds basic support for ollama, which provides a ton of models that can be run on a user's local machine (chat & embed): https://ollama.com/library

[enjalot]

this was accomplished with better huggingface support as well as ollama support