Custom inference endpoints

[himsgpt]

Problem

A lot of enterprises are building their own llm models, can we use them instead of chatgpt/hugging face, etc. Sagemaker is one option but I should be able to provide just an inference endpoint to use this for prompts in jupyter lab!

Proposed Solution

A lot of enterprises are building their own llm models, can we use them instead of chatgpt/hugging face, etc. Sagemaker is one option but I should be able to provide just an inference endpoint to use this for prompts in jupyter lab!

Additional context

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[JasonWeill]

Are you asking about locally hosted models? If so, this feature request is being discussed in #190.

[himsgpt]

Yes, enterprises have been building and deploying custom llm models in on-premises. We therefore want to use these custom endpoints in the jupyter lab instead of chatgpt, etc.

[JasonWeill]

@himsgpt Thanks for clarifying! Locally-hosted and on-prem models are high priorities for organizations that can't rely on third-party models for generative AI. Let's keep the conversation going in #190.

[c3-viral-lakhani]

@JasonWeill

We have our own inference endpoint, with model already deployed. Is it possible to configure the extension to point to the endpoint or any other way to use the endpoint directly?

Does the model have to come from GPT4All so that it can be used in jupyter-ai extension?

[JasonWeill]

@c3-viral-lakhani In #322, @dlqqq added support for OpenAI proxies. GPT4All is another way to use local models. If you have another locally deployed model, I recommend filing an issue and/or opening a pull request to add support for it to Jupyter AI, to ensure that the magic commands and chat UI work with your own endpoint. Thanks for your interest!