Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend the ability to support third party LLM through customizable API URL #143

Closed
wants to merge 2 commits into from

Conversation

vaderyang
Copy link

Add api_url , max_tokens into settings. These settings extends the aichat to support third party LLM services which using OPENAI-like API such as llama-cpp-python[server], LocalAI and etc.

Example:
start local API service first, below is an example for aichat with a local codellama on a MacBook laptop

python3 -m pip install llama-cpp-python
python3 -m llama_cpp.server --n_gpu_layers 1 --model ~/models/codellama_13b.gguf --port 3000
aichat -I http://localhost:3000/v1/chat/completions --max_tokens 2048

Vader Yang added 2 commits September 5, 2023 11:30
…chat to support third party LLM services which using OPENAI-like API such as llama-cpp-python[server], LocalAI and etc.
@vaderyang
Copy link
Author

if this project is dead, should I branch from this and grow my own version?

@jacksongoode
Copy link
Contributor

I would support it @vaderyang !

@sigoden
Copy link
Owner

sigoden commented Oct 26, 2023

see #150

@sigoden sigoden closed this Oct 26, 2023
@vaderyang vaderyang deleted the configable branch October 29, 2023 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants