GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
-
Updated
Dec 13, 2024 - C++
#
llm-inference
Here are 668 public repositories matching this topic...
A programming framework for agentic AI 🤖 (PyPi: autogen-agentchat)
chat chatbot gpt chat-application agent-based-framework agent-oriented-programming gpt-4 chatgpt llmops gpt-35-turbo llm-agent llm-inference agentic llm-framework agentic-agi
-
Updated
Dec 13, 2024 - Python
本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）
-
Updated
Dec 11, 2024 - HTML
20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.
-
Updated
Dec 9, 2024 - Python
Run any open-source LLMs, such as Llama, Mistral, as OpenAI compatible API endpoint in the cloud.
llama mistral fine-tuning mlops bentoml vicuna llm model-inference llmops llm-serving llm-inference open-source-llm llama2 openllm llm-ops llama3-1 llama3-2 llama3-2-vision
-
Updated
Dec 10, 2024 - Python
Official inference library for Mistral models
-
Updated
Nov 12, 2024 - Jupyter Notebook
High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
-
Updated
Sep 6, 2024 - C++
OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
nlp natural-language-processing ai computer-vision deep-learning transformers inference speech-recognition yolo recommendation-system performance-boost good-first-issue openvino diffusion-models stable-diffusion generative-ai llm-inference optimize-ai deploy-ai
-
Updated
Dec 13, 2024 - C++
The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!
python machine-learning deep-learning model-serving multimodal mlops ml-engineering ai-inference llm generative-ai llmops llm-serving model-inference-service llm-inference inference-platform
-
Updated
Dec 13, 2024 - Python
Superduper: Build end-to-end AI applications and agent workflows on your existing data infrastructure and preferred tools - without migrating your data.
python data database ai mongodb chatbot ml transformers inference torch pytorch pretrained-models semantic-search rag mlops distributed-ml vector-search llmops llm-serving llm-inference
-
Updated
Dec 12, 2024 - Python
LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
llama cuda-kernels deepspeed llm fastertransformer llm-inference turbomind internlm llama2 codellama llama3
-
Updated
Dec 13, 2024 - Python
Standardized Serverless ML Inference Platform on Kubernetes
kubernetes machine-learning tensorflow sklearn pytorch artificial-intelligence xgboost k8s service-mesh hacktoberfest istio model-serving kubeflow mlops knative model-interpretability kserve genai llm-inference
-
Updated
Dec 12, 2024 - Python
Sparsity-aware deep learning inference runtime for CPUs
nlp performance computer-vision inference machinelearning pruning object-detection pretrained-models quantization cpus onnx sparsification llm-inference deepsparse
-
Updated
Jul 19, 2024 - Python
📖A curated list of Awesome LLM/VLM Inference Papers with codes, such as FlashAttention, PagedAttention, Parallelism, etc. 🎉🎉
sora llm llms vllm llm-inference awesome-llm flash-attention flash-attention-2 tensorrt-llm paged-attention deepseek open-sora flash-attention-3
-
Updated
Dec 9, 2024
Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.
microservice gpu-acceleration nemo tensorrt rag triton-inference-server large-language-models llm llm-inference retrieval-augmented-generation
-
Updated
Dec 1, 2024 - Python
Code examples and resources for DBRX, a large language model developed by Databricks
-
Updated
May 1, 2024 - Python
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads
-
Updated
Jun 25, 2024 - Jupyter Notebook
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
-
Updated
Dec 13, 2024 - Python
⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡
retrieval chatbot rag habana large-language-model chatpdf llm-inference 4-bits speculative-decoding llm-cpu streamingllm intel-optimized-llamacpp neural-chat neural-chat-7b autoround gaudi3
-
Updated
Oct 8, 2024 - Python
Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.
-
Updated
Mar 22, 2024 - Jupyter Notebook
Improve this page
Add a description, image, and links to the llm-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-inference topic, visit your repo's landing page and select "manage topics."