Skip to content

Latest commit

 

History

History
140 lines (108 loc) · 4.97 KB

README.md

File metadata and controls

140 lines (108 loc) · 4.97 KB
Raw

Welcome to Paka

paka.png

Get your LLM applications to the cloud with ease. Paka handles failure recovery, autoscaling, and monitoring, freeing you to concentrate on crafting your applications.

🚀 Bring LLM models to the cloud in minutes

💰 Cut 50% cost with spot instances, backed by on-demand instances for reliable service quality.

Model Parameters Quantization GPU On-Demand Spot AWS Node (us-west-2)
Llama 3 70B BF16 A10G x 8 $16.2880 $4.8169 g5.48xlarge
Llama 3 70B GPTQ 4bit T4 x 4 $3.9120 $1.6790 g4dn.12xlarge
Llama 3 8B BF16 L4 x 1 $0.8048 $0.1100 g6.xlarge
Llama 2 7B GPTQ 4bit T4 x 1 $0.526 $0.2584 g4dn.xlarge
Mistral 7B BF16 T4 x 1 $0.526 $0.2584 g4dn.xlarge
Phi3 Mini 3.8B BF16 T4 x 1 $0.526 $0.2584 g4dn.xlarge

Note: Prices are based on us-west-2 region and are in USD per hour. Spot prices change frequently. See Launch Templates for more details.

🏃 Effortlessly Launch RAG Applications

You only need to take care of the application code. Build the RAG application with your favorite languages (python, TS) and frameworks (Langchain, LlamaIndex) and let Paka handles the rest.

Support for Vector Store

  • A fast vector store (qdrant) for storing embeddings.
  • Tunable for performance and cost.

Serverless Deployment

  • Deploy your application as a serverless container.
  • Autoscaling and monitoring built-in.

📈 Monitoring

Paka comes with built-in support for monitoring and tracing. Metrics are collected via Prometheus. Users can also enable Prometheus Alertmanager for alerting.

⚙️ Architecture

📜 Roadmap

  • (Multi-cloud) AWS support
  • (Backend) vLLM
  • (Backend) llama.cpp
  • (Platform) Windows support
  • (Accelerator) Nvidia GPU support
  • (Multi-cloud) GCP support
  • (Backend) TGI
  • (Accelerator) AMD GPU support
  • (Accelerator) Inferentia support

🎬 Getting Started

Dependencies

  • docker daemon and CLI
  • AWS CLI
# Ensure your AWS credentials are correctly configured.
aws configure

Install Paka

pip install paka

Provisioning the cluster

Create a cluster.yaml file with the following content:

version: "1.2"
aws:
  cluster:
    name: my-awesome-cluster
    region: us-west-2
    namespace: default
    nodeType: t3a.medium
    minNodes: 2
    maxNodes: 4
  prometheus:
    enabled: true
  modelGroups:
    - name: llama2-7b-chat
      nodeType: g4dn.xlarge
      isPublic: true
      minInstances: 1
      maxInstances: 1
      name: llama3-70b-instruct
      runtime:
        image: vllm/vllm-openai:v0.4.2
      model:
        hfRepoId: TheBloke/Llama-2-7B-Chat-GPTQ
        useModelStore: false
      gpu:
        enabled: true
        diskSize: 50

Bring up the cluster with the following command:

paka cluster up -f cluster.yaml

Code up the application

Use your favorite language and framework to build the application. Here is an example of a Python application using Langchain:

invoice_extraction

With Paka, you can effortlessly build your source code and deploy it as a serverless function, no Dockerfile needed. Just ensure the following:

  • Procfile: Defines the entrypoint for your application. See Procfile.
  • .cnignore file: Excludes any files that shouldn't be included in the build. See .cnignore.
  • runtime.txt: Pins the version of the runtime your application uses. See runtime.txt.
  • requirements.txt or package.json: Lists all necessary packages for your application.

Deploy the App

paka function deploy --name invoice-extraction --source . --entrypoint serve

📖 Documentation

Contributing

  • code changes
  • make check-all
  • Open a PR