LiteLLM Proxy in Kubernetes and questions

[chrbrnracn]

I'm trying to run Ollama and LiteLLM in Kubernetes. Ollama is running in a separate deployment and does not expose any external ports, only internally as a service. For now it's planned to keep Ollama separate by intention.
Now I want to run LiteLLM Proxy as OpenAI API compatible frontend for use with IDE code assisting plugins (eg. continue.dev).

• Is that possible with LiteLLM Proxy?

I'm slightly confused by the documentation.

In Docker, Deploying LiteLLM Proxy there are examples how to deploy with database.

• A different image is used compared to the plain docker run command.
  I couldn't find any documentation on the differences. From the code it looks like litellm (Dockerfile?) pulls in some additional config file while litellm-database (Dockerfile.database?) does not by default (CMD)
• The example configures a DATABASE_URL. The example looks like an external PostgreSQL
  • What is the database used for? What is the difference to the litellm image?
  • Is a pre-deployment of a data model required or is that done implicitly by the application? Which permissions are required in the database?

"Admin UI"
The docs point to Google SSO, Microsoft SSO or a hard coded username password.

• Is there some kind of internal user management and access control (RBAC?) for additional users?
• Can other authentication mechanisms be used? eg. KeyCloak

Thanks in advance for any help.

[chrbrnracn]

Replying to myself.

A different image is used compared to the plain docker run command.
I couldn't find any documentation on the differences. From the code it looks like litellm (Dockerfile?) pulls in some additional config file while litellm-database (Dockerfile.database?) does not by default (CMD)

Not sure why it's implemented this way. The litellm command understands the config parameter and uses the config file when one is provided.

The example configures a DATABASE_URL. The example looks like an external PostgreSQL

• What is the database used for? What is the difference to the litellm image?
• Is a pre-deployment of a data model required or is that done implicitly by the application? Which permissions are required in the database?

According to the data model I found in the code, the DB is at least used to store API keys and some user information.
Once the DB connection is established, the data model is deployed automatically.
I assume the difference between the Docker images is, that one tries to connect to a database, the one does not. But that needs to be confirmed from someone who knows the code.

A hint for everyone who reads this after me and wants to implement it in Kubernetes. I assume you want to use Kubernetes secrets for the DB password.
There is (currently) no option to provide the password as a file. My workaround is, to set a custom entrypoint (command) which prepares and exports the DATABASE_URL variable before calling litellm.

[ishaan-jaff]

Hi @chrbrnracn thanks for this - can we hop on a call to better understand your problem ? I'm unable to repro this on my side

Sharing my calendly for your convenience: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

[chrbrnracn]

@krrishdholakia I want to have an API key I can use with https://docs.litellm.ai/docs/proxy/virtual_keys

The docs say # [OPTIONAL] Only use this if you to require all calls to contain this key (Authorization: Bearer sk-1234)
Does this mean all calls to the proxy or all calls to the configured backend?
Also I don't want all calls to require this key but I want to treat this as an "admin" key which is only used to create API keys for regular users.

If this is the key I want, is it possible to define this as environment variable? I want to use Kubernetes secrets as source of this information.

[ishaan-jaff]

@chrbrnracn

but I want to treat this as an "admin" key which is only used to create API keys for regular users.

That is the purpose of the master key, it's the admin key

is it possible to define this as environment variable?

Yes, you can set LITELLM_MASTER_KEY in your env. Can you lmk if that works ?

[ishaan-jaff]

Updated our docs based on your feedback @chrbrnracn. Would love to give you a shoutout in our daily changelog what's your linkedin / twitter ?
7d67096

[chrbrnracn]

You can use my Github handle. The LinkedIn handle looks quite ugly. 😉

[Bruda27]

Hello!

Using VS code dev containers, I've created a devcontainer.json file that allowed Anaconda and Python 3 Environment, Im able to containerize and activate any environment like litellm --model ollama/mistral or any open source library for enabling next generation LLM applications with multiagent, like autogen studio, of course, proxy server was enabled only to run that model, running ollama mistral doesn't require proxy configuration, so in summary, check the devcontainer.json file to ensure environment variable dependencies.

Im just trying to provide my POV, hope helps on something :)