# LiteLLM Proxy Guide: Self-Hosted LLM Gateway for Teams

> Set up LiteLLM Proxy: config.yaml, virtual keys with budgets, Ollama-to-cloud fallback, Claude Code and OpenAI SDK clients, 2026 CVEs and hardening.
> Author: Roman Belov · Published: 2026-11-02 · Source: https://futurecraft.pro/blog/litellm-guide/

LiteLLM is an open-source, MIT-licensed Python library and proxy server that translate calls to 100+ LLM providers into the OpenAI format. You plug the Python SDK into one application's code. You run the Proxy (the docs call it the AI Gateway) as a shared gateway for a team: a virtual key per person, budgets, fallback between models, logs in one place. The [BerriAI/litellm](https://github.com/BerriAI/litellm) repository has about 62,000 stars, and the latest stable release as of November 2, 2026 is 1.106.6, from October 28.

Your first `config.yaml` fits in about twenty lines. The trouble starts when you hand the proxy to a team. Virtual keys and budgets silently stop working without Postgres. Some features in the docs need an Enterprise license. And the proxy itself holds the keys to every provider. In 2026 LiteLLM collected more than ten High and Critical CVEs, and in March two releases with malicious code sat on PyPI for 40 minutes.

This guide covers running LiteLLM Proxy, describing models in `config.yaml`, setting up fallback from a local model to a cloud one, issuing keys with budgets, connecting the OpenAI SDK, Claude Code and OpenCode, and operating the proxy given its vulnerability record. Fallback, proxy overhead, auth without a database and the Anthropic format were tested on LiteLLM 1.108.1 (released November 5) with Ollama 0.40.14 on a Mac with an Apple M4 Max. Docker, Postgres, Redis, the Admin UI and Enterprise features are described from the documentation.

## What LiteLLM is and when you need it

The name covers two products with different jobs.

| | LiteLLM SDK | LiteLLM Proxy |
|---|---|---|
| What it is | Python library | Separate HTTP server, port 4000 |
| Clients | Python code | Any language, OpenAI SDK, agents, curl |
| Features | `completion()` for 100+ providers, Router with retries and fallback, response cost, callbacks | Everything in the SDK plus virtual keys, budgets, rate limits, spend tracking, Admin UI, cache, guardrails, MCP Gateway |
| Storage | None | Postgres for keys and budgets, Redis for multiple instances |
| For | One application | A team, several services, agents |

Which one fits your situation:

| Situation | What to use |
|---|---|
| One Python app, two or three providers | SDK |
| One developer, one model | Neither: the provider's SDK is enough |
| Several services in different languages call LLMs | Proxy |
| You need per-person keys, limits and a spend report | Proxy + Postgres |
| A local model with a cloud backup | SDK Router or Proxy |
| You don't want to run a server | A cloud aggregator such as OpenRouter |

Where a self-hosted proxy sits in a provider architecture, next to in-code adapters and managed gateways, is covered in [Multi-Provider LLM Routing](https://futurecraft.pro/blog/multi-provider-llm-architecture/).

### LiteLLM vs OpenRouter and Portkey

| | LiteLLM | OpenRouter | Portkey Gateway |
|---|---|---|---|
| Model | Self-hosted SDK and Proxy, MIT + `enterprise/` directory | SaaS, one API and one bill | Open-source gateway (MIT, ~13,000 stars) plus hosted Portkey |
| Provider keys | Yours, stored on your server | An OpenRouter key or your own (BYOK) | Yours |
| Cost | Free software, you pay for infrastructure; Enterprise on request | 5.5% platform fee (Standard), 8% (Business); BYOK fee-free up to $25,000 a month per the pricing page, 5% above that | See the Portkey website |
| Data path | Through your host | Through OpenRouter's servers | Through your host (open-source version) |

OpenRouter prices come from its [pricing page](https://openrouter.ai/pricing) in early November 2026. The [OpenRouter guide](https://futurecraft.pro/blog/openrouter-guide/) covers the cloud option in detail. The two combine well: OpenRouter plugs into LiteLLM as one more provider.

## LiteLLM Proxy quick start

Install per the [quick start](https://docs.litellm.ai/docs/proxy/quick_start):

```bash
uv tool install 'litellm[proxy]==1.106.6'
# or
pip install 'litellm[proxy]==1.106.6'
```

Since 1.84.0 LiteLLM requires Python 3.10 or newer, and the upper bound on PyPI is `<3.15`. `uv tool install` picks a suitable interpreter on its own. Plain `pip` on an old Python, per the docs, silently installs the last compatible old version. If your system Python is 3.15 or newer, give uv the interpreter version explicitly. A one-off run looks like this:

```bash
uvx --python 3.12 --from 'litellm[proxy]==1.106.6' litellm --config config.yaml --host 127.0.0.1 --port 4000
```

Pin the version from the start. Releases ship several times a week, fixes go out to several minor lines at once, and versions 1.82.7 and 1.82.8 were malicious (details in the security section).

A minimal `config.yaml` with a local and a cloud model:

```yaml
model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
  - model_name: cloud-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY   # must start with sk-
```

The first request:

```bash
curl http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "local-qwen", "messages": [{"role": "user", "content": "ping"}]}'
```

An OpenAI SDK client changes only the address and the key:

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:4000", api_key="sk-...")  # master or virtual key
resp = client.chat.completions.create(
    model="cloud-sonnet",  # model_name from config.yaml
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)
```

The client only sees `model_name`. The config decides which provider sits behind the alias, so you can swap the model without touching application code.

### The SDK: LiteLLM without a server

If you don't need a gateway, the library makes the same call:

```python
from litellm import completion

resp = completion(
    model="ollama_chat/qwen3:8b",
    api_base="http://127.0.0.1:11434",
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)
```

For load balancing and fallback, the SDK has a `Router` class that takes the same `model_list` format as the proxy config.

## config.yaml: models, routing, secrets

A [config.yaml](https://docs.litellm.ai/docs/proxy/configs) has five sections:

| Section | What goes in it |
|---|---|
| `model_list` | Models: `model_name` (the alias clients use), `litellm_params` (provider, model, `api_base`, `api_key`, `rpm`, `tpm`), optional `model_info` |
| `router_settings` | Routing strategy, retries, cooldown, fallbacks |
| `litellm_settings` | Library behavior: fallbacks, timeouts, cache, callbacks, global budget |
| `general_settings` | Master key, database, server parameters |
| `environment_variables` | Environment variables set right in the config |

Secrets don't go into the file as plain text. The proxy resolves `os.environ/NAME` from the environment at startup.

Several entries with the same `model_name` form a group, and the proxy spreads requests across them:

```yaml
model_list:
  - model_name: fast
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 500
  - model_name: fast
    litellm_params:
      model: openai/<GPT-6 model>
      api_key: os.environ/OPENAI_API_KEY
      rpm: 300

router_settings:
  routing_strategy: simple-shuffle
```

Strategies from the [routing docs](https://docs.litellm.ai/docs/routing):

| Strategy | How it picks |
|---|---|
| `simple-shuffle` | Default, recommended for production; weighs by `rpm`/`tpm` or `weight`, picks at random without them |
| `least-busy` | The deployment with the fewest in-flight requests |
| `latency-based-routing` | The fastest one by recent measurements |
| `usage-based-routing-v2` | By current rpm/tpm usage through Redis; the docs advise against it in production |
| `cost-based-routing` | The cheapest deployment |

Model names in LiteLLM doc examples change with provider releases and don't always match current IDs. Take the real identifier from the provider's docs: for Anthropic in November 2026 that's `claude-opus-5-5`, `claude-sonnet-5-5` and `claude-haiku-5-5`.

### Ollama behind the proxy

LiteLLM supports two prefixes for [Ollama](https://docs.litellm.ai/docs/providers/ollama). `ollama_chat/` sends requests to `/api/chat`, and the docs recommend it. `ollama/` goes to `/api/generate`. `api_base` points at the Ollama server, `http://localhost:11434` by default.

```yaml
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
      keep_alive: 30m
    model_info:
      supports_function_calling: true
```

Set `supports_function_calling` when the model handles native tool calls but LiteLLM didn't detect it. Otherwise LiteLLM emulates tool calls through JSON mode. Which Ollama models actually handle tools, and why an agent needs a context of 64k or more, is covered in the [Ollama guide](https://futurecraft.pro/blog/ollama-guide/). Choosing between Ollama and LM Studio as the local backend is in a [separate comparison](https://futurecraft.pro/blog/lm-studio-vs-ollama/).

## Fallback, retries and timeouts

A fallback is a backup model the proxy switches a request to after the primary one fails. Parameters from the [reliability docs](https://docs.litellm.ai/docs/proxy/reliability):

| Parameter | What it does |
|---|---|
| `fallbacks: [{"a": ["b", "c"]}]` | If group `a` fails, tries `b`, then `c` |
| `context_window_fallbacks` | Switches when the context is exceeded; checking before sending needs `router_settings.enable_pre_call_checks: true` |
| `content_policy_fallbacks` | Switches when the provider refuses on content policy |
| `default_fallbacks` | Backup models for every group without its own rule |
| `num_retries` | Retries on the same group before moving to a fallback |
| `request_timeout` | Request timeout, in seconds |
| `allowed_fails`, `cooldown_time` | How many failures a deployment can have before it goes into cooldown, and for how long |

The docs show `fallbacks` under `litellm_settings`. The key also works under `router_settings`, which is how the test below sets it up. To turn off fallback for a single request, add `"disable_fallbacks": true` to the body.

The "local model is down, the cloud answers" scenario:

```yaml
model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
  - model_name: cloud-deepseek
    litellm_params:
      model: deepseek/deepseek-flash
      api_key: os.environ/DEEPSEEK_API_KEY

router_settings:
  num_retries: 0
  fallbacks:
    - local-qwen: ["cloud-deepseek"]
```

**Verified:** November 7, 2026, LiteLLM 1.108.1 via `uvx --python 3.12`, no Docker or Postgres, Ollama 0.40.14 with `qwen3:8b`, DeepSeek `deepseek-flash` as the backup model. A separate group with `api_base` pointing at a closed port stood in for a crashed Ollama, since it produces the same connection error. A request to it came back with the cloud model's answer, and the response headers read: `x-litellm-model-group: cloud-deepseek`, `x-litellm-attempted-fallbacks: 1`, `x-litellm-attempted-retries: 0`, `x-litellm-response-duration-ms: 1280.09`, `x-litellm-overhead-duration-ms: 234.18`.

These headers are how you know the answer came from the backup model. If you don't log them, the application gets a normal response and never learns that the primary model is down.

The same test turned up a detail about the backup model. `deepseek-flash` is a reasoning model, and with `max_tokens: 200` all 200 tokens went to `reasoning_tokens`, leaving the answer text empty. Give the backup model in the chain headroom in `max_tokens`, or put a non-reasoning model in the fallback.

How to design routing and degradation across the whole stack, not just one proxy, is in [Production LLM Stack](https://futurecraft.pro/blog/production-llm-stack/).

## Proxy overhead

The proxy adds one more network hop. To see what that costs, the test used a local model, where network latency to a provider can't hide LiteLLM's own delay.

**Verified:** November 7, 2026, LiteLLM 1.108.1, Ollama 0.40.14, warm `qwen3:8b`, requests with `max_tokens: 1`. 15 requests per path, the first 3 dropped, n=12.

| Path | Median, ms | p90, ms |
|---|---|---|
| Direct to Ollama (`/v1/chat/completions`) | 19.5 | 19.7 |
| Through LiteLLM Proxy | 26.8 | 26.9 |

That's about 7 ms per request. The `x-litellm-overhead-duration-ms` header showed 7.13, 6.77 and 5.59 ms on three separate requests. The first request after the proxy starts is slower: the fallback request above shows 234 ms of overhead, and that number includes the switch to the backup model. For cloud models, where a response takes seconds, 7 ms disappears in the noise. Concurrent load wasn't measured.

## Virtual keys, budgets and limits

A virtual key is an `sk-...` key that the proxy issues to a person, service or agent in place of a provider key. [Virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys) need two things: Postgres in `DATABASE_URL` and a master key that starts with `sk-`.

```yaml
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL
```

A key with access to one model group, a $10 budget for 30 days and a limit of 60 requests per minute:

```bash
curl http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["cloud-sonnet"], "max_budget": 10, "budget_duration": "30d",
       "rpm_limit": 60, "metadata": {"owner": "dev-1"}}'
```

You create a team with a shared budget through `/team/new`, and a key joins it through the `team_id` field:

```bash
curl http://127.0.0.1:4000/team/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"team_alias": "backend", "max_budget": 100, "budget_duration": "30d"}'
```

Budget levels from the [docs](https://docs.litellm.ai/docs/proxy/users):

| Level | How to set it |
|---|---|
| Key | `/key/generate`: `max_budget`, `budget_duration`, `rpm_limit`, `tpm_limit`, `max_parallel_requests` |
| User | `/user/new` |
| Team | `/team/new` |
| Team member | `/team/member_add` with `max_budget_in_team` |
| External customer | `/budget/new` + `/customer/new`, the `user` field in the request |
| Whole proxy | `litellm_settings.max_budget` and `budget_duration` |

`budget_duration` takes seconds, minutes, hours and days: `30s`, `30m`, `30h`, `30d`. A key over budget gets an error with `ExceededTokenBudget` or `ExceededBudget` in the message.

Details that are easy to miss:

- Without a database, budgets aren't enforced. The docs show an example: `max_budget` in the config, and the proxy keeps answering.
- Zero-cost models skip the budget check entirely. The docs describe this for models with explicit `input_cost_per_token: 0` and `output_cost_per_token: 0`, so a key with an exhausted budget can still move to a free model. LiteLLM's price map lists Ollama models at zero cost: spend on them doesn't grow, and a dollar budget won't limit them. Cap a local model with `rpm_limit` and `tpm_limit` instead.
- For a key with a `team_id`, the owner's personal budget doesn't apply. The team limits do.

**Verified:** November 7, 2026, LiteLLM 1.108.1 via uvx, no Postgres and no `prisma` package in the environment. A request without an `Authorization` header got 500 Internal Server Error, the wrong key `sk-wrong` got 400, and the correct master key got 200. The log showed `ModuleNotFoundError: No module named 'prisma'`: the auth error path calls into a database layer that isn't there.

The proxy did reject requests with no key, but with a 500 instead of a 401. Monitoring that counts unauthorized attempts by 401 won't see anything in this setup. Without Postgres the proxy is fine for a local test. For a team, the database is mandatory.

### budget_fallbacks and per-model budgets

Since version 1.92.0 a key has a [`budget_fallbacks`](https://docs.litellm.ai/docs/proxy/budget_fallbacks) field: when the key's budget for a specific model runs out, the request goes to a model from the chain instead of failing. The `budget_fallbacks` page carries no Enterprise label. But it only fires with `model_max_budget` (a key's budget per model), which the budget docs mark as Enterprise, and in the code `/key/generate` rejects it without a license, saying `model_max_budget` needs Enterprise. In practice, `budget_fallbacks` doesn't work without a license.

## Docker and production

A quick start with Postgres, per the [Docker quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start):

```bash
curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker/docker-compose.quickstart.yml
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' \
  "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
docker compose -f docker-compose.quickstart.yml up -d
```

Compose brings up the gateway on port 4000 and Postgres. Read the file before you run it and replace the image tag with a specific version. A single `docker run` with a config and no database gives you only the OpenAI-compatible API: no keys, no budgets, no spend tracking.

Images and tags per [Docker Image Security](https://docs.litellm.ai/docs/proxy/docker_image_security):

- `ghcr.io/berriai/litellm` is the main image, with `litellm-non_root` and `litellm-spend_logs` variants;
- `vX.Y.Z` tags are immutable, `main-stable` and `main-latest` are deprecated, and `latest` doesn't belong in production;
- since v1.83.0 every image in GHCR is signed with cosign, so verify the signature before you deploy:

```bash
cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<commit>/cosign.pub \
  ghcr.io/berriai/litellm:v1.106.6
```

What you need for what ([production](https://docs.litellm.ai/docs/proxy/prod), [deploy](https://docs.litellm.ai/docs/proxy/deploy)):

| Component | When you need it |
|---|---|
| Postgres | Virtual keys, teams, users, budgets, spend log, models added from the UI |
| Redis 7+ | More than one instance or worker: shared rate limits, router state, cache |
| `LITELLM_SALT_KEY` | Encrypts provider credentials in the database. Don't change it after adding models, or the saved keys become unreadable |
| `DISABLE_SCHEMA_UPDATE=true` | On proxy instances, while a separate job runs migrations |
| `/health/readiness`, `/health/liveliness` | Kubernetes probes |

The docs size it at 1 vCPU and 4 GiB per worker, and in Kubernetes at `--num_workers 1` per pod with 2+ replicas. They also suggest `proxy_batch_write_at: 60` (batched spend writes to the database) and `LITELLM_LOG=ERROR`. The `allow_requests_on_db_unavailable` flag lets requests through when the database is down, and the docs allow it only inside a VPC.

## Admin UI and observability

The Admin UI lives at `<proxy>/ui` and needs a database. You log in with `UI_USERNAME` (`admin` by default) and `UI_PASSWORD`. If no password is set, the master key works, and anyone who logs in that way gets proxy admin rights ([Admin UI](https://docs.litellm.ai/docs/proxy/ui)). From the UI you issue keys, add models without a restart, create teams, and view spend and logs.

Traces and metrics go out through callbacks:

- **Langfuse.** For Langfuse v3 and v4, the [LiteLLM docs](https://docs.litellm.ai/docs/observability/langfuse_integration) recommend the `langfuse_otel` preset via OpenTelemetry v2 over the older `langfuse` SDK callback. A step-by-step Langfuse setup, including the LiteLLM row, is in the [Langfuse tutorial](https://futurecraft.pro/blog/langfuse-step-by-step/), and which metrics and alerts to collect is in [LLM Observability in Production](https://futurecraft.pro/blog/llm-observability-langfuse/).
- **OpenTelemetry v2.** `litellm_settings.callbacks: ["otel"]` and `LITELLM_OTEL_V2=true`, with the collector address in `OTEL_EXPORTER`, `OTEL_ENDPOINT` and `OTEL_HEADERS`. Each request writes one trace: authentication, guardrails, the model call with `gen_ai.*` attributes, the database write. Prompt content stays out of the trace by default.

### Caching and guardrails

The [cache](https://docs.litellm.ai/docs/proxy/caching) is either exact (`local`, `disk`, `redis`, `s3`, `gcs`) or semantic (`redis-semantic`, `valkey-semantic`, `qdrant-semantic`). An in-memory cache lives inside one worker: a proxy with four workers has four separate caches, so from the second worker on you need Redis. The exact cache keys on a hash of the whole request, so any change to the conversation is a miss. The semantic cache, by the docs' own account, suits single prompts and fails badly on agent traffic.

Guardrails go in a `guardrails:` block with `pre_call`, `post_call`, `during_call` and `logging_only` modes. Built in are `litellm_content_filter`, Presidio for PII masking, and integrations with Lakera, Bedrock, Aporia and others ([guardrails](https://docs.litellm.ai/docs/proxy/guardrails/quick_start)). You turn on a specific guardrail per request with `"guardrails": ["name"]`.

## Connecting clients

### OpenAI SDK

Any OpenAI-format client works the same way: `base_url` pointing at the proxy, a virtual key in `api_key`, a `model_name` from the config in `model`. The example is in the quick start section.

### Claude Code

Per the [LiteLLM tutorial](https://docs.litellm.ai/docs/tutorials/claude_responses_api), Claude Code connects with two variables:

```bash
export ANTHROPIC_BASE_URL="http://127.0.0.1:4000"
export ANTHROPIC_AUTH_TOKEN="sk-..."   # virtual key or master key
claude --model cloud-sonnet            # model_name from config.yaml
```

The master key gives Claude Code access to every model on the proxy. A virtual key limits it to the models that key allows. If Bedrock sits behind the proxy, the tutorial recommends `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1`, because Bedrock doesn't accept every Claude Code beta header.

Two caveats from the [Claude Code docs](https://code.claude.com/docs/en/llm-gateway):

- Anthropic supports gateways with a compatible API format for Claude models, but it doesn't audit third-party gateways and doesn't support Claude Code on non-Claude models through any gateway. LiteLLM makes that technically possible, but quality and compatibility are on you.
- While a gateway credential variable is set, requests carry that key instead of your claude.ai subscription login. Subscription limits don't apply to them, and they're billed per token to whoever owns the key.

**Verified:** November 7, 2026, LiteLLM 1.108.1. `POST /v1/messages` with `x-api-key` and `anthropic-version: 2023-06-01` headers to the `local-qwen` group returned an Anthropic-format response: `type: message`, `stop_reason: end_turn`, Qwen3's reasoning in a separate `thinking` block, the answer in a `text` block, and usage of 16 input and 98 output tokens. Claude Code itself wasn't run in this test.

So an Anthropic-format client can talk to a local model through the proxy. Whether an 8B model can handle Claude Code's agent loop is a question about the model, and in OpenCode tests local models of that size were unstable.

### OpenCode

The OpenCode docs have no dedicated LiteLLM entry, so the generic OpenAI-compatible provider mechanism does the job ([providers](https://opencode.ai/docs/providers/)). Here's an example for OpenCode 1, in the same format as the Ollama setup in the [OpenCode guide](https://futurecraft.pro/blog/opencode-guide/):

```json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM",
      "options": {
        "baseURL": "http://127.0.0.1:4000/v1",
        "apiKey": "{env:LITELLM_API_KEY}"
      },
      "models": {
        "cloud-sonnet": { "name": "Sonnet 5.5 via LiteLLM" }
      }
    }
  }
}
```

The key in `models` must match the `model_name` in the proxy config. OpenCode 2 uses a different schema: a `providers` block with `package` and `settings.baseURL` fields. The proxy address goes in the same place.

### Local models

Ollama connects to the proxy as a regular provider (see the `config.yaml` section). Clients then use one address and one key for both the local and the cloud model, and fallback switches between them without the application doing anything. An overview of ways to run models locally is in the [local LLM guide](https://futurecraft.pro/blog/local-llm-guide/).

## What's paid: MIT vs Enterprise

The [license](https://github.com/BerriAI/litellm/blob/main/LICENSE) splits the repository in two. Everything outside the `enterprise/` directory is MIT. You can copy and modify code in `enterprise/` for development and testing, but in production it runs only with an active BerriAI subscription for the right number of seats, and you can't redistribute it. Enterprise is the same image: the `LITELLM_LICENSE` key turns the features on after a full restart.

| Free (MIT) | Needs an Enterprise key |
|---|---|
| Python SDK, Router, fallback | SSO and SCIM (SSO is free for up to 5 users) |
| Virtual keys, budgets and rate limits per key, user and team | JWT auth, assigning team admins |
| Spend tracking, Admin UI | Audit logs with retention, RBAC (organizations, teams, roles) |
| Load balancing and routing | IP allowlist, key rotation, secret managers |
| Guardrails framework, custom guardrails, Presidio | Guardrail callbacks `llmguard_moderations`, `llamaguard_moderations`, `hide_secrets`, `openai_moderations`, `google_text_moderation`, `lakera_prompt_injection`, `aporia_prompt_injection` |
| MCP gateway | Tag budgets, per-model budgets on a key (`model_max_budget`), temporary budget increases |
| Cache, logging to Langfuse and OTel | Soft-budget email alerts, spend reports, per-team logging, export to GCS and Azure Blob, guardrails per key and team, request and response size limits |

The list comes from the [Enterprise page](https://docs.litellm.ai/docs/enterprise). Pricing depends on usage volume and goes through sales. There's a free 30-day trial, and you can buy through AWS and Azure Marketplace or by invoice.

## LiteLLM security: incidents and CVEs in 2026

The proxy holds the master key, every provider key and the credentials in its database. A compromised proxy leaks all of them at once. That's why the 2026 vulnerability record matters for any self-hosted deployment.

**PyPI compromise, March 24, 2026.** Versions 1.82.7 and 1.82.8 contained malicious code that harvested credentials, and 1.82.8 also shipped a `litellm_init.pth` file, which Python runs at interpreter startup. Per the [team's report](https://docs.litellm.ai/blog/security-update-march-2026), the packages were available from 10:39 UTC for about 40 minutes, and pip installs between 10:39 and 16:00 UTC count as affected. The official proxy Docker image wasn't affected. The team names a Trivy dependency in CI as the source and brought in Mandiant to investigate. Clean versions are 1.82.6 and earlier, and 1.83.0 and later.

Major CVEs and advisories from [NVD](https://nvd.nist.gov/) and [GitHub Security Advisories](https://github.com/BerriAI/litellm/security/advisories), with CVSS scores from GitHub as the CNA:

| ID | Published | CVSS | What | Fixed in |
|---|---|---|---|---|
| CVE-2026-35030 | Apr 6 | 9.4 | Auth bypass with JWT auth enabled (OIDC userinfo cache key collision); off by default | 1.83.0 |
| CVE-2026-35029 | Apr 6 | 8.7 | `/config/update` didn't check for the admin role | 1.83.0 |
| CVE-2026-42208 | May 8 | 9.3 | SQL injection in API key verification: an unauthenticated request to any LLM route reads and modifies the proxy database (1.81.16 to 1.83.6) | 1.83.7 |
| CVE-2026-42203 | May 8 | 8.6 | SSTI in `POST /prompts/test`: code execution with any valid key | 1.83.7 |
| CVE-2026-42271 | May 8 | 8.7 | Command execution through MCP test endpoints with stdio transport | 1.83.7 |
| CVE-2026-49468 | Jun 22 | 9.5 | Auth bypass via the Host header: access to management routes without a key | 1.84.0 |
| CVE-2026-59822 | Jul 8 | 8.8 | Auth bypass in MCP Streamable HTTP via OAuth2 passthrough | 1.84.0 |
| CVE-2026-89032 | Sep 25 | 8.7 | Semantic cache served responses to other tenants (`/v1/responses`, `/bedrock/*`) | 1.101.0-rc.1 |
| CVE-2026-93355 | Sep 28 | 7.6 | JWT auth: log in as any user by email without an `email_verified` check | Not stated in the NVD record |
| CVE-2026-114612 | Oct 22 | 9.9 | Shared crypto context for secrets and sessions: an `internal_user` forges an admin token and runs commands via MCP stdio; in the default config since 1.91.0 | 1.100.4, 1.101.3, 1.102.2, 1.103.1 |
| GHSA-w4c7-34wj-hqm8 | Oct 27 | 10.0 | Enterprise only: an unauthenticated request makes the proxy read a local file (for example `/proc/self/environ` with the master key and provider keys) and send it to the attacker's address | 1.103.10, 1.104.8, 1.105.6, 1.106.5 |

Dates are NVD publication dates (for the advisory without a CVE, the GitHub date). The GitHub advisory usually comes out a few days or weeks before the NVD entry. Beyond the table, 2026 also brought SSRF and provider credential leaks through `api_base` and `user_config` in the request body (CVE-2026-84377, CVE-2026-59823, Moderate), code execution through the custom guardrail test (CVE-2026-40217) and local file reads through `vertex_ai_credentials` (GHSA-g5ff-637f-6q2m).

Nearly all the serious holes sit in authentication, management, test and MCP endpoints, JWT and SSO, not in request routing. The latest critical one was in Enterprise code: the license widens the attack surface.

A checklist for self-hosting:

1. Pin the version: `litellm[proxy]==X.Y.Z` or an image tag `vX.Y.Z`, ideally a digest, never `latest`.
2. Verify the image signature with cosign (images are signed since v1.83.0).
3. For pip and uv: make sure you don't have 1.82.7 or 1.82.8 (`pip show litellm`), and look for `litellm_init.pth` in site-packages.
4. Keep the proxy off the internet. Put it behind a VPN, mTLS, an authenticating ingress or an IP allowlist. Lock down `/ui`, `/key/*`, `/team/*`, `/config/*` and the MCP routes separately.
5. Restrict the proxy's outbound traffic to provider addresses. With GHSA-w4c7, that's exactly what cuts off exfiltration even on a vulnerable version.
6. Don't turn on the MCP gateway, JWT or SSO unless you need them.
7. Set a separate `LITELLM_SALT_KEY`, and keep the master key in a secret manager where you can, rather than in the process environment.
8. Subscribe to the repository's advisories and upgrade on them: fixes ship to several supported lines at once.

MCP endpoint risks aren't unique to LiteLLM. The general threat model is in [MCP Servers Explained](https://futurecraft.pro/blog/mcp-servers-explained/).

## LiteLLM limitations and gotchas

### Postgres is mandatory for a team

Without a database there are no keys, budgets, Admin UI or spend log, and limits in the config are silently ignored. In the no-database test, a request without a key got a 500 instead of a 401. For a team, plan Postgres from day one.

### One more hop, one more point of failure

About 7 ms per request on a local machine is small, but the proxy becomes a single point of failure for every client. Production needs 2+ replicas, Redis and health probes, which means one more system to run.

### A new version every week

Releases ship several times a week: on October 27 the same fix went out in four lines at once (1.103.10, 1.104.8, 1.105.6, 1.106.5). Behavior changes between versions. The tests in this article ran on 1.108.1, released three days after the article's date. Read the changelog and advisories before you upgrade in production.

### Placeholders in the docs

LiteLLM doc examples use model names that don't always match providers' current IDs. Check the provider's docs before you copy one into a config.

### Semantic cache and agents

A semantic cache returns the closest answer by meaning. For agent traffic with tool calls that's a source of errors, and in CVE-2026-89032 it also served answers across tenants.

### When you don't need LiteLLM

One model and one service get by with the provider's SDK. A single developer with several providers is fine with the LiteLLM SDK and no server. If nobody can run a gateway, a cloud aggregator saves time, though it costs more in fees.

## Where to start

1. Install a pinned version, `uv tool install 'litellm[proxy]==1.106.6'`, and check `litellm --version`.
2. Write a `config.yaml` with one local and one cloud model and a master key from an environment variable, then start the proxy on `127.0.0.1`.
3. Test a request with curl and the OpenAI SDK, then set up `fallbacks` and watch the `x-litellm-attempted-fallbacks` header while the primary model is unreachable.
4. For a team, bring up Postgres (compose quickstart), set `LITELLM_SALT_KEY`, and issue keys with `max_budget` and `rpm_limit`.
5. Connect clients: the OpenAI SDK, Claude Code through `ANTHROPIC_BASE_URL`, OpenCode through an OpenAI-compatible provider.
6. Work through the security checklist: private network, restricted egress, image signature, advisory subscription.