LiteLLM Proxy Guide: Self-Hosted LLM Gateway for Teams

LiteLLM is an open-source, MIT-licensed Python library and proxy server that translate calls to 100+ LLM providers into the OpenAI format. You plug the Python SDK into one application’s code. You run the Proxy (the docs call it the AI Gateway) as a shared gateway for a team: a virtual key per person, budgets, fallback between models, logs in one place. The BerriAI/litellm repository has about 62,000 stars, and the latest stable release as of November 2, 2026 is 1.106.6, from October 28.

Your first config.yaml fits in about twenty lines. The trouble starts when you hand the proxy to a team. Virtual keys and budgets silently stop working without Postgres. Some features in the docs need an Enterprise license. And the proxy itself holds the keys to every provider. In 2026 LiteLLM collected more than ten High and Critical CVEs, and in March two releases with malicious code sat on PyPI for 40 minutes.

This guide covers running LiteLLM Proxy, describing models in config.yaml, setting up fallback from a local model to a cloud one, issuing keys with budgets, connecting the OpenAI SDK, Claude Code and OpenCode, and operating the proxy given its vulnerability record. Fallback, proxy overhead, auth without a database and the Anthropic format were tested on LiteLLM 1.108.1 (released November 5) with Ollama 0.40.14 on a Mac with an Apple M4 Max. Docker, Postgres, Redis, the Admin UI and Enterprise features are described from the documentation.

What LiteLLM is and when you need it

The name covers two products with different jobs.

	LiteLLM SDK	LiteLLM Proxy
What it is	Python library	Separate HTTP server, port 4000
Clients	Python code	Any language, OpenAI SDK, agents, curl
Features	completion() for 100+ providers, Router with retries and fallback, response cost, callbacks	Everything in the SDK plus virtual keys, budgets, rate limits, spend tracking, Admin UI, cache, guardrails, MCP Gateway
Storage	None	Postgres for keys and budgets, Redis for multiple instances
For	One application	A team, several services, agents

Which one fits your situation:

Situation	What to use
One Python app, two or three providers	SDK
One developer, one model	Neither: the provider’s SDK is enough
Several services in different languages call LLMs	Proxy
You need per-person keys, limits and a spend report	Proxy + Postgres
A local model with a cloud backup	SDK Router or Proxy
You don’t want to run a server	A cloud aggregator such as OpenRouter

Where a self-hosted proxy sits in a provider architecture, next to in-code adapters and managed gateways, is covered in Multi-Provider LLM Routing.

LiteLLM vs OpenRouter and Portkey

	LiteLLM	OpenRouter	Portkey Gateway
Model	Self-hosted SDK and Proxy, MIT + enterprise/ directory	SaaS, one API and one bill	Open-source gateway (MIT, ~13,000 stars) plus hosted Portkey
Provider keys	Yours, stored on your server	An OpenRouter key or your own (BYOK)	Yours
Cost	Free software, you pay for infrastructure; Enterprise on request	5.5% platform fee (Standard), 8% (Business); BYOK fee-free up to $25,000 a month per the pricing page, 5% above that	See the Portkey website
Data path	Through your host	Through OpenRouter’s servers	Through your host (open-source version)

OpenRouter prices come from its pricing page in early November 2026. The OpenRouter guide covers the cloud option in detail. The two combine well: OpenRouter plugs into LiteLLM as one more provider.

LiteLLM Proxy quick start

Install per the quick start:

uv tool install 'litellm[proxy]==1.106.6'
# or
pip install 'litellm[proxy]==1.106.6'

Since 1.84.0 LiteLLM requires Python 3.10 or newer, and the upper bound on PyPI is <3.15. uv tool install picks a suitable interpreter on its own. Plain pip on an old Python, per the docs, silently installs the last compatible old version. If your system Python is 3.15 or newer, give uv the interpreter version explicitly. A one-off run looks like this:

uvx --python 3.12 --from 'litellm[proxy]==1.106.6' litellm --config config.yaml --host 127.0.0.1 --port 4000

Pin the version from the start. Releases ship several times a week, fixes go out to several minor lines at once, and versions 1.82.7 and 1.82.8 were malicious (details in the security section).

A minimal config.yaml with a local and a cloud model:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
  - model_name: cloud-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY   # must start with sk-

The first request:

curl http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "local-qwen", "messages": [{"role": "user", "content": "ping"}]}'

An OpenAI SDK client changes only the address and the key:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:4000", api_key="sk-...")  # master or virtual key
resp = client.chat.completions.create(
    model="cloud-sonnet",  # model_name from config.yaml
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)

The client only sees model_name. The config decides which provider sits behind the alias, so you can swap the model without touching application code.

The SDK: LiteLLM without a server

If you don’t need a gateway, the library makes the same call:

from litellm import completion

resp = completion(
    model="ollama_chat/qwen3:8b",
    api_base="http://127.0.0.1:11434",
    messages=[{"role": "user", "content": "ping"}],
)
print(resp.choices[0].message.content)

For load balancing and fallback, the SDK has a Router class that takes the same model_list format as the proxy config.

config.yaml: models, routing, secrets

A config.yaml has five sections:

Section	What goes in it
model_list	Models: model_name (the alias clients use), litellm_params (provider, model, api_base, api_key, rpm, tpm), optional model_info
router_settings	Routing strategy, retries, cooldown, fallbacks
litellm_settings	Library behavior: fallbacks, timeouts, cache, callbacks, global budget
general_settings	Master key, database, server parameters
environment_variables	Environment variables set right in the config

Secrets don’t go into the file as plain text. The proxy resolves os.environ/NAME from the environment at startup.

Several entries with the same model_name form a group, and the proxy spreads requests across them:

model_list:
  - model_name: fast
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 500
  - model_name: fast
    litellm_params:
      model: openai/<GPT-6 model>
      api_key: os.environ/OPENAI_API_KEY
      rpm: 300

router_settings:
  routing_strategy: simple-shuffle

Strategies from the routing docs:

Strategy	How it picks
simple-shuffle	Default, recommended for production; weighs by rpm/tpm or weight, picks at random without them
least-busy	The deployment with the fewest in-flight requests
latency-based-routing	The fastest one by recent measurements
usage-based-routing-v2	By current rpm/tpm usage through Redis; the docs advise against it in production
cost-based-routing	The cheapest deployment

Model names in LiteLLM doc examples change with provider releases and don’t always match current IDs. Take the real identifier from the provider’s docs: for Anthropic in November 2026 that’s claude-opus-5-5, claude-sonnet-5-5 and claude-haiku-5-5.

Ollama behind the proxy

LiteLLM supports two prefixes for Ollama. ollama_chat/ sends requests to /api/chat, and the docs recommend it. ollama/ goes to /api/generate. api_base points at the Ollama server, http://localhost:11434 by default.

  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
      keep_alive: 30m
    model_info:
      supports_function_calling: true

Set supports_function_calling when the model handles native tool calls but LiteLLM didn’t detect it. Otherwise LiteLLM emulates tool calls through JSON mode. Which Ollama models actually handle tools, and why an agent needs a context of 64k or more, is covered in the Ollama guide. Choosing between Ollama and LM Studio as the local backend is in a separate comparison.

Fallback, retries and timeouts

A fallback is a backup model the proxy switches a request to after the primary one fails. Parameters from the reliability docs:

Parameter	What it does
fallbacks: [{"a": ["b", "c"]}]	If group a fails, tries b, then c
context_window_fallbacks	Switches when the context is exceeded; checking before sending needs router_settings.enable_pre_call_checks: true
content_policy_fallbacks	Switches when the provider refuses on content policy
default_fallbacks	Backup models for every group without its own rule
num_retries	Retries on the same group before moving to a fallback
request_timeout	Request timeout, in seconds
allowed_fails, cooldown_time	How many failures a deployment can have before it goes into cooldown, and for how long

The docs show fallbacks under litellm_settings. The key also works under router_settings, which is how the test below sets it up. To turn off fallback for a single request, add "disable_fallbacks": true to the body.

The “local model is down, the cloud answers” scenario:

model_list:
  - model_name: local-qwen
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://127.0.0.1:11434
  - model_name: cloud-deepseek
    litellm_params:
      model: deepseek/deepseek-flash
      api_key: os.environ/DEEPSEEK_API_KEY

router_settings:
  num_retries: 0
  fallbacks:
    - local-qwen: ["cloud-deepseek"]

Verified: November 7, 2026, LiteLLM 1.108.1 via uvx --python 3.12, no Docker or Postgres, Ollama 0.40.14 with qwen3:8b, DeepSeek deepseek-flash as the backup model. A separate group with api_base pointing at a closed port stood in for a crashed Ollama, since it produces the same connection error. A request to it came back with the cloud model’s answer, and the response headers read: x-litellm-model-group: cloud-deepseek, x-litellm-attempted-fallbacks: 1, x-litellm-attempted-retries: 0, x-litellm-response-duration-ms: 1280.09, x-litellm-overhead-duration-ms: 234.18.

These headers are how you know the answer came from the backup model. If you don’t log them, the application gets a normal response and never learns that the primary model is down.

The same test turned up a detail about the backup model. deepseek-flash is a reasoning model, and with max_tokens: 200 all 200 tokens went to reasoning_tokens, leaving the answer text empty. Give the backup model in the chain headroom in max_tokens, or put a non-reasoning model in the fallback.

How to design routing and degradation across the whole stack, not just one proxy, is in Production LLM Stack.

Proxy overhead

The proxy adds one more network hop. To see what that costs, the test used a local model, where network latency to a provider can’t hide LiteLLM’s own delay.

Verified: November 7, 2026, LiteLLM 1.108.1, Ollama 0.40.14, warm qwen3:8b, requests with max_tokens: 1. 15 requests per path, the first 3 dropped, n=12.

Path	Median, ms	p90, ms
Direct to Ollama (/v1/chat/completions)	19.5	19.7
Through LiteLLM Proxy	26.8	26.9

That’s about 7 ms per request. The x-litellm-overhead-duration-ms header showed 7.13, 6.77 and 5.59 ms on three separate requests. The first request after the proxy starts is slower: the fallback request above shows 234 ms of overhead, and that number includes the switch to the backup model. For cloud models, where a response takes seconds, 7 ms disappears in the noise. Concurrent load wasn’t measured.

Virtual keys, budgets and limits

A virtual key is an sk-... key that the proxy issues to a person, service or agent in place of a provider key. Virtual keys need two things: Postgres in DATABASE_URL and a master key that starts with sk-.

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

A key with access to one model group, a $10 budget for 30 days and a limit of 60 requests per minute:

curl http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"models": ["cloud-sonnet"], "max_budget": 10, "budget_duration": "30d",
       "rpm_limit": 60, "metadata": {"owner": "dev-1"}}'

You create a team with a shared budget through /team/new, and a key joins it through the team_id field:

curl http://127.0.0.1:4000/team/new \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{"team_alias": "backend", "max_budget": 100, "budget_duration": "30d"}'

Budget levels from the docs:

Level	How to set it
Key	/key/generate: max_budget, budget_duration, rpm_limit, tpm_limit, max_parallel_requests
User	/user/new
Team	/team/new
Team member	/team/member_add with max_budget_in_team
External customer	/budget/new + /customer/new, the user field in the request
Whole proxy	litellm_settings.max_budget and budget_duration

budget_duration takes seconds, minutes, hours and days: 30s, 30m, 30h, 30d. A key over budget gets an error with ExceededTokenBudget or ExceededBudget in the message.

Details that are easy to miss:

• Without a database, budgets aren’t enforced. The docs show an example: max_budget in the config, and the proxy keeps answering.
• Zero-cost models skip the budget check entirely. The docs describe this for models with explicit input_cost_per_token: 0 and output_cost_per_token: 0, so a key with an exhausted budget can still move to a free model. LiteLLM’s price map lists Ollama models at zero cost: spend on them doesn’t grow, and a dollar budget won’t limit them. Cap a local model with rpm_limit and tpm_limit instead.
• For a key with a team_id, the owner’s personal budget doesn’t apply. The team limits do.

Verified: November 7, 2026, LiteLLM 1.108.1 via uvx, no Postgres and no prisma package in the environment. A request without an Authorization header got 500 Internal Server Error, the wrong key sk-wrong got 400, and the correct master key got 200. The log showed ModuleNotFoundError: No module named 'prisma': the auth error path calls into a database layer that isn’t there.

The proxy did reject requests with no key, but with a 500 instead of a 401. Monitoring that counts unauthorized attempts by 401 won’t see anything in this setup. Without Postgres the proxy is fine for a local test. For a team, the database is mandatory.

budget_fallbacks and per-model budgets

Since version 1.92.0 a key has a budget_fallbacks field: when the key’s budget for a specific model runs out, the request goes to a model from the chain instead of failing. The budget_fallbacks page carries no Enterprise label. But it only fires with model_max_budget (a key’s budget per model), which the budget docs mark as Enterprise, and in the code /key/generate rejects it without a license, saying model_max_budget needs Enterprise. In practice, budget_fallbacks doesn’t work without a license.

Docker and production

A quick start with Postgres, per the Docker quickstart:

curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker/docker-compose.quickstart.yml
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' \
  "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
docker compose -f docker-compose.quickstart.yml up -d

Compose brings up the gateway on port 4000 and Postgres. Read the file before you run it and replace the image tag with a specific version. A single docker run with a config and no database gives you only the OpenAI-compatible API: no keys, no budgets, no spend tracking.

Images and tags per Docker Image Security:

• ghcr.io/berriai/litellm is the main image, with litellm-non_root and litellm-spend_logs variants;
• vX.Y.Z tags are immutable, main-stable and main-latest are deprecated, and latest doesn’t belong in production;
• since v1.83.0 every image in GHCR is signed with cosign, so verify the signature before you deploy:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<commit>/cosign.pub \
  ghcr.io/berriai/litellm:v1.106.6

What you need for what (production, deploy):

Component	When you need it
Postgres	Virtual keys, teams, users, budgets, spend log, models added from the UI
Redis 7+	More than one instance or worker: shared rate limits, router state, cache
LITELLM_SALT_KEY	Encrypts provider credentials in the database. Don’t change it after adding models, or the saved keys become unreadable
DISABLE_SCHEMA_UPDATE=true	On proxy instances, while a separate job runs migrations
/health/readiness, /health/liveliness	Kubernetes probes

The docs size it at 1 vCPU and 4 GiB per worker, and in Kubernetes at --num_workers 1 per pod with 2+ replicas. They also suggest proxy_batch_write_at: 60 (batched spend writes to the database) and LITELLM_LOG=ERROR. The allow_requests_on_db_unavailable flag lets requests through when the database is down, and the docs allow it only inside a VPC.

Admin UI and observability

The Admin UI lives at <proxy>/ui and needs a database. You log in with UI_USERNAME (admin by default) and UI_PASSWORD. If no password is set, the master key works, and anyone who logs in that way gets proxy admin rights (Admin UI). From the UI you issue keys, add models without a restart, create teams, and view spend and logs.

Traces and metrics go out through callbacks:

• Langfuse. For Langfuse v3 and v4, the LiteLLM docs recommend the langfuse_otel preset via OpenTelemetry v2 over the older langfuse SDK callback. A step-by-step Langfuse setup, including the LiteLLM row, is in the Langfuse tutorial, and which metrics and alerts to collect is in LLM Observability in Production.
• OpenTelemetry v2. litellm_settings.callbacks: ["otel"] and LITELLM_OTEL_V2=true, with the collector address in OTEL_EXPORTER, OTEL_ENDPOINT and OTEL_HEADERS. Each request writes one trace: authentication, guardrails, the model call with gen_ai.* attributes, the database write. Prompt content stays out of the trace by default.

Caching and guardrails

The cache is either exact (local, disk, redis, s3, gcs) or semantic (redis-semantic, valkey-semantic, qdrant-semantic). An in-memory cache lives inside one worker: a proxy with four workers has four separate caches, so from the second worker on you need Redis. The exact cache keys on a hash of the whole request, so any change to the conversation is a miss. The semantic cache, by the docs’ own account, suits single prompts and fails badly on agent traffic.

Guardrails go in a guardrails: block with pre_call, post_call, during_call and logging_only modes. Built in are litellm_content_filter, Presidio for PII masking, and integrations with Lakera, Bedrock, Aporia and others (guardrails). You turn on a specific guardrail per request with "guardrails": ["name"].

Connecting clients

OpenAI SDK

Any OpenAI-format client works the same way: base_url pointing at the proxy, a virtual key in api_key, a model_name from the config in model. The example is in the quick start section.

Claude Code

Per the LiteLLM tutorial, Claude Code connects with two variables:

export ANTHROPIC_BASE_URL="http://127.0.0.1:4000"
export ANTHROPIC_AUTH_TOKEN="sk-..."   # virtual key or master key
claude --model cloud-sonnet            # model_name from config.yaml

The master key gives Claude Code access to every model on the proxy. A virtual key limits it to the models that key allows. If Bedrock sits behind the proxy, the tutorial recommends CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1, because Bedrock doesn’t accept every Claude Code beta header.

Two caveats from the Claude Code docs:

• Anthropic supports gateways with a compatible API format for Claude models, but it doesn’t audit third-party gateways and doesn’t support Claude Code on non-Claude models through any gateway. LiteLLM makes that technically possible, but quality and compatibility are on you.
• While a gateway credential variable is set, requests carry that key instead of your claude.ai subscription login. Subscription limits don’t apply to them, and they’re billed per token to whoever owns the key.

Verified: November 7, 2026, LiteLLM 1.108.1. POST /v1/messages with x-api-key and anthropic-version: 2023-06-01 headers to the local-qwen group returned an Anthropic-format response: type: message, stop_reason: end_turn, Qwen3’s reasoning in a separate thinking block, the answer in a text block, and usage of 16 input and 98 output tokens. Claude Code itself wasn’t run in this test.

So an Anthropic-format client can talk to a local model through the proxy. Whether an 8B model can handle Claude Code’s agent loop is a question about the model, and in OpenCode tests local models of that size were unstable.

OpenCode

The OpenCode docs have no dedicated LiteLLM entry, so the generic OpenAI-compatible provider mechanism does the job (providers). Here’s an example for OpenCode 1, in the same format as the Ollama setup in the OpenCode guide:

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "litellm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "LiteLLM",
      "options": {
        "baseURL": "http://127.0.0.1:4000/v1",
        "apiKey": "{env:LITELLM_API_KEY}"
      },
      "models": {
        "cloud-sonnet": { "name": "Sonnet 5.5 via LiteLLM" }
      }
    }
  }
}

The key in models must match the model_name in the proxy config. OpenCode 2 uses a different schema: a providers block with package and settings.baseURL fields. The proxy address goes in the same place.

Local models

Ollama connects to the proxy as a regular provider (see the config.yaml section). Clients then use one address and one key for both the local and the cloud model, and fallback switches between them without the application doing anything. An overview of ways to run models locally is in the local LLM guide.

What’s paid: MIT vs Enterprise

The license splits the repository in two. Everything outside the enterprise/ directory is MIT. You can copy and modify code in enterprise/ for development and testing, but in production it runs only with an active BerriAI subscription for the right number of seats, and you can’t redistribute it. Enterprise is the same image: the LITELLM_LICENSE key turns the features on after a full restart.

Free (MIT)	Needs an Enterprise key
Python SDK, Router, fallback	SSO and SCIM (SSO is free for up to 5 users)
Virtual keys, budgets and rate limits per key, user and team	JWT auth, assigning team admins
Spend tracking, Admin UI	Audit logs with retention, RBAC (organizations, teams, roles)
Load balancing and routing	IP allowlist, key rotation, secret managers
Guardrails framework, custom guardrails, Presidio	Guardrail callbacks llmguard_moderations, llamaguard_moderations, hide_secrets, openai_moderations, google_text_moderation, lakera_prompt_injection, aporia_prompt_injection
MCP gateway	Tag budgets, per-model budgets on a key (model_max_budget), temporary budget increases
Cache, logging to Langfuse and OTel	Soft-budget email alerts, spend reports, per-team logging, export to GCS and Azure Blob, guardrails per key and team, request and response size limits

The list comes from the Enterprise page. Pricing depends on usage volume and goes through sales. There’s a free 30-day trial, and you can buy through AWS and Azure Marketplace or by invoice.

LiteLLM security: incidents and CVEs in 2026

The proxy holds the master key, every provider key and the credentials in its database. A compromised proxy leaks all of them at once. That’s why the 2026 vulnerability record matters for any self-hosted deployment.

PyPI compromise, March 24, 2026. Versions 1.82.7 and 1.82.8 contained malicious code that harvested credentials, and 1.82.8 also shipped a litellm_init.pth file, which Python runs at interpreter startup. Per the team’s report, the packages were available from 10:39 UTC for about 40 minutes, and pip installs between 10:39 and 16:00 UTC count as affected. The official proxy Docker image wasn’t affected. The team names a Trivy dependency in CI as the source and brought in Mandiant to investigate. Clean versions are 1.82.6 and earlier, and 1.83.0 and later.

Major CVEs and advisories from NVD and GitHub Security Advisories, with CVSS scores from GitHub as the CNA:

ID	Published	CVSS	What	Fixed in
CVE-2026-35030	Apr 6	9.4	Auth bypass with JWT auth enabled (OIDC userinfo cache key collision); off by default	1.83.0
CVE-2026-35029	Apr 6	8.7	/config/update didn’t check for the admin role	1.83.0
CVE-2026-42208	May 8	9.3	SQL injection in API key verification: an unauthenticated request to any LLM route reads and modifies the proxy database (1.81.16 to 1.83.6)	1.83.7
CVE-2026-42203	May 8	8.6	SSTI in POST /prompts/test: code execution with any valid key	1.83.7
CVE-2026-42271	May 8	8.7	Command execution through MCP test endpoints with stdio transport	1.83.7
CVE-2026-49468	Jun 22	9.5	Auth bypass via the Host header: access to management routes without a key	1.84.0
CVE-2026-59822	Jul 8	8.8	Auth bypass in MCP Streamable HTTP via OAuth2 passthrough	1.84.0
CVE-2026-89032	Sep 25	8.7	Semantic cache served responses to other tenants (/v1/responses, /bedrock/*)	1.101.0-rc.1
CVE-2026-93355	Sep 28	7.6	JWT auth: log in as any user by email without an email_verified check	Not stated in the NVD record
CVE-2026-114612	Oct 22	9.9	Shared crypto context for secrets and sessions: an internal_user forges an admin token and runs commands via MCP stdio; in the default config since 1.91.0	1.100.4, 1.101.3, 1.102.2, 1.103.1
GHSA-w4c7-34wj-hqm8	Oct 27	10.0	Enterprise only: an unauthenticated request makes the proxy read a local file (for example /proc/self/environ with the master key and provider keys) and send it to the attacker’s address	1.103.10, 1.104.8, 1.105.6, 1.106.5

Dates are NVD publication dates (for the advisory without a CVE, the GitHub date). The GitHub advisory usually comes out a few days or weeks before the NVD entry. Beyond the table, 2026 also brought SSRF and provider credential leaks through api_base and user_config in the request body (CVE-2026-84377, CVE-2026-59823, Moderate), code execution through the custom guardrail test (CVE-2026-40217) and local file reads through vertex_ai_credentials (GHSA-g5ff-637f-6q2m).

Nearly all the serious holes sit in authentication, management, test and MCP endpoints, JWT and SSO, not in request routing. The latest critical one was in Enterprise code: the license widens the attack surface.

A checklist for self-hosting:

1. Pin the version: litellm[proxy]==X.Y.Z or an image tag vX.Y.Z, ideally a digest, never latest.
2. Verify the image signature with cosign (images are signed since v1.83.0).
3. For pip and uv: make sure you don’t have 1.82.7 or 1.82.8 (pip show litellm), and look for litellm_init.pth in site-packages.
4. Keep the proxy off the internet. Put it behind a VPN, mTLS, an authenticating ingress or an IP allowlist. Lock down /ui, /key/*, /team/*, /config/* and the MCP routes separately.
5. Restrict the proxy’s outbound traffic to provider addresses. With GHSA-w4c7, that’s exactly what cuts off exfiltration even on a vulnerable version.
6. Don’t turn on the MCP gateway, JWT or SSO unless you need them.
7. Set a separate LITELLM_SALT_KEY, and keep the master key in a secret manager where you can, rather than in the process environment.
8. Subscribe to the repository’s advisories and upgrade on them: fixes ship to several supported lines at once.

MCP endpoint risks aren’t unique to LiteLLM. The general threat model is in MCP Servers Explained.

LiteLLM limitations and gotchas

Postgres is mandatory for a team

Without a database there are no keys, budgets, Admin UI or spend log, and limits in the config are silently ignored. In the no-database test, a request without a key got a 500 instead of a 401. For a team, plan Postgres from day one.

One more hop, one more point of failure

About 7 ms per request on a local machine is small, but the proxy becomes a single point of failure for every client. Production needs 2+ replicas, Redis and health probes, which means one more system to run.

A new version every week

Releases ship several times a week: on October 27 the same fix went out in four lines at once (1.103.10, 1.104.8, 1.105.6, 1.106.5). Behavior changes between versions. The tests in this article ran on 1.108.1, released three days after the article’s date. Read the changelog and advisories before you upgrade in production.

Placeholders in the docs

LiteLLM doc examples use model names that don’t always match providers’ current IDs. Check the provider’s docs before you copy one into a config.

Semantic cache and agents

A semantic cache returns the closest answer by meaning. For agent traffic with tool calls that’s a source of errors, and in CVE-2026-89032 it also served answers across tenants.

When you don’t need LiteLLM

One model and one service get by with the provider’s SDK. A single developer with several providers is fine with the LiteLLM SDK and no server. If nobody can run a gateway, a cloud aggregator saves time, though it costs more in fees.

Where to start

1. Install a pinned version, uv tool install 'litellm[proxy]==1.106.6', and check litellm --version.
2. Write a config.yaml with one local and one cloud model and a master key from an environment variable, then start the proxy on 127.0.0.1.
3. Test a request with curl and the OpenAI SDK, then set up fallbacks and watch the x-litellm-attempted-fallbacks header while the primary model is unreachable.
4. For a team, bring up Postgres (compose quickstart), set LITELLM_SALT_KEY, and issue keys with max_budget and rpm_limit.
5. Connect clients: the OpenAI SDK, Claude Code through ANTHROPIC_BASE_URL, OpenCode through an OpenAI-compatible provider.
6. Work through the security checklist: private network, restricted egress, image signature, advisory subscription.