One Endpoint, Every Model: LiteLLM in Front of vLLM Source

1---2title: "One Endpoint, Every Model: LiteLLM in Front of vLLM"3date: "2026-11-18"4published: true5tags: ["litellm", "vllm", "ai-gateway", "dgx-spark", "self-hosted", "ai", "mcp"]6author: "Gavin Jackson"7excerpt: "LiteLLM Proxy is the missing internal glue between our developers and our models: one OpenAI-compatible endpoint that fronts the DGX Spark cluster and commercial providers, with per-developer API keys, selective model access, budgets, guardrails and MCP tool gating. Here's how it works, what the Enterprise licence adds, and what else is out there."8---910# One Endpoint, Every Model: LiteLLM in Front of vLLM1112<p align="center"><img src="/assets/ai/litellm-gateway-hero-v2-rack-cloud-streams.jpg" alt="A black gateway appliance with champagne-gold trim and a glowing amber keyhole on a rack shelf, with neon data streams rising to a glowing cloud above" width="800"></p>1314A few months back I wrote about [Pangolin's AI Gateway](/post/pangolin-ai-gateway-custom-providers-kimi-deepseek), which solved the remote-access side of our AI plumbing: identity-aware proxying in front of AI APIs for users coming in from outside. What it didn't solve was the internal problem. We have a [four-node DGX Spark ring](/post/dgx-spark-200gbe-ring-cluster) serving models through vLLM, a handful of commercial provider accounts, and developers who need access to some combination of all of it. The question was how to hand out keys without handing out the keys: the real vLLM endpoints and the real vendor credentials stay private, each developer gets their own key, and each key only opens the models that developer should be using.1516The answer turned out to be [LiteLLM](https://github.com/BerriAI/litellm), and specifically its Proxy Server. It is the missing glue I was looking for, and it has quietly become the piece of kit that ties the whole setup together.1718## What LiteLLM actually is1920LiteLLM is best known as a Python SDK that translates one OpenAI-style call into the API dialect of whatever provider you are targeting. The Proxy Server is that same translation layer packaged as a FastAPI service: it presents a single OpenAI-compatible endpoint (`/v1/chat/completions`, `/v1/responses`, `/v1/embeddings` and friends) and fans requests out to over a hundred providers. The docs track around 190 provider integrations at the time of writing, the repo sits at 58.8k stars, and the project is shipping at a serious pace (v1.110.0 is current this week).2122The important detail for us: anything that already speaks the OpenAI API works as a backend, and vLLM speaks the OpenAI API. So does just about everything else we touch, including the model endpoints our platform team publishes through GenAI on Tanzu Platform for Cloud Foundry. One proxy, one API shape, every model we own or rent behind it.2324## Getting going is genuinely easy2526This is the part that surprised me. For a first look, it is three commands:2728```bash29pip install 'litellm[proxy]'30```3132```yaml33# config.yaml34model_list:35  - model_name: qwen3.5-397b36    litellm_params:37      model: openai/qwen3.5-397b-int4-autoround38      api_base: http://spark-01:8000/v139```4041```bash42litellm --config config.yaml43```4445That gives you a proxy on port 4000 serving the 397B model off the Spark cluster, callable with any OpenAI client pointed at `http://localhost:4000`. The `openai/` prefix in the model string is LiteLLM's way of saying "this backend speaks the OpenAI protocol, here is where it lives." Native vLLM support exists too, but the OpenAI-compatible path is what I use everywhere because it keeps every backend uniform.4647For a real deployment you want the published Docker setup instead: the `ghcr.io/berriai/litellm` image plus Postgres in a compose file, with `DATABASE_URL` set. Postgres is not optional decoration. Virtual keys, budgets, spend logs, teams and the Admin UI all live there, and the proxy runs its own Prisma migrations at startup. Skip the database and `/key/generate` will simply tell you there is no connected DB. The project provides a one-line bootstrap script and a ready-made compose file; the whole thing from `docker compose up` to a working Admin UI is under ten minutes.4849## Issuing keys instead of sharing them5051This is the core of the exercise. With the proxy up and `LITELLM_MASTER_KEY` set, creating a developer key is one API call:5253```bash54curl -X POST http://localhost:4000/key/generate \55  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \56  -H "Content-Type: application/json" \57  -d '{58    "models": ["qwen3.5-397b", "glm-flash"],59    "max_budget": 50,60    "budget_duration": "30d",61    "rpm_limit": 120,62    "duration": "90d",63    "metadata": {"developer": "james", "project": "caravan-monitor"}64  }'65```6667That key can call exactly two models, has a USD 50 budget that resets every 30 days, is capped at 120 requests per minute, and expires in 90 days. Spend is tracked per request in the `LiteLLM_SpendLogs` table using LiteLLM's model cost map, and for local models the map entries are effectively zero, so a developer can hammer the Spark cluster all month without touching the budget while every commercial call is metered. `/key/info` shows you what any key has spent, and `/key/update`, `/key/block` and `/key/delete` cover the rest of the lifecycle.6869<p align="center"><img src="/assets/ai/litellm-admin-ui-virtual-keys.png" alt="The LiteLLM Admin UI Virtual Keys page, showing a table of keys with team, user, spend and budget columns, and a Create New Key button" width="1200" loading="lazy"></p>7071Everything above also has a point-and-click equivalent: the Admin UI's Virtual Keys page lists every key with its team, owner, spend and budget, and the same creation form is behind the + Create New Key button. Handy for one-off keys; the API is what you script.7273For groups of developers there are teams (`/team/new` with its own models, budget and rate limits, members added by user ID) and, my favourite access-control feature, **model access groups**. Tag models in the config:7475```yaml76model_list:77  - model_name: qwen3.5-397b78    litellm_params:79      model: openai/qwen3.5-397b-int4-autoround80      api_base: http://spark-01:8000/v181    model_info:82      access_groups: ["local-models"]83  - model_name: gpt-584    litellm_params:85      model: openai/gpt-5.286      api_key: os.environ/OPENAI_API_KEY87    model_info:88      access_groups: ["commercial"]89```9091Then a key created with `"models": ["local-models"]` inherits every model in that group, present and future, without touching the key again. The practical split for us is exactly the one above: `local-models` for the Sparks, which everyone gets, and `commercial` for anything that bills us, which is granted case by case. One caveat: wildcard access groups (a group that matches every model) are an Enterprise feature. Named groups are free.9293## One front door, local and commercial9495With that, the unified endpoint falls out naturally. A trimmed version of the shape we run:9697```yaml98model_list:99  # The ring - one vLLM endpoint presenting the big model100  - model_name: qwen3.5-397b101    litellm_params:102      model: openai/qwen3.5-397b-int4-autoround103      api_base: http://spark-01:8000/v1104      rpm: 60105  # A second deployment of the same model_name = load balancing group106  - model_name: qwen3.5-397b107    litellm_params:108      model: openai/qwen3.5-397b-int4-autoround109      api_base: http://spark-02:8001/v1110      rpm: 60111  # Small fast model on a single Spark112  - model_name: glm-flash113    litellm_params:114      model: openai/GLM-5.3-Flash-FP4-Local115      api_base: http://spark-03:8000/v1116  # Commercial117  - model_name: gpt-5118    litellm_params:119      model: openai/gpt-5.2120      api_key: os.environ/OPENAI_API_KEY121  # Models our platform team brokers through Cloud Foundry122  - model_name: cf-llama123    litellm_params:124      model: openai/llama-4-maverick125      api_base: https://genai.apps.example.com/v1126      api_key: os.environ/CF_GENAI_KEY127128router_settings:129  routing_strategy: simple-shuffle130  num_retries: 2131  timeout: 600132  allowed_fails: 2133  cooldown_time: 60134  fallbacks: [{"glm-flash": ["gpt-5"]}]135```136137Two deployments under one `model_name` form a load-balancing group; LiteLLM spreads requests across them honouring the per-deployment `rpm` hints (the default `simple-shuffle` strategy, with least-busy, latency-based, usage-based and cost-based strategies available, plus cache-aware routing if you want prompts to stick to the node that already has the context warm). `fallbacks` means a developer calling the local flash model gets quietly redirected to a commercial model if the Spark is down or saturated. There is also `model_group_alias`, which lets you publish a stable public name and swap the underlying deployment underneath it, so developer code never has to change when the cluster does.138139The result: developers get one base URL and one key. Local models, commercial vendors and the Cloud Foundry-brokered models all sit behind it, and which of those any given key can see is a line in a config file or a field in a key-creation call.140141## Guardrails and policies142143LiteLLM has a proper guardrail framework rather than a checkbox. Guardrails attach to the proxy with modes (`pre_call`, `during_call`, `post_call`) and can be switched on globally with `default_on: true`. The open-source build gives you two things: **custom guardrails**, where you point the proxy at your own code that inspects and can block or mutate any request, and **Presidio-based PII detection and masking**. A minimal Presidio setup looks like:144145```yaml146guardrails:147  - guardrail_name: pii-mask148    litellm_params:149      guardrail: presidio150      mode: pre_call151      default_on: true152```153154That scrubs personally identifiable information out of prompts before they leave the proxy, which matters the moment prompts can be routed to a commercial fallback. The named third-party integrations - LLM Guard, Llama Guard, OpenAI Moderations, Lakera, Aporia, hidden-secret detection and so on - are Enterprise-licensed, as is attaching guardrails per key or per team rather than globally. Worth knowing when you are scoping which tier you need.155156## MCP: the same key that gates models gates tools157158The feature I had not expected to care about, and now consider essential, is the **MCP gateway**. MCP servers (the tool providers for agentic workflows) register in the same config file over HTTP, SSE or stdio, and the proxy exposes them all on a single `/mcp/` endpoint:159160```yaml161mcp_servers:162  grafana:163    url: https://grafana.internal.example.com/mcp164    auth_type: api_key165    auth_value: os.environ/GRAFANA_MCP_KEY166```167168A developer's agent connects to `https://litellm.internal/mcp/` with two headers: `x-litellm-api-key` for their normal virtual key, and `x-mcp-servers` to select which servers they want. The access-control angle is the point: keys and teams carry `mcp_servers` and `mcp_access_groups` restrictions, so the same key that decides which models James can call also decides which tools his agent can touch. There is full REST CRUD for MCP servers if you want to manage them without editing config, and a `require_key_mcp_access_defined` setting for the paranoid that denies MCP access to any key without an explicit grant. One control plane for models and tools is a much better story than two.169170## Watching it171172The **Admin UI** (browse to `/ui`, log in with the master key) covers keys, users, teams, models, budgets and spend charts, and for day-to-day operation it is genuinely decent: creating a key or checking who spent what is a thirty-second job. Prometheus metrics come off `/metrics` for Grafana, and for LLM-specific observability the proxy ships native integrations with Langfuse (the `langfuse_otel` preset is the recommended one), LangSmith and plain OpenTelemetry, so full request traces land wherever your tracing already lives.173174## What the Enterprise licence adds, and what it costs175176The open-source build is MIT-licensed and, usefully, the Enterprise features ship in the same container image: set a `LITELLM_LICENSE` environment variable to your licence key and the gated features unlock in place. The `enterprise/` folder of the repo is under BerriAI's separate enterprise licence, which is at least honest about where the line is.177178The headline gated features:179180- **SSO/OIDC/SAML** for the Admin UI beyond five users (five is the free allowance), plus **SCIM** provisioning, **audit logs** and JWT auth181- **Secret manager integrations** - HashiCorp Vault, AWS/Azure/GCP secret stores - so provider credentials never sit in config files or environment variables182- The named **guardrail integrations** (LLM Guard, Lakera, Aporia and friends), and **per-key/per-team guardrails**183- **Wildcard model access groups**, spend reports, tag-based budgets, per-model budgets and temporary budget increases184- Team-based logging controls (useful for GDPR-style data handling), GCS/Azure log export185- A multi-region control plane for running proxies across regions, FIPS-compliant builds, and the support SLA itself186187Pricing is not published, and this is where researching the costs runs into a wall that turns out to be deliberate. LiteLLM Enterprise is sold as an **annual subscription, sized by the request capacity you need through the gateway**, with deployment architecture and support level factored in. BerriAI is explicit that they **never charge per token** - their argument being that a per-token model punishes you for succeeding. There is also no longer a hosted LiteLLM Cloud tier; the hosted beta was quietly dropped and the pricing page now lists only the self-hosted options. You talk to sales, you get a quote, there is a 30-day free trial licence, and it is procurable direct or through the AWS and Azure marketplaces. Air-gapped deployments are available.188189For actual numbers you have to go to third-party deal data. Vendr's benchmarks, drawn from a small handful of real purchases, put the median Enterprise deal at around **USD 27k per year**, with a typical range of USD 14k to 49k depending on capacity and support level. Treat those as ballparks rather than list prices, but they set the scale of the conversation: this is enterprise-software money, not pocket-money SaaS.190191> ### Breakout: Is it worth paying?192>193> For our scale - one cluster, a single-digit number of developers, no compliance mandate - the MIT build covers every requirement on the list. Virtual keys, budgets, access groups, routing, Presidio guardrails, MCP gating, Prometheus and the Admin UI are all free. Enterprise becomes compelling at a specific inflection point: the day SSO/SCIM, audit logging or a proper secrets backend stop being nice-to-haves and become audit findings, or the day the gateway is critical enough that you want a vendor on the phone. That day will come for the work environment before it comes for the home office, and the pricing model means the conversation will be about capacity and support, not about metering our own usage - though the deal benchmarks above say that conversation starts in the tens of thousands a year.194195## The alternatives196197LiteLLM is not the only option. I looked at the field and cut it to the five that are genuinely the same kind of thing: self-hostable, OpenAI-compatible gateways with virtual keys, budgets and model routing you could actually put in front of vLLM. Hosted-only services like Cloudflare AI Gateway and OpenRouter solve a different problem, and Kong's AI plugins only make sense if you already run Kong.198199- **[Portkey AI Gateway](https://github.com/Portkey-AI/gateway)** - open-core, MIT-licensed gateway with virtual keys, budgets and RBAC, self-hostable or SaaS. The closest like-for-like competitor, with a polished hosted control plane around it: the cloud tier starts free (10k logs/month), Production is $49/month, and SSO/SCIM plus audit logs sit in the custom Enterprise tier.200- **[Bifrost](https://github.com/maximhq/bifrost)** - Apache-2.0, written in Go by Maxim, with virtual keys, budgets and rate limits in the open-source tier (the closest feature match to LiteLLM I found) and an aggressive performance pitch: they claim microseconds of overhead and roughly 9x LiteLLM's throughput. Enterprise adds HA clustering, SSO, guardrails and an MCP gateway, with pricing behind a demo form. The most interesting newer entrant.201- **[Helicone ai-gateway](https://github.com/Helicone/ai-gateway)** - Rust, observability-first, with a claimed sub-millisecond overhead. Strong if analytics is your primary concern. Their cloud runs Hobby (free, 10k requests/month), Pro ($79/month) and Team ($799/month) tiers; the gateway itself is self-hostable, though note the licence is actually GPL-3.0 despite a stale Apache badge in the README, which matters if you plan to modify and embed it.202- **[TensorZero](https://github.com/tensorzero/tensorzero)** - Apache-2.0 Rust LLMOps platform whose gateway claims sub-millisecond p99 latency at 10k+ QPS, with observability and experimentation built in. It is more than a gateway and less focused on per-key access control; if you want the optimisation and evaluation tooling too, it is worth a serious look.203- **[new-api](https://github.com/QuantumNous/new-api)** - a fork of one-api with channel management and a polished dashboard, popular in the Chinese self-hosting community and very actively maintained. Functional and free, with one caveat: it is AGPL-3.0 with additional attribution terms, which deserves a careful read before you build anything commercial on it.204205Two honourable mentions that failed the same-kind-of-thing test: [RouteLLM](https://github.com/lm-sys/RouteLLM), the LMSYS research project on cost-quality routing - intellectually influential, effectively unmaintained since August 2024, read it for the ideas - and Envoy's AI gateway work, donated to the Linux Foundation's Agentic AI Foundation in September 2026 and the natural path only if you already run Envoy at scale.206207The pattern in this list is that most alternatives lead with either performance (Bifrost, Helicone, TensorZero) or a hosted control plane (Portkey), and only Portkey and Bifrost match LiteLLM on virtual keys and budgets as first-class features. LiteLLM's edge is the combination: the widest provider coverage, keys with budgets and access groups in the MIT tier, MCP gating, and a project velocity the others are not matching. The weakness is the same Python/FastAPI core that makes it approachable - if you are pushing serious request volumes, benchmark it against the Go and Rust options before committing.208209## Where this leaves us210211The internal gap is closed. Developers get their own keys, keys get model allow-lists and budgets, the Spark cluster is free to use and the commercial models are metered, and the same credentials gate MCP tools as well as models. Pangolin still handles the outside world; LiteLLM owns the inside one. The two complement each other neatly, and if anything the Pangolin post undersold how much of the endgame LiteLLM already implements.212213Next on the list is standing this up permanently in front of the ring with proper DNS and TLS, wiring the Admin UI behind SSO if the user count ever justifies the Enterprise conversation, and pointing [Bob](/post/bob-got-an-update-kimi-k3) at the proxied endpoint for a week to see what the spend reports say about how I actually use models.214215---216217**Related:**218219- 🔑 [LiteLLM Proxy Server docs](https://docs.litellm.ai/docs/proxy/quick_start) - quickstart, virtual keys and config reference220- 💰 [LiteLLM pricing](https://www.litellm.ai/pricing) - the Enterprise model, capacity-based and never per-token221- 🔐 [There's No Kimi Button: Custom Providers on Pangolin's New AI Gateway](/post/pangolin-ai-gateway-custom-providers-kimi-deepseek) - the remote-access half of the story222- ⚡ [Clustering Four DGX Sparks in a 200Gb/s Ring](/post/dgx-spark-200gbe-ring-cluster) - the cluster behind the `local-models` group223- 🦙 [From Ollama to vLLM](/post/from-ollama-to-vllm) - how the models got onto the Sparks in the first place224