Latest

Artificial Intelligence

llm-d, open-source software for distributed inference

llm-d 1.0 spreads inference across clusters with vLLM and SGLang and turns cost per token into the new competitive battleground for open AI.

llm-d, open-source software for distributed inference
llm-d, open-source software for distributed inference

Spreading an AI across dozens of servers sounds like plumbing until the cloud bill arrives. The llm-d stack reached version 1.0 on October 20, 2026, a release that turns loose replicas into a fleet capable of deciding which machine runs each request, according to the Google Cloud announcement.

The project was founded in 2025 with Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA as founders, and since March 24, 2026 it has been a sandbox project of the Cloud Native Computing Foundation, the foundation that hosts Kubernetes. That governance matters, because it is not the product of a cloud provider looking to sell its share, but a piece that several rival companies maintain in common. Today more than 500 people from 25 organizations contribute to the code.

Version 1.0 is measured in two magnitudes that until now were tracked separately: latency and money. IBM Research serves open models with this stack at about $0.11 per million tokens and calculates a cost 11.5 times lower than an equivalent proprietary configuration, according to the Red Hat press release. With those numbers, llm-d has turned distributed inference into a competitive battleground and has given those who serve their own models a yardstick that does not depend on their vendor's marketing brochure.

From serving a model to operating a fleet

The 0.x versions solved scope: routing that serves the prefix cache, offloading context memory to RAM or disk, separating context computation from generation, and parallelism for mixture-of-experts models. Version 1.0 adds what appears once the system is already in production: multiple tenants on the same fleet, competing priorities and updates that cannot take the service down.

The router is the piece that decides. It keeps track of which replica holds each context, weighs load and expected latency, and chooses a destination for each step of an agent. Its job is to avoid repeating computation.

Built on Gateway API Inference, when accelerator memory fills up it moves that cache to RAM, SSD or Lustre storage instead of discarding it, and avoids the eviction storms that appear with long contexts. In the PyTorch Conference demo, 1,000 isolated agents hit a single entry point served by vLLM and SGLang across a mix of TPU and GPU on Google Cloud.

Diagram of how llm-d and its platform work
Diagram of how llm-d and its platform work

The launch figures point to that same bottleneck. In a deployment of 17,000 nodes spread across three clusters, the router multiplied throughput by 2.9 with 99.9% of requests successful and less than 1% overhead, according to the measurements published by Google Cloud. In an agentic coding test on TPU Ironwood, prefix routing cut median time to first token by more than 85% and raised throughput 2.4 times compared with a conventional Kubernetes service.

None of that requires changing engines. The stack relies on vLLM and SGLang, the two most established open model servers, and accepts others through their connectors. Anyone who already has a vLLM deployment can put the routing in front and add flow control afterwards.

Cost per token enters the scoreboard

For two years the conversation about open AI was settled by quality benchmarks. Version 1.0 moves the discussion to the table of money per token served, where the numbers are easier to verify because they come from the bill and not from the test bench.

Test What was changed Published figure
Three clusters, 17,000 nodes Multi-cluster routing 2.9 times more throughput with 99.9% success
70 billion parameter model on A3 Mega Cache offload to Lustre Almost 60% fewer GPU hours with 50,000-token prompts
GLM 5.2 on 544 H100 Caching, phase separation and expert parallelism 6.6 million tokens per minute and up to 3,000 agents at once
Quantum error correction at IBM Open models on llm-d $0.11 per million tokens, 11.5 times less than a proprietary stack

The main lever is the prefix cache. Much of agentic traffic does not produce new tokens but rereads the same context over and over, and every reread that hits the cache is computation that is not paid for. Gemini, the platform formerly called Vertex AI, doubled its hit rate from 35% to 70% by going through the GKE gateway built on llm-d, and cut time to first token by more than 35% in coding agents.

There is a second path, less elegant and very profitable. During reinforcement learning training, accelerators sit idle between 40% and 60% of the time, waiting their turn between trajectory generation and weight updates. Version 1.0 interleaves asynchronous inference into those gaps and raises aggregate utilization from 40% to more than 70%.

What does someone who self-hosts their own model gain?

For a team serving its own weights, the fundamental change is not a new feature but the ability to mix hardware without touching the application that consumes the model. The project's founding promise rests precisely on that separation, because the same control plane serves any model on any accelerator.

Version 1.0 validates the same control plane on AMD, Intel and NVIDIA GPUs, on Google TPUs and on minority accelerators such as Rebellions NPUs, with guides for each platform. DigitalOcean serves the Kimi K3 model, with one trillion parameters, across AMD and NVIDIA cards with the same layer, and Roblox spreads models across cloud and its own GPUs.

The second change is that Kubernetes is no longer mandatory. Since May 2026 the router can discover servers through a file, without an API server, without custom objects and without cluster permissions, so the same logic runs on Slurm, inside a Ray job or on a bare-metal rack. For a university computing center, that is the difference between adopting the stack and rewriting it for each framework.

There remains a price that does not appear in the benchmarks, because someone has to operate all of this: Prometheus metrics, OpenTelemetry traces, uninterrupted Helm charts and on-call staff who respond when the queue grows.

The production data accompanying the launch comes from teams with that muscle. JetBrains serves its Junie Lite plan on llm-d, and Tesla and Red Hat report three times more tokens per second with Llama 3.1 70B on MI300X. These are vendor figures, though they agree on the direction: the bottleneck moves from the card to the software that distributes it.

The fine print of the benchmarks

All the percentages above are published by whoever sells the stack or whoever uses it, not by an independent laboratory. IBM's comparison with a proprietary configuration is its own, with its model and its workload. The 11.5-fold saving describes that specific case and not a general rule.

The gain depends on context reuse. A 98% cache hit rate requires traffic with long, repeated prefixes, such as coding agents or training trajectories; a chatbot with one question and one answer barely repeats anything and gets only the small part. Snap reports hit rates of 75% to 80%, and that range, rather than 98%, is what is reasonable for a conversational workload.

There is also a size threshold. Flow control, router high availability and rolling updates only pay off with several dozen replicas and several tenants. With two cards in a closet, that machinery adds failure surface without giving much back.

The bill for the change arrives all the same: the Kubernetes object model, its charts and its release cycle, plus the engine underneath. Version 1.0 stabilizes the configuration API and reduces risk with each update, though it does not eliminate platform work.

Who sets the bar for open inference?

The next stretch is already on the community roadmap: active-active routers, migration of in-flight requests when a replica fails and broader multimodal coverage. None of those pieces will change the question that version 1.0 leaves in place, which is no longer whether open source can serve a large model, but at what price per token it does so compared with a closed API.

That is the ground where llm-d has more than adoption at stake. If dollars per million tokens become comparable and auditable, the discussion about self-hosting stops being a matter of sovereignty or taste and becomes arithmetic. The arithmetic published by the vendor always calls for outside review.

José Maldonado

Author

José Maldonado

Activist and blogger on technology, free software and blockchain.

The latest
From Huang to Botín, from Zara to Byron: corporate responsibility in global capitalism

Keep reading