Latest

Technology

Superfluid, an AI harness that breaks every speed record

Superfluid is an open-source local server that distributes a single model among multiple agents, with APIs compatible with OpenAI, Anthropic and Ollama.

Superfluid, the open-source software that serves LLMs to multiple agents
Superfluid, the open-source software that serves LLMs to multiple agents

Asking a local model to do one thing while doing seven others at the same time does not work well in most home setups. As a result, the agent that summarizes your emails at night takes control of the machine, and when you type your question into the chat, you are actually waiting behind everything already in the queue. To fix this, Basecompute published on HuggingFace, now in NVIDIA's hands, a tool that tackles exactly that problem, and it did so with an uncomfortable number in hand.

With eight agents processing documents, a chat question launched at the two-second mark received its first token in 1.5 seconds with Superfluid. The same model and the same machine, but through llama-server, took 291 seconds. The difference is not in the inference engine, which is the same in both cases: it is in the layer that decides whose turn it is to speak, and that simple change has a major impact.

Superfluid and its proposition: speed and agentic multitasking

The benchmark accompanying the announcement comes from the vendor itself, so it should be read as a self-interested measurement but one with a published methodology. It was run on an Apple M5 Pro with the Qwen3.8-27B model quantized to 4 bits, and each server used the same inference engine, so the comparison isolates the serving layer.

Measurement Superfluid llama-server Ollama mlx_lm.server
First token for a chat behind 8 agents 1.5 s 291 s 380 s 612 s
Cache hits with 32 requests sharing a 4k-token prompt 97% 75% 97% 0%
Recovery when killing the worker mid-stream 1.8 s not applicable 1.8 s, truncated stream not applicable

The first row is what justifies the project. The 291 seconds for llama-server and the 380 for Ollama are not a performance failure, they are simply the result of serving in order of arrival.

If eight agents requested work before you, your question is the ninth task they have to do. Ollama's queue, moreover, has no way of knowing that your request comes from a person and not from a scheduled process. There is no control and prioritization line, and that has a cost.

The third row deserves its own comment. When the inference worker is killed mid-response, Superfluid restores service in 1.8 seconds and the session continues. Ollama also takes 1.8 seconds, but the stream arrives truncated, which means whoever was reading the response loses the ending without realizing it. llama-server and mlx_lm.server do not cover that scenario in the comparison.

How is it measured and why does the method matter?

Basecompute published the harness it used for the measurements inside the repository itself, under the name superfluid/tools/suite, and the full figures with their methodology on the project's website. That decision deserves recognition because it allows the comparison to be repeated on another machine instead of simply believed.

The measured scenario is described precisely: eight agents working in parallel on 7,500-token documents and a chat question that arrives when the system has already been busy for two seconds. All the compared options use the same inference engine, so the only variable that changes is the layer that distributes the work. Without that level playing field, the comparison would say nothing.

The preemption detail explains the rest. The interactive request does not force its way into the middle of a batch: it waits for the scheduler's next tick and takes its turn before the batch work. That nuance is the difference between a predictable system and one that interrupts half-finished computations to serve the user.

One server, three APIs and three engines

Superfluid sits in front of the inference engine and exposes three different interfaces over the same process. It speaks the OpenAI API, the Anthropic API and the OllamaAPI, so any client written for any of the three works by pointing it at 127.0.0.1:8453 without touching a single line of code. The API key is not validated, because the server is local.

Underneath, the engine is chosen according to the model format. GGUF files go to llama.cpp, MLX directories go to Apple's engine and .base packages go to BaseRT, Basecompute's own engine.

The practical advantage of that unification is that tool-calling support, structured output and chat templates are the same in all three cases, something that does not happen if they are set up separately.

Infographic of how Superfluid works
Infographic of how Superfluid works

Startup is deliberately short. An installation script with curl for macOS on Apple silicon and for Linux on x86-64 and arm64, or cargo install superfluid for those who prefer to compile it. On Apple silicon and Linux, the BaseRT engine is installed by default, while llama.cpp and MLX are added automatically when the served model requires them, so there is no need to decide in advance which runtime will be used.

Serving a model from the Hub is one line, and the repository tag chooses the quantization, as in unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_M. The same command with an MLX model works the same way by pointing to a directory in that format. There is also a launch command that starts the server if it is not running and launches the agent pointed at it; the published list includes Claude Code, Codex, pi, OpenCode, Hermes Agent, Cline, OpenClaw and Ollama's own CLI, among other compatible clients.

Sessions that survive a shutdown

The project's second design decision is less visible and probably more useful day to day. Superfluid writes every token to a log before delivering it to the client. If the worker crashes or the server restarts, the session resumes where it left off instead of starting from scratch. In a long run of an agent that has been working for twenty minutes, that avoids repeating all the previous work.

There is a third mechanism that saves compute in a less obvious way. When several requests share the same system prompt, the prefill is computed only once. In the measured scenario, 32 requests with a shared 4,000-token prompt produced 97% cache hits versus 75% for llama-server and 0% for mlx_lm.server. Ollama also reaches 97% in that row, and it is the only section of the table where there is no appreciable difference.

The fourth mechanism addresses a scenario that is growing in popularity: distributing the model across several machines. One is the head, it exposes a port for guest nodes and generates a token; the others join with that token and download the model on their own. The head places complete sessions on the nodes and only receives already-sampled tokens, so traffic between machines is small compared with moving activations. The link is encrypted with the token and, if a node dies mid-stream, its session is reassigned without duplicating tokens.

Nodes can use different engines for the same model. The example given in the announcement is a Mac running BaseRT alongside a Linux machine with llama.cpp, both serving the same model under the same logical session. For a small office with heterogeneous hardware, that mix avoids having to standardize the fleet before starting.

What the comparison does not say

The stated performance has three limits that the announcement does not hide but that are worth keeping in mind. The first is that the measurement comes from the vendor and was done on its own test bench, although the methodology and full figures are published on its website. There is not yet an independent reproduction with the same configuration.

The second is that the advantage disappears when there is no contention. The announcement itself acknowledges that, with one or with eight concurrent requests, llama.cpp served through Superfluid matches llama-server in tokens per second. The serving layer does not add performance by itself: what it adds is order when there is a queue.

The third is one of scope. The comparison measures latency and recovery, not result quality or energy consumption. Nor does it cover how much memory the server uses, a relevant figure when the goal is to share a machine with eight agents and keep using the computer for work.

Who needs a priority queue?

The question Superfluid leaves open is not whether the software works, but how many people today have the problem it solves. A user with a single agent and a single question does not need fair distribution: they need the model to respond, and for that llama-server or Ollama are enough. The situation changes when the machine becomes a resource shared among processes that know nothing about each other.

That is where the project becomes interesting and also where its risk appears. The more distribution logic is added between the client and the model, the more pieces can fail silently. Basecompute's bet is that logging every token beforehand compensates for that complexity, and that interactive priority is an operating system feature rather than a convenience.

For anyone who has long been running local models at home or in a small office, the proposition is easy to evaluate: if you have ever seen your own question wait behind a scheduled process, the problem is real and this is a reasoned attempt to solve it. If it has never happened to you, there is nothing to gain by installing it yet.

José Maldonado

Author

José Maldonado

Activist and blogger on technology, free software and blockchain.

The latest
From Huang to Botín, from Zara to Byron: corporate responsibility in global capitalism

Keep reading