Latest

Artificial Intelligence

Magnitude, software that squeezes your GPU for AI agents

Magnitude is an open source YC software that self-tunes its kernels to your hardware and runs open models up to 2x faster than llama.cpp

Magnitude, software that squeezes your GPU for AI agents
Magnitude, software that squeezes your GPU for AI agents

You download an open model, start it with a generic engine and it runs slow, eats memory and leaves the machine shivering. Blaming the hardware is the easy way out, but two engineers named Anders and Tom believe the problem lies in the middleware. Their project, Magnitude, conquered the Hacker News front page with a three-digit vote count, championing an inference engine that self-tunes to your exact chip.

The idea is simple and very elegant. Current engines choose between broad compatibility with lukewarm performance or high performance tied to specific hardware. Magnitude promises the best of both worlds: kernels with flexible parameters that are compiled and tuned on your own device before running the model. The result, always according to its own benchmarks, reaches up to double the speed of llama.cpp in agent scenarios.

The context is a summer 2025 Y Combinator that bet on them and prior work that gave them credibility. Both founders came from building an open source browser agent with thousands of stars and a hundred thousand downloads, and they hit the wall that no engine was suited to local agents. Long sessions, several at once and with the computer in use for other things. Magnitude is born from that specific scar, as its Y Combinator profile.

recalls

A different inference engine Think of an off-the-rack suit versus a tailored one. Generic engines ship precompiled kernels that fit everyone passably. Magnitude takes your measurements first and sews the kernel that performs best on your chip, all explained on itsofficial website

. Tuning takes about a minute per model.

The focus on popular architectures sharpens the trick. Instead of generalizing for every model in the world, the team writes efficient kernels for the most used open families. That concentration allows them to beat generalist engines without locking themselves into a single chip, because models of the same family share architecture and reuse the optimization only once.

Dynamic memory is the second leg. The engine reserves at startup only what is needed for the weights and expands the heap as sessions grow. Your machine remains usable for everything else while the agents work. In their measurements, each agent consumes 27% less memory.

Hybrid paged attention completes the picture. It takes ideas from datacenter engines so that several sessions share caches, but places the data with locality so that a single session does not suffer. Agents mix long turns with tools, and that pattern is the one the team optimizes with the most care.

  • What the engine tunes on your chip
  • The parameters of each kernel for your specific GPU or CPU.
  • The cache size according to available memory.
  • The draft model that accelerates generation.

Data placement when there are simultaneous sessions.

That minute of tuning pays for itself. From then on, each session starts with the kernels already tuned and memory sized to your machine. If you change GPU or update the model, the process repeats once and that's it.

The numbers they boast about and how to read them

Their own benchmarks use a Qwen 3.6 35B with 3B active, in 4 bits and 64K context, without speculative decoding. On Metal on a Mac M4 Pro with 48 GB, decoding rises from 30 to 57 tokens per second, 92% more, and prefill improves by 9%. On CUDA on a DGX Spark, decoding goes from 49 to 58 and prefill from 2,033 to 2,507, with 27% less memory per agent.

It is worth noting that these are vendor figures, measured in their laboratory and with their configuration. The history of benchmarks is full of local victories that deflate on someone else's hardware. The honest warning is to treat these numbers as a starting hypothesis and repeat the measurement on your machine before celebrating.

The comparison with the competition helps situate the bet. Datacenter engines prioritize massive batch at the expense of the individual session. Generalists prioritize compatibility at the expense of the ceiling. Magnitude wants the ceiling of the specialists with the breadth of the generalists.

For Europe there is a practical angle. Thousands of SMEs and administrations want AI without sending data abroad, and the available hardware is a zoo of Macs, Windows workstations and varied servers. An engine that tunes itself on each machine reduces the cost of that heterogeneous deployment. Less memory per agent also means capable models on modest equipment, something that tight budgets appreciate.

DirectX is already an LLVM target and revolutionizes gaming on Linux

How do you use Magnitude with your favorite agent?

It remains to be seen whether the numbers hold up outside the laboratory and whether the hybrid cloud arrives without locking in the project. The start, with an HN front page, real open source and founders who respond, invites cautious optimism. If you have a hungry local agent, this is the week to measure it against your current engine on your own machine.
Magnitude is distributed as a desktop app for Mac, Linux and Windows, and works with any Apple, NVIDIA or AMD GPU, or even CPU only. There is no fixed minimum, small machines run small models and more memory allows larger models. The model catalog indicates which quantizations fit in your memory and how fast they will run.

Magnitude offers a clean and easy-to-use interface

Connecting to agents is one click. Pi, OpenCode, Hermes, Codex, OpenClaw, Claude Code and Cline appear as direct destinations, and any other tool speaks through an OpenAI-compatible API. The engine starts models on demand when the agent needs them and shuts them down after a period of inactivity, without you having to monitor processes.

The typical flow is short. You download a model from the Discover catalog, wait for the minute of tuning, connect your agent in the connections tab and get to work. The app evaluates which variants fit in your hardware so you can choose wisely between speed and intelligence, instead of trying blindly until something fits.

An interesting technical detail is speculative decoding with a drafter assigned per model. Each model in the catalog comes with its recommended draft model, with modern speculation methods that accelerate generation without changing the result. It is one of those invisible optimizations that show up in day-to-day fluidity.

KV cache quantization follows the same philosophy. Keys at 8 bits and values at 4 bits cut long-context memory by more than half, accelerate decoding and, according to their internal tests, do not degrade coherence in extensive contexts. For agents that accumulate history with tools, that saving decides whether the session fits or not.

The wonder of Open Source

The project is published under the Apache 2.0 license and is written in Rust, including its own GPU kernel runtime with self-tuning. The ideas draw from academia and from the best open engines, with Flash attention, speculative inference and adapted radix attention. No black boxes, the code can be read and improved by anyone.

The business model looks to the hybrid future. Today the focus is the free local engine, and tomorrow an inference cloud will arrive for mixed workloads, with fluid switching between local and cloud without breaking the prefix cache. They will charge per token in that cloud, passing the engine's efficiencies on to the price. It is a reasonable plan, selling convenience when you need it without closing the code.

The public roadmap mentions three fronts. Expert streaming to run mixture of experts models larger than your VRAM, loading experts from RAM or disk just in time. A kernel compiler that fuses operations automatically. And multi-device use that distributes the load between CPU, GPUs, RAM and disk according to the detected topology.

The community responded with sharp questions at launch. About Vulkan on AMD, about the comparison with ROCm, about support for new backends and about whether the tuning is an agent optimizing kernels. The founders answered turn by turn, with Vulkan benchmarks underway and the promise to add backends if Vulkan does not reach the ceiling. That transparency during front-page hours says a lot about the team. Aspect Magnitude
llama.cpp Philosophy Kernels tuned on your chip
Compatible generic binaries Metal decode 57 versus 30 tokens per second
Test reference CUDA decode 58 versus 49 tokens per second
Test reference Memory per agent 27% less with quantized KV cache
Test reference License Apache 2.0
MIT Agent connection One click plus OpenAI API

Manual per agent

PowerShell on Windows, installation guide and practical uses

Redefining the future of local AI

The deeper reading is that local AI stopped being a hobbyist's whim. With agents living on your machine and open models improving every quarter, the bottleneck is no longer the weight of the models but the mileage the engine gets out of your chip. Tuning at destination, instead of averaging at origin, seems the right direction for a hardware park as diverse as today's.

José Maldonado

Author

José Maldonado

Activist and blogger on technology, free software and blockchain.

The latest
From Huang to Botín, from Zara to Byron: corporate responsibility in global capitalism

Keep reading