Latest

Technology

Linux 7.4: NVMe namespaces in parallel, one second less

A Linux 7.4 patch deletes NVMe namespaces in parallel: with 64 volumes, disconnecting an NVMe-oF target took over a second and now the waits overlap.

Translated from Spanish by an automated system. The Spanish original prevails. Read the Spanish original

Linux 7.4: NVMe namespaces in parallel, one second less
Linux 7.4: NVMe namespaces in parallel, one second less

A server with 64 NVMe namespaces takes more than a second to say goodbye to its controller, and that second does not come from the disk, but from a kernel wait that repeats once per volume. The patch Qiliang Yuan has submitted for Linux 7.4 changes the order of that goodbye: instead of deleting the namespaces one by one, it deletes them in parallel and waits only once for all of them to finish.

The problem is not new and affects something as everyday as disconnecting an array. When an administrator unmounts an NVMe over Fabrics target, or when a fiber link goes down, the kernel walks the controller's namespace list and removes each one. Each removal waits for a grace period of the lock-free read mechanism, and that period is paid for every entry in the list.

The improvement matters because modern storage is organized exactly that way. An array with dozens of logical volumes, a container platform with disks separated per tenant or a multitenant deployment over NVMe-oF depend on reconfiguration being fast, and every second of waiting becomes timeout time upstream.

The 20 ms of the "grace period"

The Linux kernel protects its structures with a mechanism called RCU, which allows reading shared data without blocking the writer. Before freeing a structure, whoever deletes it has to wait until no reader can be using it, and that interval is called a grace period. It is a safety guarantee, not a luxury, and that is why it cannot simply be removed.

In the namespace deletion path, that wait appears twice: once for the namespace itself and once for its multipath header. Each one costs around 20 milliseconds, a figure that is invisible on a single volume and that, multiplied by 64, becomes more than a second of blocking before the controller is freed.

The detail that makes the fix possible lies in the global nature of the grace period. It is not a per-object or per-process timer, but a property of the whole system, so several waits occurring at the same time share the same round and are resolved together.

The fix, in one sentence

The patch creates an asynchronous domain that launches the deletion of all namespaces at once and waits for them to finish before returning control. The 20 milliseconds per namespace still exist, but they stop adding up. The NVMe pull request message for 7.4 accompanies it with other fixes from the same series. The author's explanation is quoted in the ITHome technical summary with the full arithmetic of the timings.

The idea was copied from an earlier change and that says a lot about the subsystem's style. Commit 4e893ca81170, titled "nvme_core: scan namespaces asynchronously", had already solved the equivalent problem in the scan path, when the system detects the volumes of a new controller. The same approach was then applied to deletion and the structure was kept.

Scenario Before After
Controller with 64 namespaces More than 1 second due to serial waits A single shared wait round
Disconnection of an NVMe-oF target Blocks the deletion path Overlaps the waits of each namespace
ANA state change with many volumes Tens or hundreds of milliseconds No wait per namespace

The third row does not come from the same patch, but it shares the cause and appears in the same series of fixes. The multipath path state change called the function that marks a path as active for each namespace, and each call waited for its own grace period, so a state change on a large array could take hundreds of milliseconds.

What is most noticeable in production

The case that comes up most often in support conversations is not the clean unmount, but the dirty one. When a link is cut, the kernel discovers that the controller has disappeared and starts deleting all its volumes while the rest of the system keeps working. That second of waiting does not only stop the administrator: it affects the layers waiting for block confirmation.

For a cluster with frequent reconnections, the difference between a second and a few milliseconds decides whether the file system goes into read-only mode or whether the retry arrives in time. It is the kind of improvement that never appears in a performance headline and that is noticed precisely when something goes wrong.

The rest of the NVMe pull request for 7.4 also includes fixes worth mentioning, because they affect data paths. There is offset validation in R2T-type transfers, inline data and queue identifier checks, and a change in the handling of the old secret of the DH-HMAC-CHAP authentication protocol, which is now freed with the sensitive variant of the memory release function.

A leak of disk structures and a subsystem deadlock that appeared when the configuration of a namespace or a controller failed are also fixed. That last case is the typical failure that only reproduces with faulty hardware and that leaves the server unable to add volumes until the next reboot.

The debt of the slow paths

What is interesting about this patch is that it does not optimize the fast path, the one benchmarks measure and headlines celebrate. It optimizes the exit path, which is where the system gets stuck precisely when the operator is in a hurry and the user notices the problem.

That kind of work is hard to prioritize because it does not improve any visible figure. Nobody issues a press release because disconnecting an array takes 20 milliseconds instead of a second and, nevertheless, each of those seconds ends up in an open incident with storage as the suspect.

The review of the Linux 7.4 merge window already showed that this release is larger than usual in internal fixes, and the namespaces patch fits that pattern. The final version will arrive at the end of December 2026, and until then the fix will only be available to those who compile the kernel or use a distribution that backports it.

José Maldonado

Author

José Maldonado

Activist and blogger on technology, free software and blockchain.

The latest
When buying a home seems impossible, young people bet on crypto

Keep reading