NVMe 2.1 Specifications Published With New Capabilities

At the Flash Memory Summit 2024 this week, NVM Express published the NVMe 2.1 specifications, which hope to enhance storage unification across AI, cloud, client, and enterprise. Phoronix's Michael Larabel writes: New NVMe capabilities with the revised specifications include:

- Enabling live migration of PCIe NVMe controllers between NVM subsystems.
- New host-directed data placement for SSDs that simplifies ecosystem integration and is backwards compatible with previous NVMe specifications.
- Support for offloading some host processing to NVMe storage devices.
- A network boot mechanism for NVMe over Fabrics (NVMe-oF).
- Support for NVMe over Fabrics zoning.
- Ability to provide host management of encryption keys and highly granular encryption with Key Per I/O.
- Security enhancements such as support for TLS 1.3, a centralized authentication verification entity for DH-HMAC-CHAP, and post sanitization media verification.
- Management enhancements including support for high availability out-of-band management, management over I3C, out-of-band management asynchronous events and dynamic creation of exported NVM subsystems from underlying NVM subsystem physical resources. You can learn more about these updates at NVMExpress.org.

Re:

[Jeremi]

Your "fact" sounds more like an article of faith. Perhaps you'd like to back it up with some evidence or explanation?

Re:"post sanitization media verification"?

[ctilsie242]

Depends on how "sanitized" one wants to go. For physical media and compliance, yes, one has to destroy all media. However, almost all SSDs have built in encryption, either OPAL, or some other standard. So, doing a nuke of the drive, a secure erase using blkdiscard, or nvme format will ensure all the data that was on the drive is gone, especially if the controller takes the time to free up all the SSD's pages, ensuring data written is now completely gone.

There are times where this can come in handy. For example, if servers are being auctioned off, this should be something kicked off in BIOS.

Sometimes parts of the SSD can be securely erased, which is important for virtualization, so if one VM is deleted or moved, the space it took up on the SSD array is effectively zeroized, so it can't be recovered.

Overall, it is a good thing to have, but it definitely won't replace good ol' hardware compression or a shredder when it comes to disposing things as per a lot of standards.

Re:

[omnichad]

Sometimes parts of the SSD can be securely erased, which is important for virtualization, so if one VM is deleted or moved, the space it took up on the SSD array is effectively zeroized, so it can't be recovered.

NVMe drives support multiple namespaces and I almost never see this used. Though if the drive implements the security correctly this would make what you described very easy. Each virtual drive is just a namespace on the SSD, wear leveling blocks with the other namespaces. And then each would have a unique encryption key.

Although you don't need hardware support if the VM drives are encrypted at the software layer. It just adds a little more overhead.

Re:

[omnichad]

There are already plenty of drives that are always encrypted. A secure erase just involves telling the drive to throw away the key and generate a new one, marking all blocks as unused. No extra media wear.

Re:

[torkus]

Yah, I don't see why secure erase confuses people or is considered problematic.

With functional encryption, you simply need to erase they key. Unless you have a few trillion years the data is irrecoverable. The only exception would be a poor crypto implementation ... which I worry becomes more likely when you add all this extra complexity. But otherwise is quite straight forward and common.

Not exactly shiny but a lot of nice features...

[ctilsie242]

Offloading I/O onto the NVMe components is a nice thing. This means the CPU works less, because the NVMe controller can handle some items. This reminds me of the old school IBM DASD where the disk wasn't just a "dumb" storage device, but had its own processors as well.

Booting over NVMe is a really nice item. This means that the SAN can handle 100% of the I/O, and machines can be completely diskless. Instead of providing RAID to every machine via a BOSS card, one could clone the OS to a number of boot images, which will take up a small amount of space with deduplication and/or copy-on-write. With out of band management, those images can be backed up or even scanned for malware outside of the OS, which can be a win.

Overall, this is appearing to be a complete replacement for iSCSI, but with TLS based authentication (beats CHAP any day), encryption in transit, and handing off between controllers.

NVMe also offers the ability to separate a controller and storage, so a presented LUN can be split up into a chunk of disk for the boot OS, and a chunk for the data, and each storage device handled separately.

Overall, it is definitely going to be competition to iSCSI.

Re:

[Joe_Dragon]

but iscsi direct to disk? vs iscsi to some kind of raid system?

Re:

[omnichad]

This already exists for large NVMe SAN systems. It's called NVMe-oF. The only difference with the new revision is boot support.

iSCSI is just another network protocol that already supported boot but wasn't as good for anything else with faster flash storage.

Re:

[Compaq Disk Rereader]

Support for offloading some host processing to NVMe storage devices.

Yeah this was intriguing and made me think of mainframe stuff. Hopefully we see more mainframe concepts coming into the PC world again now that we're bumping up against harder wins on the CPU/GPU space, still a lot of room for improvements in other parts of the machine especially as the added complexity gets cheaper and cheaper to integrate.

Re:

[PeeAitchPee]

Heat is already a big problem with the latest gen NVMe drives. The spec may support offloads but if they keep moving more stuff to the controller they're going to have to come up with different ways to cool the package (huh huh).

Re:

[JBMcB]

This reminds me of the old school IBM DASD where the disk wasn't just a "dumb" storage device, but had its own processors as well.

DASD controllers are one of the reasons modern mainframes can handle tens of thousands of transactions a second. They understand file systems, so an application on a mainframe's CPU can ask for a record in a database, and the DASD controller will look up the location of that record and drop it directly into memory without the CPU context switching. This is one of those performance roadbumps that make no difference at all when doing regular desktop or server things, but make a huge difference when you are se

Re:

[ctilsie242]

Even on regular desktops or servers, being able to throw something into RAM without needing CPU is important. Things like games which load textures would greatly benefit from this, where textures can be scooped directly from the SSD and stuffed into the GPU without needing the CPU to assist.

Zoned NVMe drives would be great. This would completely get rid of fiber channel, while still providing the stability of having dedicated storage fabric, where no matter what compromises the network, the zones will be

It's the bus bandwidth

[PeeAitchPee]

For years, we were stuck on PCIe 3. This effectively limited max throughput from a single x16 slot to 32 MB/s. This wasn't much of a problem internally but e.g. did limit the effective top speed of *networked* spinning rust large RAID arrays and early SSDs. Same thing with SATA-based SSDs -- OK for consumers but the interface forces you to leave a lot of the device's inherent performance on the table. The current generation of NVMe devices are so fast that a single drive can now saturate a 100G duplex fiber NIC in a single PCIe 4 x16 slot. That's never before been possible. We're now rapidly progressing through PCIe 4 and PCIe 5, both of which are mainstream. We should start to see the first commercially available PCIe 6 products sometime in 2025. In all, it's the greater overall bus bandwidth that's making the difference more that any other components in the chain. It's fascinating to see the bottleneck shift to other parts of the infrastructure as we can finally run these devices at full speed, as well as more of them in the same machine.

Re:

[omnichad]

You're right, but your numbers are wrong. An ancient PCI bus is 132MB/s. A gen 3 x16 slot gets 16GB/s.

Re:

[PeeAitchPee]

Yeah, typo . . . PCIe 3 x16 is 32 *GB/s* duplex.

What are Fabrics?

[GlobalEcho]

- A network boot mechanism for NVMe over Fabrics (NVMe-oF).
- Support for NVMe over Fabrics zoning.

For those of us too lazy to use a search engine or LLM, what is this Fabrics thing?

Re:

[PeeAitchPee]

It means a bunch of NVME drives networked together via Ethernet, Fibre Channel, etc.

No, AI Does Not Need to Be Every Title

[BrendaEM]

"AI" has jumped the shark.