SOLVED Intel P4510 NVMe Format 4k

[dxun]

In preparation for a new VM pool on TrueNAS 12-U8.1, I've been going through OpenZFS docs and I came across the section around NVMe low-level formatting LBA size. After a quick scan through the inventory of disks I have, I've discovered only a pair of 1 TB Intel P4510 seem to support LBA 4k format (sadly, Intel chose to hobble the Optane 900p in this regard, similarly to how they chose to handicap it with a single NVMe namespace).

Since the OpenZFS docs mention Linux commands to format the drive to 4k, I've looked into FreeBSD analouges and came across `nvmecontrol`.

So, inspecting the P4510, here is what I get:

root@storage[~]# nvmecontrol identify nvme0 | grep 'LBA Format'
root@storage[~]# nvmecontrol identify nvme0ns1 | grep 'LBA Format'
Number of LBA Formats: 2
Current LBA Format: LBA Format #00
LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Good
LBA Format #01: Data Size: 4096 Metadata Size: 0 Performance: Best

The drive looks configured for 512 sector size and I tried to use nvmecontrol format but the man page has kind of spartan explanations that don't really show any examples how to this. Digging deeper, the NVMe Rev 1.4 spec indicates Format NVMe command (section 5-23, pp. 234) the format bit values so doing something like this should work:

root@storage[~]# nvmecontrol format -f 1 nvme0ns1
root@storage[~]# nvmecontrol identify nvme0ns1 | grep 'LBA Format'
Number of LBA Formats: 2
Current LBA Format: LBA Format #01
LBA Format #00: Data Size: 512 Metadata Size: 0 Performance: Good
LBA Format #01: Data Size: 4096 Metadata Size: 0 Performance: Best

And it does, except now I get the dreaded Error: [Errno 5] Input/output error when trying to Wipe Disk from TrueNAS UI. Setting the disk back to 512 LBA format makes the disk usable again.

Next step I had in mind was to try and re-format the whole disk (not just the only NVMe namespace) to 4k. Looking more closely at the nvmecontrol docs (such as they are), there is a remark When formatting all namespaces, all parameters should be specified. so I looked back at the NVMe spec to figure out the remaining parameters but didn't feel comfortable setting these by hand as it wasn't entirely clear what to use and I am concerned that I could throw the drive into a bad state that I'd find difficult to dig myself out with the knowledge that I have. The best I could come up with was something like nvmecontrol format -f 1 -m 0 -o 0 -l 0 nvme0 but I didn't run this.

What is the best way to format an NVMe SSD to 4k LBA size on TrueNAS?

 

[dxun]

Success - I was able to re-format drives to 4k using nvmecontrol, wipe them, create a pool, fill it up to max, do a scrub without errors and then export + destroy it.

Here is the how I did it for each disk (this, of course, destroys all data on a disk):

• gpart destroy -F <device_name>
• nvmecontrol format -f 1 <device_name>
• wipe <device_name> from UI => observe "Operation not permitted" error
• shut down the server + power up (i.e. hard reset)
• wipe <device_name> from UI => executes without errors
• gpart create -s GPT <device_name>
• create pool => operation executed wihout errors
• fill pool => operation executed wihout errors
• scrub pool => operation executed wihout errors
• export + destroy pool => operation executed wihout errors

Open questions:

• is there a way to do this without reboot?
  • looks like this may be a feature of either firmware or NVMe spec
• NVMe metadata and Protection Information is unset
  • I am not sure whether this is needed or if ZFS would tolerate these set to 1
  • the latter looks like a handy way of enforcing a CRC check on low-level, controller level
  • unsure if that would collide with ZFS' error controls checks? I would expect it wouldn't as the parity check happens way before the data reaches software abstraction layers