Recommended PCIe to M.2 NVMe Adapter for Dell PowerEdge R730XD

[mmoo9154]

I saw @Dave Hob's thread.[1] I'm posting this one because that one was from 2020 and getting a little old.

I purchased a nice 4 slot ASUS adapter from Amazon.[2] Although it's usable once the R630XD is booted (e.g. from an external USB or from one of the SATA drives), the Dell box does not allow booting from the adapter. Unfortunately, it's not recognized/enumerated at boot time by the R730XD. I posted a message up on the PowerEdge forum and a Dell tech support said the R730XD can't even boot with the Dell BOSS-S1 card. ‍

@Dave Hob didn't explicitly say he was booting from the two adapters he said work with his R730XD. I'm wondering if anyone out in the TrueNAS community has successfully booted an R730XD from a nice fast MNVe M.2 SSD and would they recommend their adapter.

[1]: https://www.truenas.com/community/threads/dell-pe-r730xd-pcie-nvme-m2-recommandation.85961/
[2]: https://www.amazon.com/gp/product/B07NQBQB6Z/
[3]: https://www.dell.com/community/Powe...-Bootable-NVMe-Adapter-for-R730XD/m-p/8382877

 

[jgreco]

Correct. NVMe devices generally do not contain BIOS ROM's and are therefore only available as bootable devices if there is built-in support in the system BIOS. That ASUS card is super expensive; you could have just used an inexpensive AliExpress card commonly resold on eBay for about half that.

 

[mmoo9154]

Correct. NVMe devices generally do not contain BIOS ROM's and are therefore only available as bootable devices if there is built-in support in the system BIOS. That ASUS card is super expensive; you could have just used an inexpensive AliExpress card commonly resold on eBay for about half that.

Thanks @jgreco. I sent the ASUS card back, so I'm looking for suggestions. What AliExpress card do you use/recommend? I'm presuming that one has the BIOS ROM and is bootable.

I'd also like one that can hold more than one M.2 NVMe card.

 

[jgreco]

Thanks @jgreco. I sent the ASUS card back, so I'm looking for suggestions. What AliExpress card do you use/recommend?

These things are basically just a printed circuit board with some traces and some M.2 sockets, along with a bracket and some electronics to regulate power. This one here is $18. There is not a lot of use for a fan when mounted inside an R730XD, as the front-to-back airflow should be sufficient on its own (and you can always add heatsinks to the individual M.2's to make that better if needed). There is very little to go wrong with them, mostly because your mainboard is expected to do all the hard work, bifurcation support being the biggest part of that.

I'm presuming that one has the BIOS ROM and is bootable.

Unfortunately, I don't think you'll find that. Such a ROM would generally need to extend the BIOS to support NVMe, and need to handle bifurcation support and all that. There are paddleboards that add M.2 SATA by way of adding an AHCI SATA controller chip, and I've seen ones that do RAID and add a BIOS for the SATA RAID chip.

 

[mmoo9154]

Thanks again for the responses.

So... There's just no realistic way to boot TrueNAS from a hyper fast SSD in an R730XD? A SATA attached NVMe really defeats all of the speed benefits. Am I misunderstanding?

 

[jgreco]

So... There's just no realistic way to boot TrueNAS from a hyper fast SSD in an R730XD? A SATA attached NVMe really defeats all of the speed benefits. Am I misunderstanding?

Not necessarily impossible, just pricey or inconvenient.

Pricey options include finding a controller with BIOS. I believe this generally requires RAID, as a practical matter. For example, you can look at a device like the HPE NS204i-p, but I have no idea if TrueNAS will support whatever RAID chipset it uses (probably does NOT). You can also look at an LSI 9400-8i with two NVMe SSD's.

Inconvenient options would include booting from a small SATADOM or USB for just the kernel and then using the kernel's drivers to mount your root device.

 

[HoneyBadger]

Generally speaking, there's no need for the boot device in TrueNAS to be "hyper-fast" - I wouldn't expect a regular SATA SSD to be any manner of a bottleneck.

 

[jgreco]

Generally speaking, there's no need for the boot device in TrueNAS to be "hyper-fast" - I wouldn't expect a regular SATA SSD to be any manner of a bottleneck.

That's probably true as well. The nature of a UNIX system booting is that you have a lot of smaller modules and files being accessed, which tends to be more latency sensitive than speed constrained.

Increasing parallelism in the system can be accomplished by something like the technique I describe in the Resources section under the Highly Available Boot Pool article --

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

Because SATA SSD is both cheap and universally supported, you might find it a lot easier to go with two (or more!) SATA SSD's mirrored for your boot pool. This has the potential to go relatively fast if there is concurrent access going on, and reduced latency compared to booting from a single SATA SSD where the controller or SATA channel can become a bottleneck.

 

[HoneyBadger]

The R730XD does also support a rear-mounted 2x2.5" SFF bay option - however, adding it after-the-fact would probably require a non-trivial amount of expenditure to get the Dell OEM parts.

Anecdotally, the speed of a reboot of a single controller unit is significantly less of a concern to me than the downtime caused by having one - whether it's one minute or five minutes, I'll still need to stop any workloads on the unit or migrate them to other storage. That's where multiple units (or HA on Enterprise, of course) comes in.

 

[NickF]

SO...

I have asked this specific question many many times. I did some testing and research and spent way too much time and money trying to get an answer. There are cheap cards that need a motherboard that supports PCI-E Bifurcation, which outside of Supermicro, Asrock, and Gigabyte is hard to get in a server board. Then there are more expensive cards that will work on anything with a ful electrical PCI-E x16 slot (some work on x8) that have a PLX switch chip like you would find on a motherboard. Those cards are awesome.

I really like this particular card: https://www.aliexpress.us/item/3256...ayWqQXF&gatewayAdapt=glo2usa&_randl_shipto=US

Which uses the PEX8747 chip:

PEX 8747

The ExpressLane PEX 8747 is a 48-lane, 5-port, PCIe Gen 3 switch device developed on 40nm technology. The PEX 8747 device offers Multi-Host PCI Express switching capability enabling users to connect a host to its respective endpoints via scalable, high bandwidth, non-blocking interconnection to...

www.broadcom.com

Should be pretty turn key. Just have to use a PCI-E slot that is a real electrical x16 slot and not a x8 inside of a physical x16.

I did some performance testing over on L1techs. If you are going the AliExpress route, I can wholly recommend LinkReal as long as you buy from their official store. https://forum.level1techs.com/t/truenas-scale-performance-testing/187486

But really, if you just want to boot..

https://www.amazon.com/StarTech-com-SATA-USB-Cable-USB3S2SAT3CB/dp/B00HJZJI84/ref=sr_1_3?crid=Z5HJLCKIDF8R&keywords=usb+to+sata+adapter&qid=1682043980&sprefix=%2Caps%2C53&sr=8-3

These work fine with a decent brand name SSD.

 

[mmoo9154]

Thanks all for the replies. @NickF, I'll be looking at your recommended adapters today. @jgreco, I'm a little less price sensitive simply because this is a one time cost shared across 400+ users (at the moment). I'm just looking for something that works.

@HoneyBadger, I'm not looking for something that is "hyper-fast". Most of the latency is going to be in pushing packets around the net. We have a 2Gb internet connection through Lumen/CenturyLink to about half the staff, but the local staff are connected with two 10Gbps directly to the NAS.

What I'm thinking the advantages of the NVMe adapter would be are:

• "hyper-fast" write cache pool for TrueNAS
• "hyper-fast" swapfile for underlying Linux OS
• Allow all 24 SATA SSD's to be used for storage pools

Since the write cache is going to be so horribly unnecessary (except for when it's utterly necessary), it seems like the extra expense to minimize it's affect on writes would payoff.

The server has dual power units and dual independent battery backups. It should *never* ever need to use the write cache, but when the day comes that both power units fail, or both battery back ups fail after the main power fails, it will be nice we have it.

Also, running the swapfile off NVMe SSD seems like it would yield dramatic performance improvements in those situations where the swapfile is being hammered. It seems like the incremental cost would payoff especially if I've already purchased the bootable NVMe adapter for the reason above.

The third reason is mostly cosmetic (a "nice to have"), but I'd still like to get the boot image out of the 24 SATA SSD array.

I suppose it might be possible to configure a relatively tiny boot partition from the first SATA SSD, configure the swapfile and write cache pool to be on the non-bootable NVMe adapter, and then partition whatever unused space is left on that SATA SSD to make it available for TrueNAS storage. But, that seems gross since the leftover partition wouldn't be symmetric with anything else in the system.

Am I thinking about the write cache pool and swapfile incorrectly? Does anyone have experience or data on how NVMe speeds might help in those situations?

 

[jgreco]

The server has dual power units and dual independent battery backups. It should *never* ever need to use the write cache,

ZFS uses main memory for your write cache. You doo not seem to quite grasp how this works, so I recommend you might wish to review how this works in ZFS. People often mistake a SLOG device for a caching device; it is not. There is a ZFS fundamentals article in the Resources section.

 

[danb35]

a motherboard that supports PCI-E Bifurcation, which outside of Supermicro, Asrock, and Gigabyte is hard to get in a server board.

If I'm not mistaken, the Dell Gen 13 (i.e., the R/Tx30 series) and newer servers do support bifurcation, at least with current BIOS versions.

 

[Ericloewe]

Yeah, Gen 13 allows for full control of the bifurcation settings in the system firmware setup menu.

 

[NickF]

All of the gen 13 boxes I’ve had were in real production so I never checked. Only ever played with gen 12 on the bench. Nice to know

 

[woodb01]

SO...

I have asked this specific question many many times. I did some testing and research and spent way too much time and money trying to get an answer. There are cheap cards that need a motherboard that supports PCI-E Bifurcation, which outside of Supermicro, Asrock, and Gigabyte is hard to get in a server board. Then there are more expensive cards that will work on anything with a ful electrical PCI-E x16 slot (some work on x8) that have a PLX switch chip like you would find on a motherboard. Those cards are awesome.

I really like this particular card: https://www.aliexpress.us/item/3256...ayWqQXF&gatewayAdapt=glo2usa&_randl_shipto=US

Which uses the PEX8747 chip:

PEX 8747

The ExpressLane PEX 8747 is a 48-lane, 5-port, PCIe Gen 3 switch device developed on 40nm technology. The PEX 8747 device offers Multi-Host PCI Express switching capability enabling users to connect a host to its respective endpoints via scalable, high bandwidth, non-blocking interconnection to...

www.broadcom.com

Should be pretty turn key. Just have to use a PCI-E slot that is a real electrical x16 slot and not a x8 inside of a physical x16.

I did some performance testing over on L1techs. If you are going the AliExpress route, I can wholly recommend LinkReal as long as you buy from their official store. https://forum.level1techs.com/t/truenas-scale-performance-testing/187486

But really, if you just want to boot..

https://www.amazon.com/StarTech-com-SATA-USB-Cable-USB3S2SAT3CB/dp/B00HJZJI84/ref=sr_1_3?crid=Z5HJLCKIDF8R&keywords=usb+to+sata+adapter&qid=1682043980&sprefix=%2Caps%2C53&sr=8-3

These work fine with a decent brand name SSD.

I realize this post is a few months old, but for the future if anyone goes through this:

The Dell R730 and R730XD both support PCI-E bifurcation. The older R720 does not. How do I know? I have an R720, R730, and an R730XD. I have the bifurcation turned on with the R730 and run a Dell 2UP PCI-e NVME card (part # 23PX6 - link below on ebay). They are used and today sell for around $40-$60ish dollars. You can get a 4UP if you like. I *believe* they will allow you to boot, however, I have not tried that and have no need to do so.

23px6 for sale | eBay

Get the best deals for 23px6 at eBay.com. We have a great online selection at the lowest prices with Fast & Free shipping on many items!

www.ebay.com

P.S. Forgot, the R730XD in particular does have an NVME U.2 enablement kit available (not cheap) that DOES turn the last 4 bays into U.2 capable drive slots. They are all bootable with the enablement kit. There are U.2 empty drive cases for sale on eBay or on Amazon that you can pop your NVME M.2 drive into if you want to go that route. However, for the price of lightly used enterprise U.2 drives on eBay (> 90% life remaining) you are better off buying those used if that is what you are looking for. They can often be purchased cheaper than the NVME stick.

 

[Nixoid]

Dell DPWC300 P/N:0NTRCY Works on r730 and r730XD (test both server)

 

[danb35]

Forgot, the R730XD in particular does have an NVME U.2 enablement kit available (not cheap) that DOES turn the last 4 bays into U.2 capable drive slots.

I'm not sure if it's the same kit, but the R630 also has such a kit available.

 

[Nixoid]

And 10Gtek M.2 NGFF M-Key to PCIe x8 NVMe will work on r730/730xd, checked it today.

M.2 NGFF M-Key to PCIe x8 NVMe SSD Adapter Card 2230, 2242, 2260 and 2280 drives | eBay

Find many great new & used options and get the best deals for M.2 NGFF M-Key to PCIe x8 NVMe SSD Adapter Card 2230, 2242, 2260 and 2280 drives at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[danb35]

That link is broken.