TruNAS 13.0-U4 - R730xd - HBA330 - Unable to See NVMe Drives

[Matt X]

Hello everyone,

I have a Dell PowerEdge R730xd. Installed an HBA330 and two SATA drives for the boot pool. TruNAS installed without a problem.

However, I have 12 NVMe drives for the main storage pool in front slots 2-13 and iDrac, TruNas, etc do not see the drives.

Unsure if I have a setting incorrect, need to buy more hardware, etc.

• All NVMe drives' status lights just keep flashing in unison on the server
• iDrac doesn't show the bays as containing drives or as empty, just skips over those bay #s when looking at storage section
• Booting into HBA util doesn't see NVMe drives
• HBA seems to be in IT mode
• I updated Dell firmware, reseated drives, all the normal baseline stuff

I searched the forum and saw R730xd / NVMe drives come up, but these are in 2.5" format and loaded into the front slots, so I am not sure the recommended PCIe card solution would work.

More relevant info:

TruNAS Version: TrueNAS-13.0-U4

Server: PowerEdge R730xd
BIOS Version: 2.16.0
Firmware Version: 2.84.84.84

Controller: Dell HBA330 Mini (Embedded)
Firmware: 16.17.01.00

Boot drives: 256GB SATA SSD x 2 (slot 0, 1)

Storage drives: 4 TB WD Ultrastar DC SN630 NVMe SSD x 12 (slot 2-13)

I can provide screenshots or any other information required.

Thank you for your help!

 

[NugentS]

How are the NVMe drives attached to the motherboard?

 

[Arwen]

Using U.2 NVMe, such as the WD Ultrastar DC SN630 NVMe SSD, requires that the slot, back plane, expander if any, cables and a special HBA. The "special" HBA either be a PCIe type or a tri-mode HBA that supports SATA, SAS or PCIe. Only the latest generation of LSI / Broadcom / Avago HBAs have such models.

I don't know if the Dell HBA330 Mini (Embedded) is a tri-mode HBA.

So, @NugentS question is quite important.

 

[NugentS]

I was wondering. I don't know the R730XD Hardware - but it doesn't support NVMe in the front slots out of the box as far as I can tell.
An H330 is NOT a TriMode HBA - its SAS (and SATA)

You will need an awful lot of lanes for 12 NVMe drives

 

[Arwen]

An 8 or 16 lane PCIe switch or tri-mode HBA should work fine for general use. Now if the application needs higher throughput or even full throughput to each U.2 NVMe drive, well, that is a different. In comes high PCIe lane CPUs, like Epyc or some of the higher end Intels.

For such uses, a PCIe switch still may be required because the PCIe slots may not be bifurcatable into x4 lanes. A PCIe switch would not add that much overhead to go from say 8 lanes to 2 x 4 lanes. Or 16 lanes to 4 x 4 lanes. Should be little to no loss in through put as the host side is not over loaded.

 

[jenksdrummer]

Hello everyone,

I have a Dell PowerEdge R730xd. Installed an HBA330 and two SATA drives for the boot pool. TruNAS installed without a problem.

However, I have 12 NVMe drives for the main storage pool in front slots 2-13 and iDrac, TruNas, etc do not see the drives.

Unsure if I have a setting incorrect, need to buy more hardware, etc.

• All NVMe drives' status lights just keep flashing in unison on the server
• iDrac doesn't show the bays as containing drives or as empty, just skips over those bay #s when looking at storage section
• Booting into HBA util doesn't see NVMe drives
• HBA seems to be in IT mode
• I updated Dell firmware, reseated drives, all the normal baseline stuff

I searched the forum and saw R730xd / NVMe drives come up, but these are in 2.5" format and loaded into the front slots, so I am not sure the recommended PCIe card solution would work.

More relevant info:

TruNAS Version: TrueNAS-13.0-U4

Server: PowerEdge R730xd
BIOS Version: 2.16.0
Firmware Version: 2.84.84.84

Controller: Dell HBA330 Mini (Embedded)
Firmware: 16.17.01.00

Boot drives: 256GB SATA SSD x 2 (slot 0, 1)

Storage drives: 4 TB WD Ultrastar DC SN630 NVMe SSD x 12 (slot 2-13)

I can provide screenshots or any other information required.

Thank you for your help!

If iDRAC or BIOS doesn't show them, they aren't compatible.

Ran into this even with spindle disks, some Dell boxes (firmware even) will blacklist hardware.

 

[HoneyBadger]

How are the NVMe drives attached to the motherboard?

As identified further down, the concise answer is "they aren't" - despite sharing a similar form factor to 2.5" SATA/SAS SSDs, 2.5" NVMe U.2 drives require a backplane and connection that is compatible with the NVMe protocol.

Only slots 20-23 on the backplane have the necessary pins to be compatible with the U.2 format, but the R730xd itself also likely requires Dell part number 0P31H2 (a PCIe x16 to 4x4 breakout board) in order to actually connect those PCIe lanes through to the motherboard.

 

[Matt X]

Sounds like we need to swap out the NVMe drives for more SATA drives to get this in order (please correct me if I am wrong).

Thank you everyone, sincerely appreciate it!

 

[Matt X]

@NugentS I have pictures showing how everything is connected, but basically it's the baseline R730xd build with the PERC replaced with the HBA. Did not put in any other cards, cables, etc.

 

[Ericloewe]

The R730XD supports up to four NVMe drives in the front bays - in specific slots, of course, but these need to be wired up to PCIe (possibly using a retimer card - switches are not necessary since the system can bifurcate down to x4/x4/x4/x4). If your backplane does not already have four SFF-8643 cables for NVMe, in addition to the two for SATA/SAS, you'll need to add them and connect them up to a PCIe x16 slot.

 

[woodb01]

The R730XD supports up to four NVMe drives in the front bays - in specific slots, of course, but these need to be wired up to PCIe (possibly using a retimer card - switches are not necessary since the system can bifurcate down to x4/x4/x4/x4). If your backplane does not already have four SFF-8643 cables for NVMe, in addition to the two for SATA/SAS, you'll need to add them and connect them up to a PCIe x16 slot.

I have an R730XD as well. The U.2 drives will only work in the last 4 slots, and ONLY if you have the PCIe / NVME enablement kit. That is a dell card and 4 jacks to plug into the last 4 jacks on the far right side of the backplane. Those kits and cables aren't cheap, generally around $150-$200 on ebay with the cables. You can buy the card and cables separately, and they often source from China, and maybe get around $120 but you will wait 2 or 3 weeks.

 

[Matt X]

Thank you for the info everyone. I am new to TruNas and gathering the hardware is a bit overwhelming.

Is there a Dell PowerEdge RXXX model that will work with the U.2 drives that I currently have (4 TB WD Ultrastar DC SN630 NVMe SSD x 12 (slot 2-13)) with minimal purchasing of other components (HBA, etc)? I am happy to buy another PowerEdge to get this project working w all drives.

In other words, what is the shortest path from where I am (w/ this pile of drives) to where I want to be (the drives working inside of a Dell PowerEdge R??? and running TruNas)?

 

[woodb01]

Thank you for the info everyone. I am new to TruNas and gathering the hardware is a bit overwhelming.

Is there a Dell PowerEdge RXXX model that will work with the U.2 drives that I currently have (4 TB WD Ultrastar DC SN630 NVMe SSD x 12 (slot 2-13)) with minimal purchasing of other components (HBA, etc)? I am happy to buy another PowerEdge to get this project working w all drives.

In other words, what is the shortest path from where I am (w/ this pile of drives) to where I want to be (the drives working inside of a Dell PowerEdge R??? and running TruNas)?

If you have an R730XD there should be 4 jacks on the right side of the backplane, that you hook the NVME card up to. Then the U.2 drives work fine. In fact, I have 4 of the WD SN630 3.84GB drives in my own right now. Here is the link to purchase the enablement kit. If you have a normal R730XD it should have the nvme / U.2 ready. https://www.ebay.com/itm/143653884917

However, this will only support 4 drives. My personal suggestion would be that if you have 12 drives maybe you should consider selling 8 of them. What I have found is that good SAS SSD 12Gb drives are about the same price and have similar performance. A last option might be to do some research on an R740XD. I may be wrong but I think some of them support more than 4 NVME/U.2 drives but I don't know.

 

[woodb01]

Just checked, the R740XD has 12 NVME enabled bays rather than the R730XD with 4 bays. Now, I don't know if that is native on the R740XD or if you need an enablement card and cables for that. However, the info says the NVME/U.2 bays are supported.

For more info on the R740XD and NVME/U.2, check out this Dell thread: https://www.dell.com/community/en/c...oweredge-r740xd-nvme/647f9b29f4ccf8a8deefe03f