No nvme drives on Dell R7515

[nick17bo]

I installed FreeNAS-11.3-U4.1 on a Dell R7515 server with an AMD 7542 CPU.
The system is working fine but no NVMe drives can be found.

I have installed Ubuntu Server 20.04. The NVMe drives are all recognized and work properly.

Does anyone have any idea what I could try?

 

[nick17bo]

I installed the NVMe drives on a Dell R640 with FreeNAS-11.2-U6. The drive work properly.

 

[nick17bo]

I installed TrueNAS-12.0-MASTER-202006220424
same problem, no NVMe Drivers

 

[Samuel Tai]

According to the R7515 manual, p. 20, only the following NVMe devices are supported:

Code:

The following table shows the supported NVMe SSDs for the PowerEdge R7515:

Table 9. Supported NVMe SSD

  Description
  SSDR,1.6,NVMEPCIE,2.5,PM1725B
  SSDR,6.4,NVMEPCIE,2.5,PM1725B
  CRD,CTL,NVME,1.6,HHHL,PM1725B
  SSDR,3.2,NVMEPCIE,2.5,PM1725B
  SSDR,12.8,NVMEPCIE,2.5,PM1725B
  CRD,CTL,NVME,3.2,HHHL,PM1725B
  CRD,CTL,NVME,6.4,HHHL,PM1725B
  SSDR,960GB,NVMEPCIE,2.5,CD5
  SSDR,3.84TB,NVMEPCIE,2.5,CD5
  SSDR,1.92TB,NVMEPCIE,2.5,CD5
  SSDR,7.68TB,NVMEPCIE,2.5,CD5
  SSDR,1TB,NVME,PCIE,2.5,P4510
  SSDR,2TB,NVME,PCIE,2.5,P4510
  SSDR,4TB,NVME,PCIE,2.5,P4510
  SSDR,8TB,NVME,PCIE,2.5,P4510
  SSDR,1.6TB,NVME,PCIE,2.5,P4610
  SSDR,3.2TB,NVME,PCIE,2.5,P4610
  SSDR,6.4TB,NVME,PCIE,2.5,P4610
  SSDR,375GB,NVME,PCIE,2.5,P4800
  SSDR,750GB,NVME,PCIE,2.5,P4800
  CRD,CTL,PCIE,750GB,HHHL,P4800

Welcome to the wonderful world of vendor lock-in!

 

[nick17bo]

The R7515 works with my NVMe Micron 9300 on Ubuntu 20.04 fine, so I think there is no hardware problem ....

 

[Alecmascot]

The R7515 works with my NVMe Micron 9300 on Ubuntu 20.04 fin, so I think there is so hardware problem ....

Install FreeBSD and see what happens.

 

[nick17bo]

With FreeBSD 12.1 same result, no NVMe Drives
nvmecontrol devlist -> No NVMe controllers found

Missing driver???

 

[nick17bo]

I installed Ubuntu again and ran lshw to list the device dependency. Here is the output:
Does somebody has any idea?

 

[nick17bo]

Here are the loaded kernel modules

 

[nick17bo]

The kernel module megaraid_sas looks interesting

Hardware -> PCI
Numeric ID (from LKDDb) and names (from pci.ids) of recognized devices:

• ....
• vendor: 1028 ("Dell"), device: 000e ("PowerEdge Expandable RAID controller 4/Di"), subvendor: 1028, subdevice: 0123
• vendor: 1028 ("Dell"), device: 000e ("PowerEdge Expandable RAID controller 4/Di"), subvendor: 1028, subdevice: 0123
• vendor: 1028 ("Dell"), device: 000f ("PowerEdge Expandable RAID controller 4/Di"), subvendor: 1028, subdevice: 014a ("PowerEdge 1750")
• vendor: 1028 ("Dell"), device: 000f ("PowerEdge Expandable RAID controller 4/Di"), subvendor: 1028, subdevice: 014a ("PowerEdge 1750")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016c ("PowerEdge Expandable RAID Controller 4e/Si")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016c ("PowerEdge Expandable RAID Controller 4e/Si")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016d ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016d ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016e ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016e ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016f ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 016f ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 0170 ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0013 ("PowerEdge Expandable RAID controller 4"), subvendor: 1028, subdevice: 0170 ("PowerEdge Expandable RAID Controller 4e/Di")
• vendor: 1028 ("Dell"), device: 0015 ("PowerEdge Expandable RAID controller 5")
• vendor: 1028 ("Dell"), device: 0015 ("PowerEdge Expandable RAID controller 5")
• vendor: 1028 ("Dell"), device: 0015 ("PowerEdge Expandable RAID controller 5")

 

[nick17bo]

Hm, LSI MegaRAID SAS Support by default (freeBSD)

 

[nick17bo]

It really doesn't seem to work on FreeBSD. I installed Ubuntu again. Everything is recognized. Then I'll have to stick with it. What a shame

 

[mav@]

@nick17bo There are some reports about issues in hot-plug support on AMD EPYCs. Try to set loader tunable hw.pci.enable_pcie_hp to 0. If it help, then it should be fixed in 12.0-RC1 in September.

 

[nick17bo]

I try the 12.0-BETA (07/29/20) and it didn't work

 

[mav@]

What exactly didn't work? Have you set the mentioned tunable? Have you tried recent nightly builds?

 

[nick17bo]

Hey Mav,
thank you very much for your support. The system is a productive system and so unfortunately I cannot try the mentioned tunable at the moment (Ubuntu is currently running).
I've tested it with TrueNAS-12.0-MASTER-202006220424 and TrueNAS-12.1-MASTER-202007280438, as well as FreeBSD-12.1-RELEASE-amd64. Unfortunately neither version worked.