host adapter does not work when nvme adaptor is installed

[dusko.nikolic]

Hello,

I have LSI SAS 9300 host adapter and it is working just fine, but when I install an NVME PCI Expander card with two NVMEs TrueNAS can't see SAS adapter any more.

NVME card is Glotrends with ASM2821 controller. The one that doesn't need PCI bifurcation in order to work.

I tried installing it in different PCI slots but no change, only NVME card is seen by TrueNAS.


Any ideas what can be the reason?

 

[Ericloewe]

Since you haven't told us much of anything about how system, we can only throw guesses at you.

This could be as simple as the two slots sharing PCIe lanes.

 

[dusko.nikolic]

Since you haven't told us much of anything about how system, we can only throw guesses at you.

This could be as simple as the two slots sharing PCIe lanes.

Motherboard is ASRock E3C246D4U2-2L2T it has 4 PCIE slots. LSI SAS is in slot 4 and NVME expander I tried in all other slots apart from slot 7 where it will not fit.

Slot 6 - 16X
Slot 4 - 8x
Slot 5 - 4x
Slot 7 - 1x

Slot 4 and slot 6 share PCI lanes but slot 6 should be 8 lanes when there is something in slot 4, otherwise it is 16 lanes.

since slots 4 and 5 do not share lanes it should work but it does not.

TrueNAS only sees two nvme drives once the expander card is placed in any slot

Processor is Xeon 2174G.

 

[Ericloewe]

Sounds like the system firmware may be misconfiguring the PCIe root complexes then. It could conceivably be that the PCIe switch is confusing the system firmware by putting out garbage on the SMBus...

Is slot 5 provided by the PCH or the CPU?

 

[dusko.nikolic]

This is from the manual for PCIe slots

NVME adapter does not have any bios or anything that I could change settings. It has supposedly a controller that does not rely on motherboard bifurcation feature. https://www.amazon.com/dp/B09NQGZ2XJ?psc=1&ref=ppx_yo2ov_dt_b_product_details

SAS adapter does have a bios which is accessible with ctrl+c during boot, however I honestly have no clue what to change there even after reading about different settings that are available

 

[Ericloewe]

NVME adapter does not have any bios

SAS adapter does have a bios

Option ROM. Not likely relevant to your current issue. As a sidenote, option ROMs that prompt you to interrupt the boot process are highly indicative of legacy BIOS, which means you should probably disable all the CSM crap unless there's a good reason not to do so. UEFI option ROMs are supposed to extend the native system firmware setup menu and provide any configuration options through there - this is case for LSI SAS controllers.

It has supposedly a controller

It's not a controller, it's a PCIe switch. PCIe switches are far from exotic and should just work - which is more or less your experience apart from the disappearing SAS controller.

So, where does that leave you? It's looking like either a bug in the system firmware or, less likely, some conflict on the SMBus - I say less likely because I can't see why a Super China Happy Sun card would implement anything on the SMBus...

There are a couple of things you can try:

• Move the SAS controller to Slot 6. This is silly, but has been known to work on other systems.
• Update the system firmware - big potential if you're not at the latest version.
  • If not, you can report this to ASRock and see if you can grab their interest.

 

[dusko.nikolic]

Ok. Thanks for the effort. I bought card and nvme drives to use for ZFS log but after some reading I realised that I don't really need that. Anyway, I will try them on windows machine

Thanks again.