X11SSM-F + AOC-SLG3-2M2 + Optane

[RSA]

Hello, i'm newbie in freenas forum )
i have problems with nvme drives on my system:
FreeNAS-11.1-U4
Supermicro Micro ATX DDR4 LGA 1151 Motherboard X11SSM-F-O last BIOS 2.1a
4 pcs of 16GB Module DDR4 2133MHz ECC Unbuffered
Xeon E3-1235L V5
Fractal Design Node 804 with Seasonic 460W power supply
5pcs WD80EZFX in RAIDz1

i also have AOC-SLG3-2M2 Supermicro card for 2 nvme drives
and today i got 2pcs of intel optane 16Gb drives for testing
first problem - the system can't start, stopped at POST screen and continuous beep. OK, np - go to read a lots of manual and got solution:
Disable the jumpers of SMB to PCI Slots (JI2C1/JI2C2), go to BIOS PEG Port Configuration >> change SLOT7 Bifurcation to [x4x4]
Now system can start normally !
But i can't find any new drives in freenas (
i don't know - its result of wrong BIOS settings or freenas...

 

[kdragon75]

Have you checked lspci in the shell? This will list all detected PCI devices. If you see them in there, try camcontrol devlist. This should list all storage devices.

 

[RSA]

lspci - not any nvme devices (((

 

[RSA]

i think reason is Optane 16Gb is x2 and AOC-SLG3-2M2 for x4 cards.

 

[RSA]

checked with ASUS Hyper M.2 mini card - all ok, system has 1 nvme drive, works fine

 

[kdragon75]

So you tested using one ASUS Hyper M.2 mini card in the AOC-SLG3-2M2 and it worked?

 

[RSA]

no no, ASUS Hyper M.2 is a card like AOC-SLG3-2M2, but for one nvme drive. AOC-SLG3-2M2 for two. And one Optane PCI x2 works fine with ASUS. So the problem is not BIOS settings.

 

[Ericloewe]

I think you need the SMBus to allow the system to configure PCIe bifurcation, which is needed to use multiple devices in one physical PCIe slot without a switch.

@Stux, you looked into this stuff a while back. Any insights?

 

[RSA]

if i try to switch on SMBUS (JI2C1/JI2C2 to enable) - system can't start (
found only one method to start system: JI2C1/JI2C2 to disable and manual switch SLOT7 Bifurcation to x4x4

 

[Stux]

I think you need the SMBus to allow the system to configure PCIe bifurcation, which is needed to use multiple devices in one physical PCIe slot without a switch.

@Stux, you looked into this stuff a while back. Any insights?

I don’t think the SMbus is used for bifurcation control. There have been issues with hw compatibility in the past which have been solved by taping over pins 3&4 (iirc) to disable smbus.

Bifurcation is essentially a bios setting. For it to work you should have retimer chips on the multi-card adapter.

No need for a full PCIe switch.

Unfortunately some of these multi m2 cards don’t seem to have the retimer chips.

SuperMicro *do* make a dual m2 card which does work.

My research focused on bifurcation on supermicro’s x10 platforms.

 

[RSA]

a lot of thanks for your support !
i need to try use it with PCIe x4 nvme m.2 drives, like Samsung 960 Pro..but need to find some money to buy )

 

[RSA]

Update )
today did some tests with 2x Samsung 960 Pro 512.
And good news - system work fine, both recognized, really super speed drive.
So, AOC-SLG3-2M2 can't work with Optane, reason - Optane is 2x line device and AOC-SLG3-2M2 support only 4x devices.
Any 2x nvme drives like Optane, few models Plextor - doesn't work with AOC-SLG3-2M2, but you can use these 2x devices with cards like ASUS Hyper x4 mini

If you have X11SSM-F and want to use 2 pci express nvme m.2 drive per slot (with AOC-SLG3-2M2 card) , you need to do:
1. BIOS update to 2.1a
2. Disable the jumpers of SMB to PCI Slots (JI2C1/JI2C2) - from 1-2 to 2-3 (need to do in some cases, not always, in my situation - reason is 4pcs of 16GB DIMMs and AOC-SLG3-2M2)
3. Change in BIOS SLOT7 Bifurcation to [x4x4]
4. Use only PCIe x4 devices !

 

[Stux]

Is any of that info from Supermicro support?

Sounds like something that could be fixed with a bios update.

 

[RSA]

i'm really hate supermicro support.
their recomendations: Base on AOC-SLG3-2M2 specs, the X11SSM-L is NOT support....(but in real AOC-SLG3-2M2 is listed as supported for X11SSM-F with BIOS 2.1a) or another one recomendation - install win10, go to diskpart, you will find disk and after just format drive...
Spend one week and got recomendations like from housewife 18 years old

 

[Stux]

Thinking about it further, the problem is probably that 2x2x etc bifurcation is not supported.

I know that some platforms support bifurcation down to 1x, but not these ones.

I’m curious if there is any combination of a 4x and 2x device in the AOC which would work.

 

[RSA]

i did some kind of investigation for AOC card )
it has 2 chip onboard: 8pin Atmel 732 SPI Serial EEPROMs and 20pin ICS9DBL411B
first chip - just EEPROM 256kb - i think some kind of firmware for AOC
second chip - Four Output Low Power Differential Buffer for PCI Express , it has no PLL but works directly with SMBus, uses like fanout buffer between PCIe and CPU clock. it has 4 lines, so i think it can support up to 4 devices.
And my suggestion - problems with PCIe devices from AOC firmware. Maybe 256kb is too small to add support for PCIe x2 and PCI x4 devices at one time and they do only for x4 :)))))
i don't have any supermicro mainboards to check it, but i will try to find. I think any PCIe x2 devices can't work with any Supermicro mobo and AOC-SLG3-2M2