PCIe->M2 Adapter for SLOG

[PhiloEpisteme]

Hi folks,

I'm considering upgrade paths for my system.

At the time of writing my system specs are as follows:

Case: Fractal Design Define R6 (Build Report)
FreeNAS Release: FreeNAS-11.2-RELEASE-U2
Board: SuperMicro X11SSM-F
Processor: Intel Core i3-7100
Memory: 32 GB
HBA: LSI/Broadcom SAS9207-8i
NIC: Chelsio T520-CR
Storage Pool 1: 1x RAIDZ2 vdev @ 6x3TB disks
Storage Pool 2 (encrypted): 1x RAIDZ2 vdev @ 6x2TB disks
Boot pool: 1x mirrored vdev @ 2x 64GB SATA DOM

My current PCIe slot usage is as follows

x8: LSI 9207-8i HBA
x8: Chelsi T520-CR 10Gbps NIC
x4: empty
x4: empty

Longer-term, I'd like to add SLOG devices as well as grow my storage space. I figure in a year or two I'll be in a position to need to upgrade my storage capacity and because my case is out of drive bays I'll likely make use of an additional chassis or replace the one I have.

Option 1
Keep my current chassis, add the mirrored SLOG devices via a PCIe8x -> 2x PCIe4x M2 adapter, and use an additional JBOD chassis w/expanders for extra HDDs.

Proposed PCIe slot usage
x8: LSI 9207-8e HBA (or faster/larger if ever a need arrises)
x8: AOC-SLG3-2M2 with 2x 100GB Intel Optane DC P4801X SSDs
x4: LSI 9207-8i HBA (this is running max 8 HDDs. The 4x port should have plenty of bandwidth)
x4: Chelsi T520-CR 10Gbps NIC (per an email from Chelsio, this card should perform just fine an x4 PCIe 3.0 slot)

Note that the add on card is not listed as compatible with my board but an email from supermicro states that it does work

Yes, the AOC-SLG3-2M2 can support on X11SSM-F with:
1. Install it on PCIe-Slot 7 -> 2x M.2 SSDs.
2. Install it on other PCIe slots -> only 1x M.2 SSD.

New Parts + Cost
AOC-SLG-2M2 - $40
2x 100GB DC P4801x - $590 ($290 each)
LSI-9207-8e - $60


Option 2
Keep my current chassis, add the mirrored SLOG devices via both x4 PCIe 3.0 slots, and use an additional JBOD chassis w/expanders for extra HDDs.

Proposed PCIe slot usage
x8: Broadcom/LSI 9400-8e8i Tri-Mode Storage Adapter
x4: Chelsi T520-CR 10Gbps NIC
x4: Intel Optane 900P SSD 280GB
x4: Intel Optane 900P SSD 280GB

New Parts + Cost
2x 280GB 900P - $510 ($255 each)
9400-8e8i - $550 I don't see a ton but those I do see are pricy

Option 3
Keep or replace my current chassis entirely, add the mirrored SLOG devices via both x4 PCIe3.0 slots, use the final 8x slot for a single HBA to run all of my drives.

Proposed PCIe slot usage
x8: LSI 9207-8e (or faster) and move all drives to an external case
x4: Chelsi T520-CR 10Gbps NIC
x4: Intel Optane 900P SSD 280GB
x4: Intel Optane 900P SSD 280GB

New Parts + Cost
2x 280GB 900P - $510 ($255 each)
LSI-9207-8e - $60


If the additional/replacement chassis is roughly equal across all three options it looks like the cheapest option would be option 3 but only barely. Option 1 seems the most flexible to me though. My current thinking is to pick up the AOC-SLG3-2M2 and test it out on two drives I have at home which don't have PLP to see if it works. If it does work just fine I think I like option 1 the best.

I'm curious what folks think of the above options and whether folks have any experience with that specific PCIe -> M2 adapter card or other similar cards made by other manufacturers.

 

[Ericloewe]

Note that the add on card is not listed as compatible with my board but an email from supermicro states that it does work

Cool little bit of information. It's often hard to figure out what bifurcation setups are supported by what motherboards.

I'm curious what folks think of the above options and whether folks have any experience with that specific PCIe -> M2 adapter card or other similar cards made by other manufacturers.

It's a slightly complicated subject.

The simplest of adapters are really just a bunch of (impedance-controlled, length-matched, etc.) wires that convert the mechanical interface and everything should just work, barring any manufacturing defects or crap engineering. These only support a single device.

The other end of the spectrum is populated with PCIe switches. These are also pretty much guaranteed to work, but are somewhere between "crazy expensive" and "go into crippling debt expensive", in good part due to a quasi-monopoly on PCIe switches by LSI/Broadcom/Avago who bought out the only other serious player just as NVMe was taking off. Microchip has since announced their own product line, but they're unlikely to make too much of a dent in pricing, assuming they can get products as mature as LSI's. PCIe switches are a part of the PCIe spec and require nothing weird and allow for semi-arbitrary numbers of devices.

In the middle, there are passive cards that split one larger PCIe slot into several M.2/U.2 sockets. These require that the motherboard support bifurcation on the slot they're placed in, for whatever configuration is needed (a x16 slot might need to support x4/x4/x4/x4, for instance). If Supermicro says your combination works, you should be fine. Some BIOSes have configuration options for this stuff.

 

[PhiloEpisteme]

Cool little bit of information. It's often hard to figure out what bifurcation setups are supported by what motherboards.

Unless someone posts here that they definitely have seen it work or not work with my board I may purchase the card and post the results here. Do you think you testing with a pair of Samsung 970 EVOs (that I happen to have sitting around) will suffice? I'm willing to risk the cost of the card but not the cost of the M2 drives with PLP in the event the card has issues. Also, do you have suggestions for testing beyond that the BIOS and OS recognize the drives and that I can configure them for use as SLOG devices in the UI?

The simplest of adapters are really just a bunch of (impedance-controlled, length-matched, etc.) wires that convert the mechanical interface and everything should just work, barring any manufacturing defects or crap engineering. These only support a single device.

I would assume any card which supports 2x NVME drives wouldn't fall into this category if they make use of a single 8x slot.

In the middle, there are passive cards that split one larger PCIe slot into several M.2/U.2 sockets. These require that the motherboard support bifurcation on the slot they're placed in, for whatever configuration is needed (a x16 slot might need to support x4/x4/x4/x4, for instance). If Supermicro says your combination works, you should be fine. Some BIOSes have configuration options for this stuff.

This is where my doubt comes in; the card above seems to fall into this category. SuperMicro's site lists which boards are tested to work with the card above; note that some of the boards are listed as having only been tested with a single M2 drive and others with 2 M2 drives installed into the card. I imagine this is due to the lack of slot bifurcation on some boards. Okay, so now the complicated part in the form of a list of facts (:shrug:)

• Per the compatibility list on the AOC-SLG-2M2 the X11SSL-F is listed as having been tested to work with a single M2 drive
• Per the compatibility list on the AOC-SLG-2M2 some boards are listed as tested to work with 2 M2 drives, suggesting that the X11SSL-F would only work with 1 drive else they would list 2 drives; this perhaps suggests lack of support for slot bifurcation.
• Per the compatibility list on the AOC-SLG-2M2 The X11SSM-F is NOT listed as having been tested to work
• Per the compatibility list on the X11SSM-F the AOC-SLG-2M2 is listed as compatible
• Per the compatibility list on the X11SSL-F the AOC-SLG-2M2 is listed as compatible
• The manual for the X11SSM-F is shared with the X11SSL-F
• The shared manual for the X11SSM-F and X11SSL-F contain no information about slot bifurcation etc.
• SuperMicro has, per my post above, said that the X11SSM-F board worked with that card using 2 M2 drives. This would be ideal.

Conflicting information I would say.

 

[Ericloewe]

Unless someone posts here that they definitely have seen it work or not work with my board I may purchase the card and post the results here. Do you think you testing with a pair of Samsung 970 EVOs (that I happen to have sitting around) will suffice?

Yeah, if it works with one PCIe device, it will work with any PCIe device (unless something is just out of tolerance so that some devices still operate and some don't).

Also, do you have suggestions for testing beyond that the BIOS and OS recognize the drives and that I can configure them for use as SLOG devices in the UI?

As far as the card goes, no. Just move on to whatever testing makes sense in your scenario (using a RAMdisk as SLOG to get an idea of the absolute best performance you can get with SLOG, etc.). If the cards are recognized and work correctly for a little while under load, everything should be compatible.

I would assume any card which supports 2x NVME drives wouldn't fall into this category if they make use of a single 8x slot.

The one you're considering definitely is in the latter category.

The manual for the X11SSM-F is shared with the X11SSL-F

They're the same board, but one has the C232 PCH and the other has the C236.

SuperMicro has, per my post above, said that the X11SSM-F board worked with that card using 2 M2 drives. This would be ideal.

This is probably trustworthy. They'd give some boilerplate "not tested" answer otherwise. The rest is just down to it being hard to update all documentation for each additional device they test.

 

[PhiloEpisteme]

Thanks for the follow up.

Yeah, if it works with one PCIe device, it will work with any PCIe device (unless something is just out of tolerance so that some devices still operate and some don't).

I was under the impression that this was not necessarily always true. I thought that if the board does not support port bifurcation you may only get support for a single M2 module on the card. Interesting that SuperMicro would list some as having been tested with a single M2 drive and other with 2. In any case, I feel much more confident about purchasing the card, will report back.

Edit: I'm realizing I misunderstood your reply. I thought you were meaning if it supports 1 device it'll definitely support 2.

 

[Johnnie Black]

Unless someone posts here that they definitely have seen it work or not work with my board

Someone already did :)

https://www.ixsystems.com/community/threads/x11ssm-f-aoc-slg3-2m2-optane.63589/

 

[PhiloEpisteme]

Someone already did :)

https://www.ixsystems.com/community/threads/x11ssm-f-aoc-slg3-2m2-optane.63589/

Ha, great! I had searched and searched and somehow missed it. I've got the card on the way already so hopefully itll work as above. :)

 

[mistermanko]

Ha, great! I had searched and searched and somehow missed it. I've got the card on the way already so hopefully itll work as above. :)

So in the end, did it work out with the AOC-SLG3-2M2?

 

[PhiloEpisteme]

Yup, it worked just fine. I did have to adjust bios settings for port bifurcation but that wasn't any issue. I only tested it with 970 evos. Haven't yet put together the savings for the DC 4801x drives.

 

[mistermanko]

Yup, it worked just fine. I did have to adjust bios settings for port bifurcation but that wasn't any issue. I only tested it with 970 evos. Haven't yet put together the savings for the DC 4801x drives.

Do you think it would work with the X10SLL-F? It's not mentioned in the AOC tested part list on SM's site.
EDIT: For what setting should I look in the BIOS?

 

[PhiloEpisteme]

EDIT: For what setting should I look in the BIOS?

Its been a while since I did it with mine but you're looking for the PCIE bus setting. One of them should have a setting to change the lane configuration such that it is x4x4 rather than x8. I recall that the manual for my board did not explicitely mention this functionality. I emailed Supermicro prior to purchasing the AOC to be sure it worked and the advised me that it would work and how to set the bios correctly. Taking a quick look at your board's manual I would guess you can find it under PCIe/PCI/PnP Configuration. A quick chat with supermicro might help clarify as well.

 

[mistermanko]

Its been a while since I did it with mine but you're looking for the PCIE bus setting. One of them should have a setting to change the lane configuration such that it is x4x4 rather than x8. I recall that the manual for my board did not explicitely mention this functionality. I emailed Supermicro prior to purchasing the AOC to be sure it worked and the advised me that it would work and how to set the bios correctly. Taking a quick look at your board's manual I would guess you can find it under PCIe/PCI/PnP Configuration. A quick chat with supermicro might help clarify as well.

Just as a follow up, I emailed SM and they gave me green light for using the add-on card with my board. Although the AOC is in the tested part list, I cannot find any BIOS setting to change the PCIe bifurcation. I ordered one and hope for the best.

 

[Ericloewe]

Do you have the latest BIOS? Bifurcation options are a fairly recent thing in system firmware land.

 

[mistermanko]

Do you have the latest BIOS? Bifurcation options are a fairly recent thing in system firmware land.

According to Supermicro's site I have. Though no sign of any bifurcation option so far.

EDIT: unfortunately, the SM rep misread my email and referred to the X11SLL-F for compatibility with that add-on card. So I have my hope very low now. The order was is placed already, I'll try it anyways, who knows maybe Single-SSD would work with that card, as it should just transform m.2 to PCIe.
If that doesn't work I'm out of ideas. Except for an expensive HBA...

2nd EDIT: Maybe an external SSD connected through USB? But that sound not very reliably when I think about it.

 

[PhiloEpisteme]

@mistermanko my guess is without bifurcation you'll only be able to use 1 of the two slots with your board. I'm not 100% sure which slot you'll be able to use. It may be that in an x8 slot you can use either. Or, it may be that you have to use the slot whose traces match up with the first set of lanes that correspond to those in an x4 slot. If that is the case you might try using the card in an x4 slot anyway to save that x8 slot for something else that might really benefit from it.

2nd EDIT: Maybe an external SSD connected through USB? But that sound not very reliably when I think about it.

What do you want to do with it? And if you're willing to use an AOC to plug in SSDs, depending on your use case, why not pick up an HBA to add more SAS/SATA ports and use that rather than USB?

Except for an expensive HBA...

Here is one that I think is pretty modestly priced.

 

[mistermanko]

What do you want to do with it? And if you're willing to use an AOC to plug in SSDs, depending on your use case, why not pick up an HBA to add more SAS/SATA ports and use that rather than USB?

Because my SATA ports on my board are maxed out, I'm looking for a cost effective solution to attach 1 - 2 boot devices to my system. HBA seems overkill (in wattage and cost) for just two more devices. That's the whole ordeal I'm currently facing.

 

[mistermanko]

Here is one that I think is pretty modestly priced.

Ok, I thought it over. Now placed an order for a used H310 HBA and a breakout cable. Gonna send the AOC addon-card back.