FreeNAS Supermicro X11SPH-nCTF OCulink (NVMe PCIe3.0x4) question

[klousf]

Hi There,

I am putting together a building list for a FreeNAS server as a replacement for my old Synology NAS units.
I want a state-of-the-art system and I am thinking of the following components:

• Fractal Design Define R6 case
• Seasonic Focus Plus Platinum SSR-650PX
• Supermicro X11SPH-nCTF motherboard
• Intel Xeon Silver 4210 CPU
• Noctua CPU Cooler DX-3647
• 4X 32 GB DDR4 2666MHz ECC RDIMM memory
• 8X HDD 6TB WD GOLD 7200rpm SATA3 (W6003FRYZ) (Datapool with 2 RAIDZ2 vdevs)
• 2X SSD 1TB Intel DC P4510 NVMe PCIe3.0x4 (Datapool with 1 mirror vdev)
• 2X Supermicro SSD 64GB SATADOM for booting FreeNAS (mirror)

The 8 HDD disks are going to be connected to the SATA ports of the motherboard.
The 2 Intel SSD will be connected to the 2 OCulink ports. Basically these are PCIe NVMe ports and with the right connector (CBL-SAST-1011) connected to the Intel SSD's U2 port (SFF8639) but will FreeNAS see them, and is it supported?
Is there someone with experience with this setup?
I found this post with a similar OCulink question but still no answer if it works or not.

Regards,

Frank

 

[Chris Moore]

This all looks to be very good hardware that should serve you well. No criticism. One of the great things about FreeNAS is that it doesn't really need "state of the art" to give good performance because the limitation on performance is the mechanical drive, not the CPU. Processor load on my NAS tends to fluctuate between 1 and 10% and only goes up if I am using the Plex plugin to transcode video. If you are willing to consider used hardware, you can save significantly and still have great performance. The improvement offered by new hardware are very minor while the cost is difference is huge.
If you are interested, you could look over the components listed here:

Specific build changes to upgrade as high as 512GB of RAM

This is intended as a modification to the list of hardware I put together here: https://www.ixsystems.com/community/resources/specific-build-components-list-up-to-32gb-ram.109/ Be aware, this is a totally different system board with a totally...

www.ixsystems.com

The only change would be the system board, CPU and RAM.

 

[klousf]

Hi Chris,

Thank you for your input, I know FreeNAs is not that demanding for hardware but there are other reasons for choosing this specific system. There is a change that I go for installing ESX6.7 to run some VM's and running FreeNAS in one of the VM's. In that scenario I do need the extra power. Also I did consider go for a few generations earlier motherboards and processors but I found out that I can not get the parts from my preferred supplier. I do no want to shop on e-bay for the parts. That is why I want to go for this build, but do you think the OCulink ports (with the installed SSD's) are show up in FreeNAS? That is my concern on this build.

Regards,

Frank

 

[Killer-Bimmer]

Nice you have similar taste:)

ESXi bare-metal FreeNAS Expansion Build

Cheers All! First Post, but long-time lurker; have posted on FB page a few times… Appreciate the community and all the knowledge especially from the long-time members! Sometimes your patience is awe inspiring lol… Hopefully I don’t try it here, but probably will I’ve have been running a system...

www.ixsystems.com

Instead of using the Intel SATA have you thought about using the LSI3008 for your 8 drives? You'll need to flash the firmware to IT. https://www.supermicro.com/wftp/driver/SAS/LSI/3008/Firmware/

See no reason that FreeNAS won't see, but haven't used these connections on the board yet.

 

[klousf]

Nice you have similar taste:)

ESXi bare-metal FreeNAS Expansion Build

Cheers All! First Post, but long-time lurker; have posted on FB page a few times… Appreciate the community and all the knowledge especially from the long-time members! Sometimes your patience is awe inspiring lol… Hopefully I don’t try it here, but probably will I’ve have been running a system...

www.ixsystems.com

Instead of using the Intel SATA have you thought about using the LSI3008 for your 8 drives? You'll need to flash the firmware to IT. https://www.supermicro.com/wftp/driver/SAS/LSI/3008/Firmware/

See no reason that FreeNAS won't see, but haven't used these connections on the board yet.

So that's a killer setup and a very nice RACK too. I will follow your post. I am indeed aware of the LSI3008 on the motherboard and I will use the controller if I am going for the ESX solution with FreeNAS as a VM. The only thing is re-flash the controller because my supplier warned me of the warranty if I will do that. Did you flash your LSI3008 already, and did you ran into problems?

 

[Rand]

The 2 Intel SSD will be connected to the 2 OCulink ports. Basically these are PCIe NVMe ports and with the right connector (CBL-SAST-1011) connected to the Intel SSD's U2 port (SFF8639) but will FreeNAS see them, and is it supported?

Occulink uses basic nvme drivers, it makes no difference to FreeNas whether the disk is atached to an Occulink Port, is an AIC Card, or even attached to a PCIe splitter (bifurcation) or multiplier (pcie switch) card.
O/c some of the latter might introduce additional latency (or NUMA issues) but thats another topic;)

Two drives are AIC, two are on Oculink - can you tell which are which?;) (this is on a DPI-NT, but looks the same on my SPH)

root@freenas[~]# dmesg|grep -i nvme
nvme0: <Generic NVMe Device> mem 0xaae10000-0xaae13fff irq 34 at device 0.0 numa-domain 0 on pci4
nvme1: <Generic NVMe Device> mem 0xb8610000-0xb8613fff irq 40 at device 0.0 numa-domain 0 on pci6
nvme2: <Generic NVMe Device> mem 0xc5e10000-0xc5e13fff irq 48 at device 0.0 numa-domain 0 on pci8
nvme3: <Generic NVMe Device> mem 0xc5d10000-0xc5d13fff irq 49 at device 0.0 numa-domain 0 on pci9
nvd0: <INTEL SSDPED1D480GA> NVMe namespace
nvd1: <INTEL SSDPED1D480GA> NVMe namespace
nvd2: <INTEL SSDPE21K375GA> NVMe namespace
nvd3: <INTEL SSDPE21K375GA> NVMe namespace

 

[klousf]

Occulink uses basic nvme drivers, it makes no difference to FreeNas whether the disk is atached to an Occulink Port, is an AIC Card, or even attached to a PCIe splitter (bifurcation) or multiplier (pcie switch) card.
O/c some of the latter might introduce additional latency (or NUMA issues) but thats another topic;)

Two drives are AIC, two are on Oculink - can you tell which are which?;) (this is on a DPI-NT, but looks the same on my SPH)

root@freenas[~]# dmesg|grep -i nvme
nvme0: <Generic NVMe Device> mem 0xaae10000-0xaae13fff irq 34 at device 0.0 numa-domain 0 on pci4
nvme1: <Generic NVMe Device> mem 0xb8610000-0xb8613fff irq 40 at device 0.0 numa-domain 0 on pci6
nvme2: <Generic NVMe Device> mem 0xc5e10000-0xc5e13fff irq 48 at device 0.0 numa-domain 0 on pci8
nvme3: <Generic NVMe Device> mem 0xc5d10000-0xc5d13fff irq 49 at device 0.0 numa-domain 0 on pci9
nvd0: <INTEL SSDPED1D480GA> NVMe namespace
nvd1: <INTEL SSDPED1D480GA> NVMe namespace
nvd2: <INTEL SSDPE21K375GA> NVMe namespace
nvd3: <INTEL SSDPE21K375GA> NVMe namespace

So you actually have it working? That sounds hopeful for me. :) Thanks!

 

[Rand]

Yes, working out of the box on my X11SPH-nCTPF