Building my first TrueNAS system...

[jorisvervuurt]

Hi all! I'm new here and need some help.

I'm planning on building my first TrueNAS Scale system based on a Ryzen 5600 CPU (not an APU) with 64GB ECC RAM and an ASRock Rack main board supporting IPMI. The board I'm looking at has two onboard Broadcom BCM57416 10GbE NICs, but I can't find anywhere if the drivers for the BCM57416 are present in TrueNAS Core/Scale?

I know Intel and Chelsio are recommended, but unfortunately the main board model that uses the X550 chipset is nowhere in stock here in The Netherlands. Now you might say: use a PCIe card. I did look into that option, but unfortunately the CPU does not have enough PCIe lanes to support x16 and x8 simultaneously. I'm planning on using an ASUS Hyper M.2 x16 Gen 4 Card with four NVMe drives (requires x4x4x4x4 bifurcation) so besides that I'm only left with an x1 port which is not enough for a 10Gb card.

The drivers for the Broadcom BCM57416 are apparently baked-in starting with Debian 11 (if I read correctly), but I'm not sure if that also means they are in TrueNAS Scale? Does anybody know? Your reply is greatly appreciated!

 

[Fleshmauler]

I think I have multiple similarities in my current build - except the I got the x570D4U (not the 2L2T version with the 2 10gig nics) since the one with dual 10gig was ALWAYS sold out... I solved this problem by using the pcie 4.0 x1 port & it does work just fine, details found here.

/edit; I just realized that the topic quoted dumped both the intel version of the dual 10gig and the broadcom version into a single thread. Uhhh, good chance the post I'm looking at is your exact situation since it a very recent one & the intel ones seem sold out, but can't be reasonably certain

Show : Not 100% guaranteed

I regards to the dual 10gig NICs, per some random guy on lvl1 techs, seems that built in 10gig works just fine with Truenas Scale & that the dude seems a touch confused about IPMI.

/edit

...This is an assumption that you're talking about the Asrock x570 D4U-2L2T/BCM, but given what you've posted I can't imagine any other motherboard that could possibly line-up so perfectly with all the features & that specific CPU.

 

[jorisvervuurt]

Thanks for your reply @Fleshmauler! I am indeed looking at the X570D4U-2L2T/BCM.

The -2T version would be a better choice (Intel X550 with 2x 10GbE) and no Gigabit ports except for the IPMI port. Unfortunately, that one is available nowhere either. Just the BCM version is in stock.

Looking at the forum posts and searching through Discord, it indeed seems that the the chances are big that TrueNAS Scale has support for the BCM57416 through the bnxt driver. If I went with the X570D4U-2L2T/BCM that means I can still use the NVMe AOC with four drives (for some reason the WD Red SATA 2.5” SSD drives are priced the same as the much faster NVMe drive), skipping SATA power and data cables. :)

Ideally I’d want to use an Intel X520 based (or X710, assuming that one is supported as well) SFP+ card but then again I would only be able to use two of the four slots in the NVMe card.

While I appreciate the tip, I personally don’t like modifying the motherboard or card. The card you mentioned uses RJ45 and a chipset that’s not really recommended as well. Then again the same thing goes for Broadcom…

It’s really the one thing that is stopping me from pulling my wallet… Ideally I’d use the NVMe AOC card, but I’m seriously thinking about going for SATA drives (even though it would cost the same and they’d be much slower than the NVMe drives). The motherboard had two M.2 slots, but I’ll be using one for the boot drive and will keep the other one as a spare should I need a SLOG or something in the future.

 

[Fleshmauler]

Yeah, it wasn't recommended at all - I ended up modifying a pcie 4.0 x1 riser instead of the card or the motherboard since it was most risk appropriate; works fine - can ALMOST get the 10gig speeds. Only recommended it since you were already looking at ethernet on an already not recommended //controller//edit.

When you mentioned you're going that hard core on full m.2 storage I thought you had a niche use-case & didn't think of questioning it. If the main consideration is just price at the cost of all potential pcie slots (save the x1), can I ask the use-case?

 

[Etorix]

While I appreciate the tip, I personally don’t like modifying the motherboard or card. The card you mentioned uses RJ45 and a chipset that’s not really recommended as well. Then again the same thing goes for Broadcom…

Don't get the recommendation wrong: RJ45 from a reliable Intel NIC is WAY better than a Broadcom NIC.
And even if there were a driver for this NIC in SCALE this does not mean that the driver is solid enough for the abuse ZFS may throw at it.

Better stay away from this motherboard. Beside, if you're serious about NVMe storage you should look into Xeon Scalable or EPYC to have more lines than what consumer-grade CPU can provide.

 

[jorisvervuurt]

Thank you both for the replies. The system will replace an old Synology NAS (which has been in use for more than 8 years) and another (Intel N6005-based) mini PC that runs some Docker containers (Home Assistant, Zigbee2MQTT, Mosquitto and a few web servers). It's basically going to be a home server that will be working like a NAS, run those same services and eventually will be used for some VMs as well. Due to space constraints, I'm looking at microATX boards and cases; the system is not going into a rack.

Some months ago, I upgraded my entire home network to 2.5Gb / 10Gb (OPNsense router with Intel i226 NICs, 1Gbps symmetrical, and everything after that is Ubiquiti Enterprise stuff, including WiFi 6E access points). I'm basically future-proofing for the next 5-10 years or so, so I'm also building the TrueNAS system like that.

I've been looking at the Supermicro and ASRock Rack catalogues again and I think I've come to the conclusion that it's simply not possible to have the PCIe 4.0 x16 NVMe AOC card together with the network interfaces I want, at least not with an AM4 board. I'd indeed have to go for a (much more expensive) Xeon-based system. If only the ASRock Rack X570D4U-2L2T (the one with the two Intel X550-AT2 NICs instead of Broadcom NICs) was in stock somewhere... that would be the ideal board I think (except that it would be RJ45 instead of SFP+).

The X470D4U2-2T might be interesting too (PCIe 3.0 instead of 4.0, but it does have X550 NICs and seems to be in stock at least in one place here in The Netherlands); although it does have an older chipset. That one would allow me to use the PCIe 3.0 x16 version of the ASUS NVMe card (still much faster than SATA drives of course!). I think I'm gonna call some companies tomorrow to see if I can get my hands on that X470-based board. :)

 

[Etorix]

Xeon Scalable does not have to be terribly expensive if you go for 1st/2nd generation Silver or Gold 5000, possibly refurbished. X11SPM boards are around 500 E, which should be close to AsRockRack server Ryzen boards. Plus, you get more PCIe lanes (3.0) and wallet-friendly RDIMM.

Otherwise, if you can't find a X570DU4 or B550D4U with a X557 NIC the X470D4U2-2T is a safer option for TrueNAS than the /BCM board. A short run of Cat. 7 cable, with a RJ45 transceiver for a SFP+ network, should be fine.

 

[jorisvervuurt]

Thanks again. Looking at those boards, the X11SPM-TPF does seem quite nice. Can't find it anywhere in stock though. The X11SPM-TF I did find in stock (one left, it has RJ45 instead of SFP+). Looking at the CPUs is a problem though, can't really find them in stock anywhere, not even second-hand on the sites I know here in The Netherlands. One 'refurbished' one was over 1000 euros; that's a bit too much lol.

I think I'm gonna go for the X470D4U2-2T, however I did see that it doesn't take the Ryzen 5600 (the one without GPU; this one has ECC support according to AMD's site). It does take the 5600X, which if I'm correct is just a slightly faster 5600 (again without GPU). There's nothing mentioned about ECC support though, so that kind of is a gamble...

 

[unseen]

I'm running the X470D4U board with the standard GB Ethernet ports and can assure you that ECC memory is fully supported. (On TrueNAS CORE at least.)

 

[jorisvervuurt]

I'm running the X470D4U board with the standard GB Ethernet ports and can assure you that ECC memory is fully supported. (On TrueNAS CORE at least.)

Cool! Are you also using the 5600X?

 

[Fleshmauler]

Pretty sure the 5600x is listed as support for Asrock here

//Edit/ I just saw you are using the non-x; which I couldn't imagine why it wouldn't be supported

 

[unseen]

Cool! Are you also using the 5600X?

I don't run lots of jails or virtual machines on my NAS (I have a Proxmox cluster for that), so I went for the cheaper Ryzen 5 3600 instead.
(I've updated my signature to include the full details of my current TrueNAS server.)

 

[jorisvervuurt]

Thanks! I’ve just pulled my wallet and ordered the X470D4U2-2T (the one with the two Intel X550-AT2 NICs)! Delivery will take one to two weeks though, so I still have some time to determine which CPU and other components I’m going to order.

 

[unseen]

The last incarnation of my TrueNAS server was also on an ASRock Rack server motherboard (the E3C224D2I). They make nice reliable motherboards, so I'm sure you'll be happy with the X470D4U2-2T.

 

[mosfet]

Cool! Are you also using the 5600X?

The 5600 just happens to be quite a bit newer (as in released, not in technology) than the 5600X, and a bit newer than 5600G even. Probably not gonna get around to updating the docs. Any AGESA from the past 18-24 months probably supports 5600 just fine. AMD tends to future proof those by a few months, it seems like.

 

[jorisvervuurt]

I'll just chime in again: I have cancelled the order for the X470 board after looking at some more reviews and the delivery time getting delayed. I also noticed that the X470 board hasn't received BIOS updates for more than two years. I have now decided to go all out on a future proofed (and way overkill) AM5-based system. This will be my first TrueNAS Scale system, but I hope I can get a lot of years of use out of this.

- SilverStone ALTA G1M case

- Fractal Design Ion SFX-L 500W Gold PSU (manufactured by Seasonic)

- ASRock Rack B650D4U motherboard

- AMD Ryzen 9 7900 CPU (12 core, 24 threads, 65W TDP)

- Noctua NH-D12L CPU cooler (with extra mounting bars that allow a 90 degree mounting position)

- 2x Kingston KSM48E40BD8KM-32HM (32GB ECC, DDR5 4800 MHz, so 64GB in total and I can upgrade to 128GB if needed...)

- 2x Western Digital WD Red SN700 500GB PCIe 3.0 x4 NVMe boot drives (mirrored, on the motherboard)

- ASUS Hyper M.2 x16 Gen 4 (using x4/x4/x4/x4 bifurcation on the x16 slot; ASRock has confirmed that the B650D4U supports it)

- 4x Western Digital WD Red SN700 2TB PCIe 3.0 x4 NVMe main drives (in a pool of two 2-way mirrors, so 4TB effective storage)

- Intel X550-T2 dual 10GbE network card (PCIe 3.0 x4)

I'll add some more Noctua fans once I have the system up and running, so I have an idea of how much additional cooling it needs (the SilverStone case comes with one big 180mm fan that blows air upwards).

All components will be delivered next friday, so next weekend is going to be fun!

 

[jorisvervuurt]

Yesterday, most components arrived (except for the two 500GB SSDs). Unfortunately, I came to the conclusion that the Noctua NH-D12L isn’t compatible with the B650D4U motherboard when the x4 PCIe slot is in use. I could use it in a sub-optimal position (air blowing to the side of the case instead of upwards), but it would not be efficient.

I have decided to use the stock AMD Wraith Prism cooler I got with the CPU (with RGB turned off, of course) and return the Noctua one.

Today will be building day! The two 500GB SSDs will arrive today, as well as a 900VA APC UPS (BR900MI). :)

 

[unseen]

Fingers crossed that your APC UPS works with the version of NUT that is included with TrueNAS.
In theory, it should work with the "usbhid-ups" driver, but since APC stopped cooperating with the Network UPS Tools project, their devices have become less compatible.

 

[jorisvervuurt]

Fingers crossed that your APC UPS works with the version of NUT that is included with TrueNAS.
In theory, it should work with the "usbhid-ups" driver, but since APC stopped cooperating with the Network UPS Tools project, their devices have become less compatible.

The build is in progress! I might connect the UPS to my OPNsense router and use that as a NUT server. The UPS I have, is a model from 2019 with a serial/USB data port. Let’s hope it still works. :)

 

[jorisvervuurt]

Ran some initial tests with the UPS under Ubuntu; it was detected immediately without any changes to the system, so I’m hopeful it’ll work out of the box. :) The system idles at 40W power consumption, full load (using the `stress` tool) it peaks at 141W. The old Synology NAS idled at 30W, so it’s not even that much more. :)