Hardware Recommendations Guide Discussion Thread Rev 2a) 2021-01-24

[jgreco]

Hi, I am considering to build a NAS to run TrueNAS. I went to Resources to get the Hardware Recommendations Guide but it was made two years ago. Is there a more up-to-date guide?

No. Your NAS will not really be any faster on "more up-to-date" hardware, and drivers for hardware (like the now infamous Intel 2.5GbE chipset) generally don't make it into FreeBSD and Linux in a stable form for several years. You're not building a gaming PC. The latest and greatest isn't as important as stuff being stable and well tested.

 

[Davvo]

There are a few new motherboards that are worth looking, the new supermicro's SoC ones. But they are expensive and you won't find them used.

 

[Etorix]

There are a few new motherboards that are worth looking, the new supermicro's SoC ones.

Which ones? The X12SDV series (Xeon D-1700/2700)?
If so, notwithstanding the price, they have the same drawback as the X11SDV series (Xeon-D 2100): They idle around 70 W , which is what a X10SDV (Xeon D-1500) would draw at full load. For a server which consolidates many active services/VMs/containers, it may be worth it. For pure storage, which is what a NAS is supposed to be about, I'd stick to A2SDi (Atom C3000) and X10SDV as far as SoC-based plateforms are concerned—even though the latter dates from the Broadwell era.

 

[Ericloewe]

There are a few new motherboards that are worth looking, the new supermicro's SoC ones. But they are expensive and you won't find them used.

And often not even new.

There have been a few developments, but none of them make me want to get up and write:

• Newer Intel LGA1xxx paltforms are nice, but ridiculously expensive and availability is dubious
• AMD has no equivalent to the above, only semi-OEM-supported things
• The Xeon E5 market has not had any meaningful entries since... The Xeon E5 v4s. Xeon Scalable pushes cost and capability up, Epyc even further.
• The embedded stuff is cool, but:
  • C3xxx is still the latest available Atom-based line, to mere mortals
  • Broadwell-D was superseded, but only half-heartedly (see post #303 above).

Which ones? The X12SDV series (Xeon D-1700/2700)?
If so, notwithstanding the price, they have the same drawback as the X11SDV series (Xeon-D 2100): They idle around 70 W , which is what a X10SDV (Xeon D-1500) would draw at full load.

Any word on the Xeon D-1700s, specifically? The platform presentation looked mighty interesting, but I haven't come across real reviews yet,

 

[Ericloewe]

As a sidenote, I bought a bunch of used X10-era stuff late last year. The only thing I miss is an NVMe driver included in the system firmware of the X10SL-series boards. The X10SDV and A1SRi/A1SDi lines both have it and make for a nice, convenient boot option.
Ok, I also feel like the Xeon E3 v3/v4 platform is horrendously dated, relying substantially on PCIe 2.0 from the PCH, but this is not a huge deal.

 

[winnielinnie]

Your NAS will not really be any faster on "more up-to-date" hardware, and drivers for hardware (like the now infamous Intel 2.5GbE chipset) generally don't make it into FreeBSD and Linux in a stable form for several years.

Infamous? Wait, time-out. Pause. Hold on.

Did I doom myself? I thought starting with TrueNAS Core 12.0-U8, Intel I225 (2.5GbE) NICs were supported?

Here is the card as reported on my TrueNAS Core 13.0-U3.1 system:

Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)

(click to expand more details)

Show

05:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)
Subsystem: Intel Corporation Device 0000
Flags: bus master, fast devsel, latency 0, IRQ 19
Memory at 3fb00000 (32-bit, non-prefetchable)
Memory at 3fc00000 (32-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
Capabilities: [a0] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Capabilities: [140] Device Serial Number XX-XX-XX-XX-XX-XX-XX-XX
Capabilities: [1c0] Latency Tolerance Reporting
Capabilities: [1f0] Precision Time Measurement
Capabilities: [1e0] L1 PM Substates

It's only use is for rsync/SMB/NFS on a single client. A direct connection between TrueNAS and client with a single CAT6a cable.

It reports as 2500Base-T, and iperf3 tests yield a consistent 2.38 Gbps. Large sequential file transfers over SMB also corroborate this.

 

[Ericloewe]

Hey, if they're working, consider yourself lucky. Between hardware defects and an immature driver, even Intel's had it rough in the 2.5GbE space.

 

[jgreco]

Did I doom myself? I thought starting with TrueNAS Core 12.0-U8, Intel I225 (2.5GbE) NICs were supported?

The card was out about three years ago and for about two years people kept showing up APO (all p***ed off) that it wasn't "supported" yet. My recollection is that it went through several rounds of issues in upstream FreeBSD. I think it is fair to say that it typically takes several years for your typical device, especially insipid ones like the Realtek 2.5 or Aquantia 10G, to filter their way into the upstream OS and become stable enough to be "usable". This is true for both FreeBSD and Linux, based on a quarter century of observation.

 

[winnielinnie]

I'll be a "canary" of sorts if anything bad happens later on, but so far so good.

Nothing special was needed, speeds are good (and consistent), and no hiccups (yet).

What might be working in my favor is that it's a direct CAT6a cable, server to client, port to port. Nothing goes through any switch or home "router".

 

[jgreco]

I'll be a "canary" of sorts if anything bad happens later on, but so far so good.

Nothing special was needed, speeds are good (and consistent), and no hiccups (yet).

What might be working in my favor is that it's a direct CAT6a cable, server to client, port to port. Nothing goes through any switch or home "router".

Traversing a residential grade "router" would be bad. However, there should be a crapton of good quality 2.5/5G ethernet switches out there. We perfected 1GbaseT two DECADES ago and 10GbaseT a DECADE ago; we know the RF and signal processing necessary to make these things work swimmingly well. The motivation for 2.5/5G is basically because of modern greater-than-Gbps wifi and PoE, and because some companies see this as an opportunity to sell new gear. One would hope that if you would get a nice 2.5G switch that it would properly switch at 2.5Gbit/sec.

 

[Ericloewe]

You forget, of course, that the small switch market is dominated by Realtek. It's worked out okay for 1GbE, let's see how 2.5GbE and up go...

 

[Etorix]

Any word on the Xeon D-1700s, specifically? The platform presentation looked mighty interesting, but I haven't come across real reviews yet,

Neither did I. I re-read the STH presentation article before answering to post #302… and noticed the power consumption graph, with the same unpleasant idle figure as for the X11SDV which were discussed in a build thread not too long ago.
Broadwell-D has been superseded in Intel's portfolio… but for different use cases than a nice little home NAS.

In the current market, the best hardware recommendation is to look for suitable refurbished hardware of the X10 generation (E5 or D-1500) because anything newer is hard to find (C246 motherboards for Coffee Lake Core i3/Xeon E) and/or overpriced for what it is (said C246…) or what it is for (anything newer than C246!), if not downright unobtainium (DDR5 ECC someone???) and/or not-yet fully supported (hybrid architecture, 2.5GbE).
Very unfortunate.

 

[Ericloewe]

X11SS-series boards are also good, but the price delta is substantial relative to X10SL-series boards. Probably twice the price, when CPU and RAM are taken into account, but you get:

1. Mildly updated IPMI
2. Factory support for the UEFI NVMe driver (something Supermicro really dropped the ball on for X10SL boards, for no good reason - even the A1SRi/A1SDi boards got the driver, despite having a tiny EEPROM for the system firmware)
3. DDR4 and up to 64 GB of DRAM
4. Much better platform, with more PCIe lanes and PCIe 3.0 on the PCH

if not downright unobtainium (DDR5 ECC someone???)

My typical places to look up prices either have very limited options available (e.g. 64 GB DIMMs only) or none at all. Not to mention the pricing, it's way worse than when DDR4 was new.

 

[Etorix]

"Mildly updated" is an euphemism for the switch from Java-based ASPEED 2400 to HTML-based 2500. I find the former essentially unworkable, so that's a big point for the X11 generation.
But, from a quick search, C236 has the same issue as C246: The few shops which have a motherboard in stock want over 300E for it, which I find excessive for what it is.

Between refurbished old X10 hardware, overpriced "new old stock" X11 and expensive new X12/X13, I currently see no obvious general purpose recommendation right now—so no reason to update the guide.

 

[Ericloewe]

"Mildly updated" is an euphemism for the switch from Java-based ASPEED 2400 to HTML-based 2500. I find the former essentially unworkable, so that's a big point for the X11 generation.

All X10 boards (X10SL, X10SR, X10SDV, X10DR, A1*) support the same HTML5 iKVM as the X11 boards. There are few, if any, meaningful user-facing differences between the AST2400 and AST2500. Supermicro's variant of AMI's RACADM BMC software is also pretty damn similar.
Support was added around the same time the X11 boards first showed up.

 

[Whattteva]

I honestly don't understand all the hate over the Java iKVM. I actually prefer it over the HTML5 just cause the client is more convenient to launch. Login is just a simple button click without any typing once configured and it never logs me out unless I logout explicitly.

 

[Whattteva]

All X10 boards (X10SL, X10SR, X10SDV, X10DR, A1*) support the same HTML5 iKVM as the X11 boards. There are few, if any, meaningful user-facing differences between the AST2400 and AST2500. Supermicro's variant of AMI's RACADM BMC software is also pretty damn similar.
Support was added around the same time the X11 boards first showed up.

My X10SLL-F doesn't have HTML5 support, is that because I need a firmware update?

 

[Ericloewe]

I honestly don't understand all the hate over the Java iKVM. I actually prefer it over the HTML5 just cause the client is more convenient to launch. Login is just a simple button click without any typing once configured and it never logs me out unless I logout explicitly.

I'm having a hard time picturing this problem, it doesn't match my experience at all. The only drawback of the HTML5 client is no virtual media support.

My X10SLL-F doesn't have HTML5 support, is that because I need a firmware update?

Yes, definitely update your BMC firmware. Clear the browser cache when you're done.

 

[danb35]

I honestly don't understand all the hate over the Java iKVM.

IPMI frustration

When I first learned of IPMI, it sounded like a great idea--a built-in IP KVM for your server, so you could access its console from, theoretically, anywhere. That's been some years ago, and since then, the Java thing has gotten worse and worse. At this point, I can't get the remote console to...

www.truenas.com

 

[HoneyBadger]

Infamous? Wait, time-out. Pause. Hold on.

Did I doom myself? I thought starting with TrueNAS Core 12.0-U8, Intel I225 (2.5GbE) NICs were supported?

From what I recall, the issues with the i225-V were largely on earlier silicon revisions, where some combination of firmware/driver caused them to fail to establish a connection on the system resuming from sleep until they were disabled/reenabled. 1Gbps networking seemed to exacerbate the issue.

Huge pain for a gaming desktop (ask me how I know) - but should be largely inconsequential for a server build.