First Truenas build, suggestions needed!

[Nassi]

The original post, if you want to read it​

Show

Hi,
I'm planning to build a NAS with budget around 500€ - 1 000€ (MB & CPU only). I'm going to buy used DDR4 ECC ram from ebay, 32GB or 64GB sticks.
Of course, you don't have to use the entire budget, and even used devices could be an option as long as they are reasonably easily available in the EU area.

The purpose of the build is to be a home NAS for 1-2 users. The NAS is mainly used as a media server, but it also houses all personal stuff, backups, etc.
No apps or VM's are intended to be used in this machine, there will be a separate build where docker and VM's are used. This separate device also backup most important files from this new NAS.

I want the pools to be encrypted against physical theft. The NAS contains a lot of personal data and I don't want it to be used by potential thieves. Some time ago we had thieves and they were only one (locked) door away from the current Unraid NAS. I didn't think about encrypting NAS before, but this made me change my mind. This presumably has an effect on the required CPU power?

The motherboard should have at least two pcie ports for HBA and NIC (SFP+ because of lower power consumption). I prefer separate cards so that the update is easier than with the Embedded version.
I will possibly upgrade to a 10Gbe LAN. I already have a 24bay rack case and I will start with 6 wide raidz2. (HDD)

Electricity is relatively cheap here where I live. on average about 0.1€/kwh including all fees and tax. Nevertheless, I want the NAS to be energy efficient. However, the device is in idle mode for most of the day. For this reason, idle consumption should be as low as possible.
I know that peripherals and hard drives consume most of the electricity. However, it probably doesn't make sense to build a NAS with EPYC which idles without drives at 80w if only purpose of build is to be a pure NAS. On the other hand, options that consume very little power, such as Atom-based devices, cost almost the same as much more durable ones. Of course, they consume a little less power, but does this come at the expense of performance? Even if it's just a NAS, I still want the device to not sweat under any circumstances.

Minimum RAM requirement for me is 128GB, so consumer parts are not an option. I prefer Supermicro motherboards but also interested of Asrock Rack.
I have been watching Xeon Silver, Xeon W, Xeon E and EPYC lineup. I have no experience with any of these and the forums don't seem to have direct experience with my intended use. Almost all cases are more of a "home lab that includes a NAS" type of builds.

Options I researched before starting the build:​

Show

INTEL
Supermicro X12STH-LN4F + Intel Xeon E-2336 - support 128GB ECC RAM (~850€ NEW)
Higher clock speed than Xeon Silver, useful in file transfer?

Supermicro X11SPM-F + Intel Xeon Silver 4208 - support 1.5TB 3DS ECC RAM (~870€ NEW)
What 3DS mean???

AMD
Supermicro M11SDV-4C-LN4F - support 512GB ECC RAM (~630€ NEW)
Only 1 pcie slot but 10Gbe NIC can be used via m.2 riser?

Supermicro H12SSL-i + AMD Epyc 7252 - support 2TB ECC RAM (~910€ NEW)
Idle power consumption propably highest of all these options?

ASRock Rack X570D4U-2L2T/BCM + AMD Ryzen 5 5600 - support 128GB ECC RAM (~770€ NEW)
Does support ECC but not ECC logging if I understood correctly, does it matter if I don't even know what it means?

Purpose of use​

- Home NAS for my family
- Mainly a media server (Movies, photos, office stuff etc.)
- Only NAS functions, all applications, VMs, Docker, etc. run on a separate Proxmox server in the same rack

Build goals​

- As reliable as possible
- Slightly overkill, but still "energy-efficient" build
- Snappy experience when browsing and moving files

Current state of build:​

Feel free to suggest hardware which is not acquired yet!

Used stuff is ok, but should be available in the EU. I try not to buy anything from China because of the risk of a cheap copy.
I prefer server grade components.

Acquired
Not acquired

Case	Logic Case SC-4324S - 24-Bay 4U case (6Gb/s backplanes)
CPU	Intel Xeon Silver 4112 (4c/8t)
CPU Cooler	Noctua NH-D9 DX-4677 4U + 3647 mounting kit from Noctua (Free)
MB	Supermicro X11SPL-F
RAM	6 x Samsung 32GB DDR4 2400 MHz ECC REG M393A4K40CB1-CRC (192GB total)
PSU	Seasonic 650W PRIME TX-650
SLOG	Intel OPTANE 118GB SSD P1600X (might add later if something needs sync writes)
L2ARC	Can I use consumer grade NVME drives? Does the mirror bring any advantage? If I use special VDEV, I guess L2ARC is not needed? Otherwise, the idea would be to use L2ARC in "metadata only" mode.
Special VDEV	Can I use consumer grade NVME drives in 3 or 4 way mirror? As far as I understand, special devices are normal VDEV (not a logging device), so the usage is not greater than with any other part of main pool? For this reason, there seems to be no need for an optane write duration?
Pool	Start with 6 x Seagate Exos X 20TB (RaidZ2) OR striped mirror, which I can start with one pair of 20TB disks. What is your recommendation? 6 wide RaidZ2 and striped Mirror do not have a big difference in the usable space. The expandability of the Mirror and starting with a just one pair of disks would be attractive, because it will probably take me quite a while to fill even one pair. RaidZ2, on the other hand, would bring more peace of mind.
Boot	Servethehome recommending the WD Blue NVME drive (~35€ Amazon.de) I prefer a widely and well-known brands
HBA	Would prefer a one 24i HBA, LSI SAS 9305-24i could be a good option, but 12Gb/s is useless in my situation.
10 GbE NIC	Will add later but will take suggestions

 

[danb35]

Minimum RAM requirement for me is 128GB

Um, why? Sure, more RAM is always better with ZFS, but that sounds awfully high as a "minimum requirement" for a system that's only doing file-sharing.

But given that requirement, the Supermicro X11SPH-NCTF along with whatever Xeon Scalable you like should fit the bill. It supports up to 2 TB of RAM, has dual on-board Intel 10G Ethernet (RJ45; if you prefer SFP+, go for the -NCTPF instead), has a Broadcom 3008 SAS controller onboard, has adequate PCIe slots if you need them--pretty much everything you'd want.

Another option, and likely less expensive yet, would be a dual-socket board like mine--those seem to be more widely available used, and thus the prices are a bit lower.

 

[Nassi]

Um, why? Sure, more RAM is always better with ZFS, but that sounds awfully high as a "minimum requirement" for a system that's only doing file-sharing.

I apologize for the wrong choice of words. It's not a minimum requirement, it's that I want at least 128GB of memory because ZFS loves memory.
Used ECC RAM is also so cheap that I see no reason to use less.
Perhaps more RAM would also be helpful with a possible 10 GBE LAN? In addition, as a media server, the files are very large, largest ones between 50GB-100GB. So fitting anything useful in to RAM cache would be hard with 32GB RAM as example.

 

[joeschmuck]

This presumably has an effect on the required CPU power?

Ensure your CPU has AES-NI built-in, this greatly improves Encryption performance. Also, ensure you read all about Encryption. Once you build the system and have completed the stability testing, I would highly recommend you play around with it. Setup some encrypted datasets, power off the system, power on. Leave one hard drive out when your create your pool(s) and after you put some play data on it, pretend a drive has failed and you must replace it with the drive you left out. If you do not do this and I highly advise you take your own notes, you are apt to have problems when a drive does fail a few years down the road.

You never stated how much storage capacity you were aiming for since you are also asking about power consumption. I know you are specifically asking about motherboards and CPUs but when asking for recommendations, knowing more about what the system requirements are can be helpful.

When it comes to power consumption, you end up paying one way or another. If you need low power, go NVMe. They make a 30TB PCIe x4 drive, if you buy a Xeon Scalable CPU, you should have lots of PCIe lanes to support 8 drives, in a RAIDZ2 would give you about 160TB. That saves all the power that the hard drives would have consumed. Just an option for the future. And if you do want a lot of PCIe lanes, you need to plan that from the start. You can't add PCIe lanes if your motherboard/CPU only supply 20 lanes to the PCIe slots. My new system is Rust Free, ASRackRock B650D4U. There are some limitations that I was aware of before I purchased it (read the User Guide) and then a few things I overlooked but got through. I'm actually happy with my setup, but it was not cheap. But I also do not see it failing for the next 10+ years, unless lightning strikes it.

I have nothing more to add at this time except for you to read the forum threads, a lot of people asking what to purchase for a new system. Some folks want high-end systems, some want power-saving systems, and of course you have a few people just wanting a reliable home NAS that is quiet.

 

[danb35]

it's that I want at least 128GB of memory because ZFS loves memory.

Fair enough, and you're right that RDIMMs are pretty cheap these days. More RAM never hurts.

As I mentioned, dual-socket boards are more widely available used than single-socket boards. Here's the same board I have with two Xeon Silver 4114s and heatsinks for US$375, though I don't know whether they ship internationally:

Supermicro X11DPH-T LGA 3647 Motherboard with 2x 2U HS Tested 2x Gold 5118 CPU | eBay

Socket: LGA 3647 Socket P. Heatsink: 2x 2U Heatsink. Supermicro Servers. UnixsurplusCom - Supermicro Systems. PlusBoards is a division of UNIXSurplus, a large Server dealer in Silicon Valley, California.

www.ebay.com

Lots of RAM capacity (I believe it also maxes out at 2 TB), dual Intel 10 GbE onboard (but no HBA), dual m.2 NVMe sockets. I'm still using my Chelsio NIC because the onboard 10 GbE is RJ45 rather than SFP+. It's worked great for me with TrueNAS SCALE for the last few months.

 

[Nassi]

I lean towards the Xeon Scalable system. What processor would you recommend for my intended use? Bronze or Silver?

In summary:
- Only NAS functions
- 1-2 users
- possibly 10gbit LAN
- Encrypted Pool(s)
- Primarily SMB share, but possibly NFS/CIFS Between NAS and Proxmox Server/Linux Clients
- Idle power consumption "low as possible"
- Start with 6 wide raidz2 pool (probably Seagate Exos X - X20 20TB)

Current Prices:
Xeon Bronze 3204, 6C/6T, 1.9GHz, No Boost, Boxed (135 € New)
Xeon Silver 4208, 8C/16T, 2.10-3.20GHz, Tray (417 € New)
Xeon Silver 4210, 10c/20t, 2.20-3.20GHz, Tray (405 € New)
Xeon Silver 4214, 12c/24t, 2.20-3.20GHz, Boxed (475 € New)

Bronze 3204 would be much cheaper, but I'm worried about low clock speeds. What do you think?
All processors are 85W TDP, can I assume that idle consumption is very similar to all options?
What kind of idle consumption can I expect with the Xeon Scalable system? (without HDD's, HBA etc.)

 

[danb35]

- Primarily SMB share, but possibly CIFS Between NAS and Proxmox Server/Linux Clients

SMB and CIFS are effectively the same thing--did you mean NFS instead of CIFS?

As to CPU(s)--and for that matter the motherboard--I'd strongly recommend buying used. A few months ago, the Gold 6132s I'm using were going for about US$30 each on eBay. Xeon Silver 4114s are going for about US$10-15 each. Paying hundreds of dollars each just seems silly. If these items/prices aren't readily available in .fi, it may be worth looking for a package forwarding service.

What kind of idle consumption can I expect with the Xeon Scalable system? (without HDD's, HBA etc.)

Well, let's work some math. My system was averaging 450W until I removed four spinners yesterday, when it dropped to 425W, so that's about 6W/drive. I have 26 spinners remaining in the system, so that's ~156W, which would bring idle down to ~270W. That's with two CPUs, a LSI 3008 HBA, a two-port 10G NIC (in addition to the two ports on the motherboard), 128 GB of RAM (in 4x 32GB DIMMs), three SATA SSDs, and then system fans and the like. Take out the HBA and the NIC and it might go down to 250W, maybe a bit less because the fans wouldn't be doing as much.

Going to a single processor would definitely save some power, though it's hard to say how much--I haven't been able to find power consumption figures for just the CPU.

 

[Nassi]

SMB and CIFS are effectively the same thing--did you mean NFS instead of CIFS?

As to CPU(s)--and for that matter the motherboard--I'd strongly recommend buying used. A few months ago, the Gold 6132s I'm using were going for about US$30 each on eBay. Xeon Silver 4114s are going for about US$10-15 each. Paying hundreds of dollars each just seems silly. If these items/prices aren't readily available in .fi, it may be worth looking for a package forwarding service.

A used processor could also be a viable option. Is there a "real" difference between *1** and *2** Xeons at home use? (mainly in power consumption)
Ebay seems to be full of cheap *1** generation Xeons. As example, Xeon Silver 4116 ~20-25€ + shipping from EU.

Well, let's work some math. My system was averaging 450W until I removed four spinners yesterday, when it dropped to 425W, so that's about 6W/drive. I have 26 spinners remaining in the system, so that's ~156W, which would bring idle down to ~270W. That's with two CPUs, a LSI 3008 HBA, a two-port 10G NIC (in addition to the two ports on the motherboard), 128 GB of RAM (in 4x 32GB DIMMs), three SATA SSDs, and then system fans and the like. Take out the HBA and the NIC and it might go down to 250W, maybe a bit less because the fans wouldn't be doing as much.

Going to a single processor would definitely save some power, though it's hard to say how much--I haven't been able to find power consumption figures for just the CPU.

This is under a load, right? I assumed that consumption will be around 50W at idle. No VM's, jails, docker etc running, just truenas ready to transfer some files between client and NAS.

 

[Etorix]

Here I list interesting options:

You're on gnerally good tracks, but mind that your different options require different types of RAM: UDIMM (Xeon E, Ryzen), or RDIMM (Xeon Scalable, EPYC, and the embedded boards you're not considering although they would be well suited to a simple NAS).

What 3DS mean???

3DS is a subkind of LRDIMM.

ASRock Rack X570D4U-2L2T/BCM + AMD Ryzen 5 5600 - support 128GB ECC RAM (~770€ NEW)
Does support ECC but not ECC logging if I understood correctly, does it matter if I don't even know what it means?

It depends whether you trust AsRockRack to have done things right or whether you're a perfect control freak and want to trace what's happening with RAM errors…

 

[Etorix]

Is there a "real" difference between *1** and *2** Xeons at home use?

No.

This is under a load, right? I assumed that consumption will be around 50W at idle.

How many drives?
50 W may be possible with few spinners, but then you don't need a HBA. And if you have too many drives to rely on SATA ports from the chipset you're NOT going to reach that 50 W target.

 

[Nassi]

How many drives?
50 W may be possible with few spinners, but then you don't need a HBA. And if you have too many drives to rely on SATA ports from the chipset you're NOT going to reach that 50 W target.

The question concerned the base system (CPU, RAM and MB)

Low core embedded base systems like Xeon-D idle around ~20w
Low core EPYC base systems idle around ~80w
Xeon Scalable probably something in between?

 

[Etorix]

Xeon D-1500 (Broadwell-D) idles around 20 W. Xeon D-2100 (Skylake-D) idles at or over 50W, and I would expect the corresponding 1st/2nd generation Xeon Scalable to be similar, or slightly higher.

 

[Nassi]

I've been thinking about different choices again.

The 41** gen Xeon silver processors are so cheap that I think I'll go with this route. What do you think about the amount of Cores?
Basically, the most powerful processor in the series, the 4116 (12c/24t) costs the same as the 4108 (8c/16t).
The availability of 4112 (4c/8t) seems to be poor and the price is higher than the previous ones.

Does a Truenas benefit from number of processor cores in my use case?
Could you assume that with a larger number of cores, the power consumption would be higher in all situations?

At first I was thinking of getting the Supermicro X11SPM-F, but now I'm leaning more toward the Supermicro X11SPL-F. There are significantly more of pcie slots in the X11SPL-F model, so there is more room for expansion.

I thought the pcie x16 slots were important. (X11SPL-F does not have any and that's why I skipped this option first)
However, pcie x8 slots seem to be more practical in NAS use?
Is there some situation where a x16 slot would be necessary when using a NAS?

 

[Etorix]

For the above use case

In summary:
- Only NAS functions
- 1-2 users
- possibly 10gbit LAN
- Encrypted Pool(s)
- Primarily SMB share, but possibly NFS/CIFS Between NAS and Proxmox Server/Linux Clients
- Idle power consumption "low as possible"
- Start with 6 wide raidz2 pool (probably Seagate Exos X - X20 20TB)

Xeon Scalable looks like overkill (and not optimal for the last point). But if you go this road, keept an eye on Gold as well.
SMB is single threaded per user, so your use case should not use more than two threads but would benefit from high base clocks. Say, something like a Gold 5122 (4C/8T, 3.6 GHz, currently 96 E refurbished). Or whatever you can find for a good price, really…

Could you assume that with a larger number of cores, the power consumption would be higher in all situations?

I don't think that the number of cores matter for for idle power; this is more about the platform in general. Power consumption under maximal load is another matter, but you're only ever going to hit that during a scrub.

However, pcie x8 slots seem to be more practical in NAS use?
Is there some situation where a x16 slot would be necessary when using a NAS?

It depends what you're using the slots for! A HBA is typically x8. 10/25 GbE NICs would come as x8, at least in PCIe 3.0 version, but 100 GbE would take a full x16. NVMe drives are typically x4, so a bifurcated x16 slot with one adapter is just the same as two x8 slots with two adapters—and if you're not bifurcating the slot but using a PLX switch for even more drives, a x16 slot is better.

 

[Whattteva]

Current Prices:
Xeon Bronze 3204, 6C/6T, 1.9GHz, No Boost, Boxed (135 € New)
Xeon Silver 4208, 8C/16T, 2.10-3.20GHz, Tray (417 € New)
Xeon Silver 4210, 10c/20t, 2.20-3.20GHz, Tray (405 € New)

All processors are 85W TDP, can I assume that idle consumption is very similar to all options?
What kind of idle consumption can I expect with the Xeon Scalable system? (without HDD's, HBA etc.)

NAS1 (in my signature) idles around 160W with Xeon Silver 4210, 4 sticks of DDR4 RAM, 2 SSD's, 4 HDD's, and a quad-port 1G NIC.

My previous system with only a lowly 2C/4T with same specs minus 1 SSD and 2 RAM (DDR3) sticks idles much lower at only 60-70W despite also being much older (4th-gen i3). Both were Supermicro systems with built-in BMC (X10 vs X11). The difference of 1 SSD and 2 RAM sticks can't possibly account for all that power consumption difference. More cores/threads generally = higher idle consumption.

 

[Nassi]

NAS1 (in my signature) idles around 160W with Xeon Silver 4210, 4 sticks of DDR4 RAM, 2 SSD's, 4 HDD's, and a quad-port 1G NIC.

wow, that is a lot!

With rough math:
quad-port 1G NIC - 20w
4 HDD's - 4 x 5w = 20w
2 SSD's - 2 x 5w = 10w
RAM - 4 x 5w = 20w
MB - 20w

This leaves the CPU's share of power consumption at 70w (idle), and I believe that those rough watt numbers are on the upper end?
Something in this equation doesn't add up when compared to the measurements made by servethehome.

https://www.servethehome.com/intel-xeon-silver-4116-linux-benchmarks-and-review/

The Intel Xeon Silver series is highly optimized for low power consumption over performance so it is a big part of the story. The Silver 4116 was using up to 134W in our test with 10Gbase-T links active, a SAS 3008 12gbps controller and a number of drives / DIMMs in the system.

https://www.servethehome.com/intel-xeon-silver-4108-linux-benchmarks-and-review/

We normally only publish power consumption data on full systems we review. There are too many variables for this data to land in a CPU test. At the same time, we wanted to highlight the Intel Xeon Silver 4108 which hit a maximum of around 93W measured at the outlet in our testing. While it may be an 85W TDP part, with 10Gbase-T links active, a SAS 3008 12gbps controller and a number of drives / DIMMs in the system. The Intel C622 PCH has a 17W TDP. If you subtract out some power consumption for the rest of the system, the Xeon Silver 4108 is absolutely sipping power. This chip borders on Intel Xeon D territory and below Intel Xeon E3-1200 V6 even with more memory installed.

 

[danb35]

Here's the fundamental problem: requiring at least 128 GB of RAM means you need (in modern Intel CPUs) a Xeon Scalable. Requiring power consumption "as low as possible" really needs a lower-end CPU. One of those requirements is going to have to budge.

 

[Whattteva]

wow, that is a lot!

With rough math:
quad-port 1G NIC - 20w
4 HDD's - 4 x 5w = 20w
2 SSD's - 2 x 5w = 10w
RAM - 4 x 5w = 20w
MB - 20w

This leaves the CPU's share of power consumption at 70w (idle), and I believe that those rough watt numbers are on the upper end?
Something in this equation doesn't add up when compared to the measurements made by servethehome.

https://www.servethehome.com/intel-xeon-silver-4116-linux-benchmarks-and-review/


https://www.servethehome.com/intel-xeon-silver-4108-linux-benchmarks-and-review/

Well, to be fair, I forget to mention they also run vastly different motherboards. It is probably not just the CPU. The 4210 is also running on a much larger motherboard with 8 instead of 4 DIMM's (though only 4 populated), runs registered RAM instead of unbuffered ECC, has 2x10G Ethernet instead of 2x1G ethernet, and has more PCIe slots and lanes. I'm sure the motherboard adds quite a bit of power itself. That being said, the CPU is definitely still contributes a significant portion still.

As @danb35 said, requiring 128 GB of RAM usually means you will have increased power consumption because the motherboard will generally tend towards the higher end with more capabilities.

 

[Etorix]

Here's the fundamental problem: requiring at least 128 GB of RAM means you need (in modern Intel CPUs) a Xeon Scalable. Requiring power consumption "as low as possible" really needs a lower-end CPU. One of those requirements is going to have to budge.

Xeon E tops at 128 GB, so gives up "at least".
Embedded Xeon-D can take RDIMM for more than 128 GB RAM.
Xeon D-1500 (Broadwell-D, X10SDV) is certainly "lower-end" by now, but does well enough for storage.
Xeon D-2100 (Skylake-D, X11SDV) idles above 50 W, and may not be much of an improvement over 1st/2nd gen. Xeon Scalable.
Xeon D-1700 (Ice Lake-D, X12SDV) just fit within the budget. Its idle power is intermediate between the previous two generations, and it is still "modern" in my book (PCIe 4.0 and 25 GbE on-board). But one has to buy new.

 

[danb35]

Sounds like my information was out of date, probably more in line with the Xeon E3/E5 days--though X10 series is probably older than I'd be recommending for a new build at this point. Though I still think it's a valid point that if you want to leave the door open for > 128 GB of RAM with a reasonably-modern system, low power consumption will have to budge.