Build advice

[gdarends]

Hello,

I have a QNAP NAS that's been running for a little over 10 years now and have been meaning to upgrade to Truenas/ZFS for a while.

I've been reading some guides and have come up with the following hardware specs:

Motherboard	Supermicro X11SCH-F Motherboard Micro-ATX Single Socket H4 (LGA 1151) for Intel 8th Generation Intel Core
CPU	Intel Core i3-9300 SRCZU 3.70GHZ 4C 4T 8NB 62W LGA1151 CPU Processor
CPU Cooler	Cooler Master Hyper 212 Black Edition CPU Air Coolor, Silencio FP120 Fan, 4 Copper Direct Contact Heat Pipes for AMD Ryzen/Intel LGA1200/1151
RAM	Supermicro (Micron) 32GB 288-Pin DDR4 2666 (PC4-21300) ECC UDIMM
Boot Drive	Kioxia 256GB SSD XG6 M.2 2280 PCIe Gen3 x4 SED Encryption NVMe KXG6AZNV256G Solid State Drive
PSU	EVGA SuperNOVA 550 G3, 80 Plus Gold 550W, Fully Modular, Eco Mode with New HDB Fan
Chassis/Case	Fractal Design Node 804 - Black - Cube Compact Computer Case - mATX
Storage	3x Seagate IronWolf 4TB NAS Internal Hard Drive HDD – CMR 3.5 Inch SATA 6Gb/s 5900 RPM 256MB Cache (ST4000VN006)

I'm planning to do a RAIDZ-2 with 6x4TB HDD (I'm only buying 3 new ones cause I already have 2 Hitachi and a WD)
I will be using the machine for backup. But I also have a list of services that I run such as:
- PiHole
- Unifi Controller
- NUT Server
- Plex & Friends
- SynchThing

So, does the specs hold up for what I'm trying to do with the system? Shoud I bump up the RAM to 64GB?
Should I add SSD for virtualization and opt for TrueNAS Scale?

 

[Davvo]

Supermicro X11SCH-F Motherboard

Make sure to read this thread. Great mobo.

Supermicro X10 and X11 motherboard FAQ

Despite being the default choice for most of us, Supermicro X10 and X11 motherboards have certain quirks. These range from poorly documented solutions to relatively common issues to deliberate choices by Supermicro that may raise questions. This...

www.truenas.com

Cooler Master Hyper 212

Used supermicro board might come with a nasty backplate installed (glued) that prevents the installation of most CPU coolers.
Take a look at this link, you can find compatibile ones.

[Official] CPU Heatsink Recommendations

Air Cooling Notes Generally speaking, Arctic makes a product that works for every solution. They are affordable, high performance, quiet, and well built. LGA115X = LGA1156, LGA1155, LGA1150, LGA1151, LGA1200 LGA115X (hole/backplate) Consumer motherboards usually have pass through holes. Xeons...

forums.serverbuilds.net

WD

Make sure it's CMR and not SMR. Generally, it's considered better to not mix drives.

RAIDZ-2 with 6x4TB

What concerns me Is that half of your pool Is formed by used drives with (statistically, hitachis are beasts) higher chance of dying. Without buying more disks you could do Either a RAIDZ3 or a 3x 2-way mirrors each composed of one new and one used. If you throw in an additional disks, you can do nice things with an hotspare (like 2x RAIDZ1).

Should I add SSD for virtualization

Better 2 in mirror, as always.

 

[Etorix]

I've been reading some guides and have come up with the following hardware specs:

All fine choices. @Davvo is wrong on one point: Up to and including 9th generation, Core i3 do support ECC. (Unless you have sourced the i3-9300 for a fair price, I'd go for a cheaper 9100 though.)

I'm planning to do a RAIDZ-2 with 6x4TB HDD (I'm only buying 3 new ones cause I already have 2 Hitachi and a WD)

Some actually prefer mixing different drives to mitigate against the risk that drives from the same batch fail in short succession.

So, does the specs hold up for what I'm trying to do with the system? Shoud I bump up the RAM to 64GB?

If you can bump RAM, it never hurts…

Should I add SSD for virtualization and opt for TrueNAS Scale?

The choice between CORE and SCALE depends how you want to manage your services.
SCALE may look like the natural choice, and is certainly easier if you can find TrueCharts for everything you need. If not, then running plain Docker containers is more cumbersome than it could be. There's also an issue with Kubernetes creating an awful lot of automatic snapshots one cannot get rid of.
If you're comfortable setting up Docker images with docker-compose and/or managing images from Portainer, it may be better to make a Ubuntu VM for that… and this can as well run from CORE.

 

[Davvo]

@Davvo is wrong on one point: Up to and including 9th generation, Core i3 do support ECC.

Yup, my bad. Just edited.

You might also want to know that right now the previous version of SCALE was on a completely different scale (pun totally intended) compared to CORE in terms of (specific) performance; I don't know if the new release (22.02.2) changed things, but those performances were expected to change with future releases of SCALE.

Performance testing: CORE 12 vs 13 vs SCALE

Lawrence Systems just posted a video with the results of some performance testing with CORE 12 vs CORE 13 vs SCALE. tl;dr appears to be that SCALE performs abysmally in random writes: https://www.youtube.com/watch?v=BoiHHnBDg0E

www.truenas.com

Edited for corrections.

 

[gdarends]

Used supermicro board might come with a nasty backplate installed (glued) that prevents the installation of most CPU coolers.
Take a look at this link, you can find compatibile ones.

[Official] CPU Heatsink Recommendations

Air Cooling Notes Generally speaking, Arctic makes a product that works for every solution. They are affordable, high performance, quiet, and well built. LGA115X = LGA1156, LGA1155, LGA1150, LGA1151, LGA1200 LGA115X (hole/backplate) Consumer motherboards usually have pass through holes. Xeons...

forums.serverbuilds.net

Make sure it's CMR and not SMR. Generally, it's considered better to not mix drives.

What concerns me Is that half of your pool Is formed by used drives with (statistically, hitachis are beasts) higher chance of dying. Without buying more disks you could do Either a RAIDZ3 or a 3x 2-way mirrors each composed of one new and one used. If you throw in an additional disks, you can do nice things with an hotspare (like 2x RAIDZ1).

Better 2 in mirror, as always.

The MOBO is brand new, unopened, according to seller.

The WD is a WD40EFRX which I looked up is a CMR drive.
I do plan to upgrade the old drives when cash flow permits. I too am concerned about them.
I can do all Hitachi's (HGST Ultrastar HUS724040ALE640 4TB 64MB 7200RPM SATA 6Gb/s)
They are about the same price as the Seagate IronWolfs.

All fine choices. @Davvo is wrong on one point: Up to and including 9th generation, Core i3 do support ECC. (Unless you have sourced the i3-9300 for a fair price, I'd go for a cheaper 9100 though.)


Some actually prefer mixing different drives to mitigate against the risk that drives from the same batch fail in short succession.


If you can bump RAM, it never hurts…


The choice between CORE and SCALE depends how you want to manage your services.
SCALE may look like the natural choice, and is certainly easier if you can find TrueCharts for everything you need. If not, then running plain Docker containers is more cumbersome than it could be. There's also an issue with Kubernetes creating an awful lot of automatic snapshots one cannot get rid of.
If you're comfortable setting up Docker images with docker-compose and/or managing images from Portainer, it may be better to make a Ubuntu VM for that… and this can as well run from CORE.

The CPU is listed at $124.77. What's the difference between the 9300 and the 9100?
I did a quick search and the 9100 is more expensive than the 9300. Maybe because it comes with the cooler.

The mobo allows up to 128GB or RAM, is there a benefit to upgrading to that amount of RAM? I was planning on upgrading in the future, but I am not sure it is necesarry.

 

[Davvo]

The MOBO is brand new, unopened, according to seller.

Reading the specifications it says that it's compatibile with supermicro's passive cooler that seems to need the backplate to be installed. As far as I can tell the backplate is not sold with the cooler, so I would suppose it's already installed on the mobo.
The list in the link I posted previously Is not comprehensive, in the end you just need a LGA115X cooler with threaded supports that can be screwed in the backplate extrusions (ie I am using this).
Anyway, you could just buy the mobo first and then the cooler once you verify if It has or not the backplate.

The WD is a WD40EFRX which I looked up is a CMR drive.
I do plan to upgrade the old drives when cash flow permits. I too am concerned about them.
I can do all Hitachi's (HGST Ultrastar HUS724040ALE640 4TB 64MB 7200RPM SATA 6Gb/s)
They are about the same price as the Seagate IronWolfs.

Indeed, the WD should be CMR, congrats!
A good pool design can spare you lots of troubles.
Personally I am using IronWolfs but HITACHIs are renowed too. That specific model at the very least has higher RPM than the Seagate's model you intended to use though.
Check what you have, I wouldn't mix disks with different RPM.

The mobo allows up to 128GB or RAM, is there a benefit to upgrading to that amount of RAM? I was planning on upgrading in the future, but I am not sure it is necesarry.

If you go with CORE, 64GB (4x16) should be enough.
It all boils down to your use, if you go SCALE you could start with two sticks of 32GB each (total 64 then) and eventually add more if you see the need for.
You could even just start with a single 32GB stick: you wouldn't be able to use dual-channel but, although I am not too sure, it shouldn't impact performances too much.
Anyway, I wouldn't go lower than 32GB.
That CPU supports up to 64GB.

 

[gdarends]

Reading the specifications it says that it's compatibile with supermicro's passive cooler that seems to need the backplate to be installed. As far as I can tell the backplate is not sold with the cooler, so I would suppose it's already installed on the mobo.
The list in the link I posted previously Is not comprehensive, in the end you just need a LGA115X cooler with threaded supports that can be screwed in the backplate extrusions (ie I am using this).
Anyway, you could just buy the mobo first and then the cooler once you verify if It has or not the backplate.

Indeed, the WD should be CMR, congrats!
A good pool design can spare you lots of troubles.
Personally I am using IronWolfs but HITACHIs are renowed too. That specific model at the very least has higher RPM than the Seagate's model you intended to use though.
Check what you have, I wouldn't mix disks with different RPM.

If you go with CORE, 64GB (4x16) should be enough.
It all boils down to your use, if you go SCALE you could start with two sticks of 32GB each (total 64 then) and eventually add more if you see the need for.
You could even just start with a single 32GB stick: you wouldn't be able to use dual-channel but, although I am not too sure, it shouldn't impact performances too much.
Anyway, I wouldn't go lower than 32GB.

Think I'm gonna go with the Arctic Freezer 34 CO. In the recommendation list you sent it lists it as YES / YES for hole / backplate. So I guess that should work for both cases.

Also, I'm just gonna get 4 Hitachi's and use the WD elsewhere.

 

[Davvo]

Think I'm gonna go with the Arctic Freezer 34 CO. In the recommendation list you sent it lists it as YES / YES for hole / backplate. So I guess that should work for both cases.

Yup, solid choice. Just make sure the sizes are right.

Also, I'm just gonna get 4 Hitachi's and use the WD elsewhere.

You could use it as an hotspare for the time being.

 

[Davvo]

@G. Arends just realized that CPU supports up to 64GB of RAM.
Sticks of 16GB is the way.

 

[gdarends]

@G. Arends just realized that CPU supports up to 64GB of RAM.
Sticks of 16GB is the way.

Good catch!
Thanks

 

[BCB57]

Hello,

I have a QNAP NAS that's been running for a little over 10 years now and have been meaning to upgrade to Truenas/ZFS for a while.

I've been reading some guides and have come up with the following hardware specs:

Motherboard	Supermicro X11SCH-F Motherboard Micro-ATX Single Socket H4 (LGA 1151) for Intel 8th Generation Intel Core
CPU	Intel Core i3-9300 SRCZU 3.70GHZ 4C 4T 8NB 62W LGA1151 CPU Processor
CPU Cooler	Cooler Master Hyper 212 Black Edition CPU Air Coolor, Silencio FP120 Fan, 4 Copper Direct Contact Heat Pipes for AMD Ryzen/Intel LGA1200/1151
RAM	Supermicro (Micron) 32GB 288-Pin DDR4 2666 (PC4-21300) ECC UDIMM
Boot Drive	Kioxia 256GB SSD XG6 M.2 2280 PCIe Gen3 x4 SED Encryption NVMe KXG6AZNV256G Solid State Drive
PSU	EVGA SuperNOVA 550 G3, 80 Plus Gold 550W, Fully Modular, Eco Mode with New HDB Fan
Chassis/Case	Fractal Design Node 804 - Black - Cube Compact Computer Case - mATX
Storage	3x Seagate IronWolf 4TB NAS Internal Hard Drive HDD – CMR 3.5 Inch SATA 6Gb/s 5900 RPM 256MB Cache (ST4000VN006)

I'm planning to do a RAIDZ-2 with 6x4TB HDD (I'm only buying 3 new ones cause I already have 2 Hitachi and a WD)
I will be using the machine for backup. But I also have a list of services that I run such as:
- PiHole
- Unifi Controller
- NUT Server
- Plex & Friends
- SynchThing

So, does the specs hold up for what I'm trying to do with the system? Shoud I bump up the RAM to 64GB?
Should I add SSD for virtualization and opt for TrueNAS Scale?

This is probably a controversial suggestion here, but you may want to consider the "consumer grade" option for some of your hardware. My homebuilt TrueNAS file/media server has been running flawlessly since I put it together eight months ago, using NAS-rated HDDs with mid/high-end consumer PC components for the rest of the hardware.

Here's the build and part list: https://pcpartpicker.com/b/mf3tt6 As noted in the remarks, my only regret was not getting a motherboard with more SATA ports. The PCIE expansion card I added to get around this limitation (for storage expansion) is probably the weak link in the system now, but it's been humming along fine since its installation in March. System boots and runs just fine "headless," although I added an inexpensive small monitor for use whenever needed. So far, so good... and of course all my important data is backed up elsewhere.

 

[Davvo]

@BCB57 my bottom line is ECC RAM: imho it makes no sense to use ZFS without It.
That's what prevented me from using consumer-grade parts (mainly MOBO).

ECC vs non-ECC RAM and ZFS

I've seen many people unfortunately lose their zpools over this topic, so I'm going to try to provide as much detail as possible. If you don't want to read to the end then just go with ECC RAM. For those of you that want to understand just how destructive non-ECC RAM can be, then I'd encourage...

www.truenas.com

You spent a few hundreds to build a storage space, but you can't trust it to keep your files safe (as you said, your mission critical datas are elsewhere).
I am perplexed, but if it works for you (maybe It was an experiment?) it's ok I guess.

 

[gdarends]

@BCB57 my bottom line is ECC RAM: imho it makes no sense to use ZFS without It.
That's what prevented me from using consumer-grade parts (mainly MOBO).

ECC vs non-ECC RAM and ZFS

I've seen many people unfortunately lose their zpools over this topic, so I'm going to try to provide as much detail as possible. If you don't want to read to the end then just go with ECC RAM. For those of you that want to understand just how destructive non-ECC RAM can be, then I'd encourage...

www.truenas.com

You spent a few hundreds to build a storage space, but you can't trust it to keep your files safe (as you said, your mission critical datas are elsewhere).
I am perplexed, but if it works for you it's ok... I guess.

My sentiments exactly. I wen't from QNAP, which worked fine, but was always concerned with data integrity, and I have noticed oddities. I had an mp3 file which had a different mp3 mixed in. So it would play both songs, as if they were merged, at least parts of it.
So yeah, ZFS and particularly with ECC memory was on the priority list.

 

[Etorix]

I did a quick search and the 9100 is more expensive than the 9300. Maybe because it comes with the cooler.

Wow! then, if you bring your own cooler (Intel stock would do fine…) go for the 9300 and its extra MHz.
Supermicro X11SC boards have an integrated backplate, which works fine with standard LGA1151-2 coolers (stock Intel, Noctua).

The mobo allows up to 128GB or RAM, is there a benefit to upgrading to that amount of RAM? I was planning on upgrading in the future, but I am not sure it is necesarry.

The limit is in the CPU. Higher-end Xeon E3 may allow more. But if you want to go beyond 64 GB it would advisable to for a platform which uses RDIMM.

The server motherboard with IPMI (and ECC) is the way to go for ZFS. Don't bother with consumer parts.

 

[BCB57]

@BCB57 my bottom line is ECC RAM: imho it makes no sense to use ZFS without It.
That's what prevented me from using consumer-grade parts (mainly MOBO).

ECC vs non-ECC RAM and ZFS

I've seen many people unfortunately lose their zpools over this topic, so I'm going to try to provide as much detail as possible. If you don't want to read to the end then just go with ECC RAM. For those of you that want to understand just how destructive non-ECC RAM can be, then I'd encourage...

www.truenas.com

You spent a few hundreds to build a storage space, but you can't trust it to keep your files safe (as you said, your mission critical datas are elsewhere).
I am perplexed, but if it works for you it's ok I guess.

Of course it's backed up... that ECC RAM won't help you in a fire, flood etc. I'd never rely on a single point of failure. As to the other comments about ECC, I'm sure there is technical merit to them. Will be certain to come back here and eat plenty of crow if/when that time comes. :)

 

[Davvo]

Of course it's backed up...

Sorry, didn't read the "backed up" in your original post.
3-2-1 is the holy way.

edit: be aware that if you are basing your backups on the files in that nas (like cloud sync or replication) they might not be intact.

Anyway, I am interested in @G. Arends 's pool layout, having solved the reliability issue you still don't want to go with that single vdev of 6 disks in RAIDZ2 for I/O limits.

 

[Etorix]

I/O limits on a backup pool? IMHO, a single raidz2 is just fine for mostly being a replication target, and even with used drives in the mix it is unlikely that three drives will fail at the same time.

 

[Davvo]

I/O limits on a backup pool? IMHO, a single raidz2 is just fine for mostly being a replication target, and even with used drives in the mix it is unlikely that three drives will fail at the same time.

I just prefer the flexibility and ease of expansion, with a single 6 disks vdev he might be uncomfortable in the future if his needs change.
Besides, I am worried by the resilvering.
edit: Anyway, resource is here.

Picking a ZFS Pool Layout to Optimize Performance

Part 1 of the blog post is here, covering basics, striped pools, and mirrored vdevs: https://www.ixsystems.com/blog/zfs-pool-performance-1/ Part 2 covers RAID-Z and some example workload scenarios...

www.truenas.com

 

[gdarends]

I just prefer the flexibility and ease of expansion, with a single 6 disks vdev he might be uncomfortable in the future if his needs change.
Besides, I am worried by the resilvering.
edit: Anyway, resource is here.

Picking a ZFS Pool Layout to Optimize Performance

Part 1 of the blog post is here, covering basics, striped pools, and mirrored vdevs: https://www.ixsystems.com/blog/zfs-pool-performance-1/ Part 2 covers RAID-Z and some example workload scenarios...

www.truenas.com

My expansion plan might be 4xHDD RAIDZ-1. But not sure yet. Depends on if I go CORE or SCALE. I'm leaning towards CORE.
But if performance improves on SCALE, I'm thinking to do mirror of SSD.

 

[joeschmuck]

My expansion plan might be 4xHDD RAIDZ-1.

unless you plan to build several pools/VDEVs, you are best to start out with the correct number of hard drives up front. You need to know the capacity you need for data storage for the next 5 years (most drives will last about 5 years), and then double that capacity if you are a storage hog. So you have capacity now. Next you need redundancy, how many drives can fail before you lose all your data? Many of us use RAIDZ2 and two drives can fail. Next, how many drives do you want to have in your system? Some folks are limited by case size or other factors. I'd recommend using a RAID-Z calculator (like one of my links below) to play around with the numbers. Maybe you only need 5TB and RAIDZ2, then you could see if it's 4 drives, 5 drives, or even 6 smaller drives. Maybe you want 4 drives so you can space them out and keep them cool (that is what I do now).

Pro's and Con's to think about: Smaller physical drives resilver faster. Many smaller drives crammed together get hotter. Larger drives are generally cheaper for capacity per dollar. To upgrade to a larger capacity you need to replace all the drives, it's not incremental, more drives could cost much more money. If you want to change the number of drives in your system, yo need to destroy your pool and rebuild it.

I really like TrueNAS (I started when is first came out as FreeNAS) and I have no complaints. You get what you put into it. It sounds like you are going down the right path, getting a good quality motherboard, putting in 32GB RAM (Question: is that a single stick or two sticks? I would go for two 16GB sticks so you run in Interleave mode, and if one should fail, you still have the other to run the system.) Some folks really love that 804 case but myself, it's too small. You are cramming a lot into it. If this is the path you go down, when you put it together just think Air Flow! Use zip ties to bundle the cables nicely, if you need to add more fans or better fans, do so (you may not know this until after it's together), and keep it in a cool place. Check those system and hard drive temps. If you are running near 50C, that is too hot in my opinion to continuously run. You can look in my description for my case and you will see a fair sized full case, lots of airflow, cool drives. And go figure, it's quiet too.

Mixing drives from different placed, manufacturers, lots, etc was a thing done more in the past. Drive manufacturers had a period of time that things were just not going well. These days most manufacturers are doing better, just stay away from Archive/SMR drives.

Best of luck to you.