NAS for software development, CCTV, VMs, "backup"

[JustJason]

Hi all!

I was hoping someone could chime in on a FreeNAS box design I was considering.

USE:
I need it to serve files and if possible to run MySQL (or other DB) in a jail or VM. No transcoding or multimedia stuff.
It will be used as a filesystem to a dev environment and also to store CCTV content.
The former meaning NFS - mostly to serve VM images (ie random seeks, r/w mix, in large files) as well as access to lots of small files (git, SVN). The latter (CCTV) is only as a backup store of 4k clips, and not performance critical.
In the future I may consider iSCSI for the VM images, but I haven't had much luck with that in the past, so probably not.
I'll also do daily replication of a remote ZFS system (low traffic).
There'll be occasional disk imaging of local laptops for backup.
There'll be some SMB3, but not on the same datasets as the NFS.
Some of the data sets (files) I work on can be quite big (time-series multi gig data).

CONFIG:
I was thinking going RAID-Z3 with 7 HGST He12 drives (because the case I chose has 8 bays, so why not Z3 over Z2).
I won't be using de-duplication but I'd like to give LZ4 compression a try.

HARDWARE:
The HGST He12 (4kN) drives are 12TB and have the same £/TB as the He10 10TB drives, which have better pricing than the He10 8TB.
Many of the HGST drives come in SE, ISE, other variants.. this is to do with disk encryption and erasing. I won't be using any of that but I plan to mix SE and ISE drives so that there's a higher chance that they come from different batches (and lower the chance of correlated failure).

In terms of Mobo+CPU, initially I thought Supermicro X11SSH-CTF (maybe X11SSM-F) and E3-1260L v5 (45W) because I wanted a bit of power and no more than 45W TDP.
Alas the E3-1260L v5 seems to be difficult to buy new in the UK.

The alternative would be Supermicro A2SDi-H-TF (C3785).

I'm thinking 64GB ECC RAM. (vs 84 TBs of disks). I might be able to push 96GB, but only on the A2SDi-H-TF as the X11SSH-CTF only supports 64GB. Also, DDR4 ECC RAM isn't cheap.

SSD (maybe dual) as a boot drive obviously, but I was also considering using the NVMe M2 slot for a ZIL (depends on where I end up running my MySQL instance). The ZIL decision will come after I have the system up and running and performance profiled.

I'd obviously like to saturate 1GbE, and I chose 10GbE ports for future-proofiness. My future build and compute servers will all have 10GbE.

So:
X11SSH-CTF + E3-1260L v5 OR A2SDi-H-TF (C3785)
64GB ECC RAM
7 * He12 12TB as RAID-Z3.
NSC-810A case (8*hot swappable 3.5")

Any thoughts?

With compression and variable blocks sizes does the 2^N+3 number-of-disks rule for RAID-Z3 still hold?

Many thanks,

Jason

PS I'll re-purpose an HP N54L as a RAID-Z2 secondary "backup" server, and eventually also get a tape drive for real backup.

 

[kdragon75]

Welcome to the forums @JustJason. I have looked over your intended uses and have a few point to make.
The big one will be around performance for small random IO. Using RAIDz2/3 with synchronous writes (STRONGLY encouraged for databases and VMs) will be painfully slow. A SLOG may help but it will still never be faster than your backing pool. Generally for database/small file IO striped mirrors give MUCH better performance and your IO capacity is largely a function of the number of vdevs (more information). The downside is that you lose half of your capacity to redundancy. For the database, the data set should be set with a record size that matches your database.

I was thinking going RAID-Z3 with 7 HGST He12 drives (because the case I chose has 8 bays, so why not Z3 over Z2).
I won't be using de-duplication but I'd like to give LZ4 compression a try.

If you use striped mirrors over RAIDz3, you are only losing one more disk and will have much better performance for your VMs. As for lz4, definitely use it for your VMs. I have a compression ratio of about 1.5! You can turn it off for archiving compressed video as it wont save much if any and will jut waste CPU trying. I see the same thing on my photo and backup archives as the files are generally compressed before they reach the NAS.

The alternative would be Supermicro A2SDi-H-TF (C3785).

Thats a great board if your running LOTS of small VMs but at only 2GHz may feel sluggish.

 

[JustJason]

Thank you for your response @kdragon75 and the link and for the welcome :)

Tell me, what specific configuration were you thinking about when you said striped mirrors?

I'm happy to have 8 drives, or 6, or 7, but AFAICT if I stripe across, say, four mirrors of two drives (a mirror here is a vdev) then I'd have data loss after losing two drives (assuming they're from the same vdev/mirror set).
Is that what you meant?

So it's a choice between RAID-Z3 with a tad more usable space but worse performance, or striped mirrors with much better performance but lower MTTDL and less usable space.

My VM and DB work isn't performance critical, but if it were I do also have other options..
I could use a local (to the compute host/workstation) NVMe drive and only copy/restore the VM or DB when I jump between projects (very cumbersome).
Or I could buy an additional HP Microserver Gen10, set up FreeNAS with (non-mirrored) striped disks (maybe even SSDs), and have the ZFS replicate to the main fileserver every night. The checksumming in ZFS on the Gen10 would detect errors in a stripe, right? Then I could do away with mirrors on this additional server. The main server would remain on RAID-Z3.
Or.. actually.. this is dev work.. so asynchronous writes may not be the end of the world.

Is the daily replication a bad idea?

I guess it comes down to this - what is "painfully slow"?

Do you think I'd be ok with 64GB RAM?

Thanks.

 

[kdragon75]

I'm happy to have 8 drives, or 6, or 7, but AFAICT if I stripe across, say, four mirrors of two drives (a mirror here is a vdev) then I'd have data loss after losing two drives (assuming they're from the same vdev/mirror set).
Is that what you meant?

Exactly.

Or I could buy an additional HP Microserver Gen10, set up FreeNAS with (non-mirrored) striped disks (maybe even SSDs), and have the ZFS replicate to the main fileserver every night.

You could do that but a RAID0 style strip is something I wouldn't touch with a 10 foot pole. Evcn with ZFS if you have an error on a pure stripe array, you will have damaged data. It will tell you, but your still going to back for that file under the best case.

The checksumming in ZFS on the Gen10 would detect errors in a stripe, right?

Detect and correct.

so asynchronous writes may not be the end of the world.

Trust me... Just get the SLOG and keep everything sync. Do you really want to roll back or rebuild your tool chain/ dev environment in the middle of your work? Its just not worth the hassle of corrupted VMs.

Is the daily replication a bad idea?

Replication is never a bad idea!

I guess it comes down to this - what is "painfully slow"?

Build it and test! If your the only user, the performance may not be a huge deal.

 

[Ericloewe]

E3-1260L

The platform is fine, but drop the L. It doesn't do you any good, only hinders you.

 

[JustJason]

The platform is fine, but drop the L. It doesn't do you any good, only hinders you.

I was *just* about to order the 1260L V5.
I'm worried about noise and heat of a CPU with >45W TDP. Most E3 V6's have a TDP ~72W. The 1260L V5 is 45W.
Could you tell me how the low power versions hinder?

The Passmark scores for the 1260L V5 (Skylake) is approx the same as the 1240 V6 (Kaby Lake). They're both around 10100 for multi-core and 2100 for single core.
The only disadvantage I see of the 1260LV5 is availability and slightly older micro-architecture.
Having said that the HT bug in SkyLake and Kaby Lake was fixed with new microcode a few years ago.

@kdragon75 Thank you so much for your response. I went off to do lots of research before replying but got sidetracked. Will reply ASAP.

 

[Ericloewe]

The lower-power version isn't going to be more efficient, it's just artificially capped.

The Passmark scores for the 1260L V5 (Skylake) is approx the same as the 1240 V6 (Kaby Lake). They're both around 10100 for multi-core and 2100 for single core.

Then it's a crap benchmark because the clocks are very different on those two.

 

[jgreco]

I was *just* about to order the 1260L V5.
I'm worried about noise and heat of a CPU with >45W TDP. Most E3 V6's have a TDP ~72W. The 1260L V5 is 45W.
Could you tell me how the low power versions hinder?

The low TDP CPU is generally a bad choice unless you actually have a design constraint, such as "inability to dissipate more heat" - and that means unABLE to, not "I just want it to run cooler and I think this is how to accomplish that."

There's actually a perverse result here in that if you get the standard TDP part, and run a CPU-heavy workload on it, it runs momentarily hotter to get the workload done, but it gets it done fast, and therefore it falls back down to idle very quickly. The idle between the low-TDP and standard parts are identical. So if you select a low TDP CPU that's got to be maxxed out for a longer period of time, not only does your workload take a lot longer to complete, but also it may actually burn more energy overall doing so.

So the point here is this: if you want the CPU to run cool, don't give it work to do. If you give it work to do, you want it done as efficiently as possible. The low-TDP part guarantees low-TDP by damaging the CPU's ability to be as efficient as possible at getting your job done, which doesn't actually save energy, but does make your workload slower, and may actually make it burn more energy too. Low TDP parts are available to prevent devices from self-destructing in environments where the high TDP parts won't work because the heat can't be dissipated that quickly.

CPU's don't make noise, by the way. The cooling might, but you can compensate by using a large heatsink and larger fans. I encourage this.

 

[Arwen]

Here is another example, though not Intel or server related.

I bought a PC with a single core AMD64 Sempron, 2.8Ghz, 45watt. Part of the reason was that I could upgrade the CPU easily. Normal use was fine, but updates to the OS & apps, (very disk intensive), were slow. So I bought a quad core AMD64 Athlon IIx4, also 45watts, but only at 2.6Ghz. The upgraded CPU of course ran rings around the old one.
Two un-expected side effects;

1. The speed adjusting CPU fan ran slower, (meaning noticably quieter), with the new CPU when the PC was idle.
2. The idle power draw was noticably less, according to my kill-a-watt meter.

On the other comment about CPUs with limited TDP. It's not just about the heat. It could be about the power supply. Certain embedded server applications may use limited power delivery, in addition to limited cooling.

 

[jgreco]

Here is another example, though not Intel or server related.

I bought a PC with a single core AMD64 Sempron, 2.8Ghz, 45watt. Part of the reason was that I could upgrade the CPU easily. Normal use was fine, but updates to the OS & apps, (very disk intensive), were slow. So I bought a quad core AMD64 Athlon IIx4, also 45watts, but only at 2.6Ghz. The upgraded CPU of course ran rings around the old one.
Two un-expected side effects;

1. The speed adjusting CPU fan ran slower, (meaning noticably quieter), with the new CPU when the PC was idle.

This may be a bit of an extreme case due to the increase in core count and generational evolution, but it is certainly valid along the general lines of discussion. The Sempron is substantially less competent than the Athlon, though, usually sporting less cache and in your case I'm guessing an older design. You got a hell of a whammy of an upgrade there.

A quad core CPU that is only using one or two cores on average is likely to consume less power and also (with modern CPU's) likely to be able to turbo the ones that are active beyond the CPU's rated speed, all of which are big benefits.

1. The idle power draw was noticably less, according to my kill-a-watt meter.

On the other comment about CPUs with limited TDP. It's not just about the heat. It could be about the power supply. Certain embedded server applications may use limited power delivery, in addition to limited cooling.

Well that's what I meant by "design constraint." ;-)

 

[JustJason]

Thank you all very much for your comments.

So the choice is between E3-1260LV5 (Skylake) and E3-1270V6 (Kaby-Lake).. 45W vs 72W, 2.9/3.9 GHz vs 3.8/4.2Ghz.

I've done some digging and there is no IPC difference between Skylake and Kaby Lake so the only difference between the two is in CPU clock. (ok, and 2400MHz RAM vs 2166 MHz, and a few more PCIe lanes).

I also cannot underclock the CPU in BIOS because the X11SSH-CTF doesn't support it AFAICT.

So although I was sold on the 1270V6 briefly I think I'm going to stick with the 1260LV5 - the NSC-810A is a relatively cramped case and the airflow isn't great. There's limited CPU clearance, so I've chosen the Noctua NH-L9i as the cooler. That's rated to 65W TDP, but it can cool more with good case airflow.

The limited cooling is a design constraint (jgreco FTW) and I want a guarantee that the CPU won't dissipate more than 45W which it will do with a 72W part if a process misbehaves.

I'm also not 100% sure that a low-TDP CPU is like that just because of a lower clock - we're either at 2.9GHz (non-turbo) @ 45W, or 3.8GHz at 72W. That's a 31% clock bump and a 60% TDP bump. It should be more linear (ie TDP bump should be closer to 31%). This would suggest the -L variant is also from a different bin and may not have such a performance hit.

What clinched it, though, was measured performance. I agree that the PassMark results were iffy, but check this out:
https://www.phoronix.com/scan.php?page=article&item=wild-linux-40&num=2
There's some question about the scientific method (he uses different RAM,SSD - only 8GB on 1260LV5, hence slow SQLite) but the 1260LV5 is really holding its own in most tests, at least against non -L Skylake versions.

With the safety guarantee of a 45W TDP, I'm going to keep trawling for places to source a E3-1260L V5.

EDIT: Here's a test of different Skylake Xeons:
https://www.phoronix.com/scan.php?page=article&item=intel-skylake-xeon9&num=3
For the most part the E3-1260LV5 is very competitive with the E3-1280V5.
Especially here in the Linux Timed Kernel compilation:
https://www.phoronix.com/scan.php?page=article&item=intel-skylake-xeon9&num=5
A kernel build is closer to the FreeNAS processes than the compute intensive benchmarks, and the -L is only 3% slower.
Of course, the 1280 doesn't seem to run that much hotter than the 1260L, so it's perhaps not so cut and dry:
https://www.phoronix.com/scan.php?page=article&item=intel-skylake-xeon9&num=8