Skylake Build

[zuzuzzzip]

Hello Everyone,

Glad to join this very knowledgeable community of FreeNAS users!

I've been looking at building a ZFS box for a while now and have finally come up with the following setup:

• CASE: Real Power RPS19-G3380
• MOBO: Asus P10S-E/4L
• CPU: Intel Xeon E3-1260L v5
• RAM: 2x Samsung 16GB DDR4 ECC 2133MHz
• DISK: 6x HGST DeskStar NAS 4TB
• PSU: SeaSonic SS-400FL2

I am still awaiting my miniSAS HD to 4x SATA cable.
Then I will install FreeNAS :)

Initially I had a Haswell build setup, but due to complications with delivering the motherboard and waiting for 4+ months, I cancelled the order and went for Skylake.

Initially I thought to go for a single RAIDZ2 vdev with the 6 disks.
But reading on about RAID10 vs RAIDZ2 ... (See link)
I'm thinking to set up 3 mirror vdev's.

What do you guys think?

Grtz!

 

[jgreco]

Since we have no idea what your use for the filerserver is, there's no way for us to advise you on RAIDZ2 vs mirrors.

RAIDZ2 is the hands-down winner for storing and retrieving large sequential files. It protects your data well at a modest cost.

Mirrors are the only way to go if you're doing random access work, such as heavy database work, VM storage, etc., but to get the same level of data protection as RAIDZ2, you have to go with three-wide mirrors, which is only 33% space efficiency.

 

[zuzuzzzip]

Sorry, should have mentioned it earlier.

It's for an (maybe above average) home build.
It will mainly be used for storing media, documents and backup files.
I will also be running a few VMs (say 5, max 10) and using part of this storage as iSCSI or NFS datastore.

Also, not sure if I should make 2 pools to seperate the use cases ... but then I will be more limited in space.
So I prefer not to do this.

 

[SweetAndLow]

You will want mirrors if you plan to use iscsi and run vms. I would also suggest skipping the L processor but to late for that

 

[zuzuzzzip]

Yeah, it was important for me that it would as energy-efficient as possible.
It's just for a homelab so no biggy if VMs don't perform that great.

I'll probably run FreeIPA in a vm and then maybe a docker host with 3 containers or something. Just for testing :)

If I would use NFS in stead of iSCSI, would that change your advice?

 

[jgreco]

Yeah, it was important for me that it would as energy-efficient as possible.

The L CPU isn't more energy efficient. The L CPU basically just caps the amount of power that could possibly be used at a given moment. It's meant for situations where the host unit isn't ABLE to cope with additional TDP or power consumption.

For example, if you have an "L" CPU, that runs at 2.4 GHz and the Sandy version PassMark's at 6500. But a conventional CPU will do a good bit better. Both CPU's idle at the SAME power consumption. But you've selected a CPU that's slower, so to complete a task that might momentarily spike a conventional Xeon's power consumption up for a little bit, your CPU has to rev up for maybe twice or even thrice as long.

This is a substantial factor because for NAS the CPU is doing the parity calculations for RAIDZ, protocols, etc. You haven't actually saved yourself any power, and in fact it might be using more power because it takes a much longer time to do the work.

So if you're in a colocation center where you have a hard power budget of 100 watts, and your build consumes 40 watts before the CPU, then, yes, something like the L CPU is a good choice because it guarantees that you're going to stay within that envelope. But for the average user, they usually have no business buying an L version, they should just buy a standard CPU. Cheaper, faster in the cases where that's needed, etc.

It's just for a homelab so no biggy if VMs don't perform that great.

I'll probably run FreeIPA in a vm and then maybe a docker host with 3 containers or something. Just for testing :)

If I would use NFS in stead of iSCSI, would that change your advice?

Block storage is memory intensive, and really wants mirrors if you're going to do anything more than a trite VM or two. You may want more than 16GB.

 

[jgreco]

And that last comment means that it isn't about iSCSI, it's about block storage. Works the same on NFS, iSCSI, CIFS, AFP, or diddling blocks locally with bhyve.

 

[zuzuzzzip]

I'm sorry, I forgot to add "2x". (adjusted OP)
I have 32GB of RAM in total.

Damn, should've posted here before I bought the L CPU ...
The specs pages don't tell you this kind of thing.

We'll see how it runs tomorrow, but will the impact be so high that I should change the CPU?
By the sound of your comment, it does ...
If I run a bench tomorrow, would the results show me the impact?

 

[jgreco]

I'm sorry, I forgot to add "2x". (adjusted OP)
I have 32GB of RAM in total.

That is very good.

Damn, should've posted here before I bought the L CPU ...
The specs pages don't tell you this kind of thing.

Yeah, I wish they'd actually give us idle consumption figures. The best we've been able to determine is that they're very close and quite possibly identical.

We'll see how it runs tomorrow, but will the impact be so high that I should change the CPU?

Not really sure. The thing that works out for you here is that a NAS isn't necessarily flatlining its cores, and the 1260L v5 can actually turbo up to 3.9 when only a single core is calling for it. That's not bad. By comparison, the 1230 v5 is $50 cheaper, has a base speed of 3.4, and turbos up to 3.8 (but I think it'll do it on at least two cores). I'm too lazy to actually dig up the turbo specifics. For a single core, they'd be indistinguishable. But also they'd probably be using just about the same amount of power.

The big point is more that the L CPU's are irrational from a "power saving" point of view. Everybody seems to think they save power, but that isn't their purpose. When your system actually wants to get some compute work done, it should be fine to burn watts to get that done quickly. If you actually can't afford to let it do that, such as you've got 16 machines that barely fit into a 20A circuit and having half of them "let loose" (running wild with an extra 50W each) would pop a circuit breaker, then the "L" CPU makes good sense.

By the sound of your comment, it does ...
If I run a bench tomorrow, would the results show me the impact?

If you have something to compare it to, probably. I don't see any stats out there for the v5's. Here's some legacy Sandy stuff to show:

https://www.cpubenchmark.net/compare.php?cmp[]=1203&cmp[]=1199

The "L" CPU there is about 4/5ths the speed of the plain 1230, but is priced at a premium. The modern stuff will have different numbers.

 

[Bobbi]

The L CPU isn't more energy efficient. The L CPU basically just caps the amount of power that could possibly be used at a given moment. It's meant for situations where the host unit isn't ABLE to cope with additional TDP or power consumption.

For example, if you have an "L" CPU, that runs at 2.4 GHz and the Sandy version PassMark's at 6500. But a conventional CPU will do a good bit better. Both CPU's idle at the SAME power consumption. But you've selected a CPU that's slower, so to complete a task that might momentarily spike a conventional Xeon's power consumption up for a little bit, your CPU has to rev up for maybe twice or even thrice as long.

This is a substantial factor because for NAS the CPU is doing the parity calculations for RAIDZ, protocols, etc. You haven't actually saved yourself any power, and in fact it might be using more power because it takes a much longer time to do the work.

Incredible! You just saved my purse from paying 300,- EUR for a Xeon E3-1240L! I didn't know about that stuff... Thank you very much!!!

 

[Stux]

Other reason (same reason really) is if you're thermally constrained. Ie big NAS in a tiny box.

 

[zuzuzzzip]

In the end I was forced to mount my CPU cooler in some ghetto way.
So in my case (get it?) it was a good thing to get the L cpu as my system would otherwise most likely run too hot :).

All-in-all the performance has been terrific too, so no regrets so far.