How much RAM do I really need?

[djdwosk97]

Okay, hopefully this will be my last thread on the matter before I I make a purchase. So I've been going back and forth and I just can't decide what it is that I want to do.

I have three options and I keep bouncing back and forth between the three but my main point of concern is RAM capacity. Right now and for the foreseeable future my storage setup will be as follows:
---Two 1/2tb drives (mirrored) for backup purposes (photos, documents, etc...)
---Two raidz2 arrays consisting of five 8tb drives each (currently I don't have this, but I think this will be the max I'll ever need to go during the lifetime of the server) intended for media storage and streaming via plex.

So would 32gb of ram be sufficient for my needs assuming I leave the movies uncompressed (so 30-40gb each), and assuming 4K bluray rips will be like 80gb each (probably?) and assuming there will be 4 clients streaming at any one time (at most).

And would backing up my server to a second (offsite) server increase the ram requirements -- if so how much would k need then?

My three options are spend $370 for a haswell i3 system (x10ssl), $510 for a skylake i3 system (x11ssm), or $515 for a Xeon 2603v3 (x99) system. (All using 16gb ECC with the haswell i3 limited to 32gb of ram, skylake i3 limited to 64gb, and the Xeon e5 limited to 128gb). The Xeon seems like an all around better choice than a Skylake i3 other than the fact that the Skylake build uses a more standard server config.

1. Haswell i3 - $370
   1. i3-4130 ($110) (and only upgrade to an i3 because buying a G3258 would be silly I think)
   2. Supermicro X10SSL-F-O ($140)
   3. 16gb ECC DDR3 ($110)
2. Skylake i3 - $510
   1. i3-6100 ($120)
   2. Supermicro X11SSM-F-O ($210)
   3. 16gb non-registered ECC DDR4 ($140)
   4. SSD ($40)
3. Xeon E5 - $515
   1. Xeon E5 2603v3 ($220)
   2. Asrock X99 Extreme4 ($165)
   3. 16gb registered ECC ($115)
   4. Cooler Master Hyper T4 ($15)

I was leaning towards just upgrading to a Haswell i3 ECC system, but then I thought about the 32gb RAM limitation (not that it's necessarily a limitation that will ever apply to me).

 

[Mlovelace]

Why the two pools? Seems like a waste of two ports to have mirror 0.5TB drives. Just make another dataset for the backups on the main pool and do 12x 8TB drives. I'd go with the x99 build in case you find you need more memory later.

 

[Mirfster]

So would 32gb of ram be sufficient for my needs...

I would say that you are fine with 32; especially since you did not mention running any VMs.

 

[djdwosk97]

Why the two pools? Seems like a waste of two ports to have mirror 0.5TB drives. Just make another dataset for the backups on the main pool and do 12x 8TB drives. I'd go with the x99 build in case you find you need more memory later.

Well the reason being that I have extra drives and extra ports right now -- right now I have two (shitty) 1 tb drives mirrored for my critical data and in the future I'll likely upgrade to two 2tb drives mirrored. As long as I can spare the ports I'd prefer my critical data to be mirrored. Would there be any reliability concerns going with an Asrock x99 extreme4 and 16gb registered ECC memory (as far as I can tell they're compatible).

 

[djdwosk97]

I would say that you are fine with 32; especially since you did not mention running any VMs.

Yeah, I don't see myself running any vm's just using it for a media server, a couple tb of permanent storage, and maybe a weekly backup of two-three computers locally.

What would the max amount of storage be for 32gb of ram to still be enough assuming raidz2 configs and would that amount change if I wanted to have a backup server in a different location?

 

[Mirfster]

If I recall the recommendations are 1 gb of ram for every 1 tb.

CyberJock's "Hardware recommendations (read this first)" mentions:

ZFS loves to use memory. The more you have the faster it will be. If performance starts slacking that's your queue to add more RAM. 8GB is the minimum for FreeNAS and do not go below that. You aren't as special as your mommy told you and you risk your data if you think you are.

For most home users 16GB is a very good sweet spot. If you plan to run lots of jails like Plex or Minecraft you should consider going with 32GB of RAM. It is better to have too much than too little.

Be careful bout filling your slots with low-density RAM. It is better in the long term to buy 2x8GB sticks than 4x4GB sticks as this save you money in the long term when upgrades are going to be desired.

So based on 8gb (minimum) for FreeNas and the 1gb for every 1tb, it should be safe to assume that when using 32gb you can think 24tb of raw space. If you have more ram, then all the better. FreeNas loves ram and it doesn't hurt to have more.

I don't think that ram would be the bottleneck on server location, more than likely it will be connection speed.

 

[djdwosk97]

If I recall the recommendations are 1 gb of ram for every 1 tb.

CyberJock's "Hardware recommendations (read this first)" mentions:


So based on 8gb (minimum) for FreeNas and the 1gb for every 1tb, it should be safe to assume that when using 32gb you can think 24tb of raw space. If you have more ram, then all the better. FreeNas loves ram and it doesn't hurt to have more.

I don't think that ram would be the bottleneck on server location, more than likely it will be connection speed.

That 1gb per tb has to reach a point of diminishing returns (depending on the use case). Seeing as I'm not doing anything intensive -- just streaming media my guess is that point would be <=32gb of RAM. Meaning that 32gb of RAM is probably sufficient for two raw 40tb RaidZ2 array (24tb usable each) and then some. But again, that's just my uneducated guess. As it is I'm running three 4tb drives (JBOD) and two 1tb drives (mirrored) off of 8gb of RAM with an ARC hit ratio of 80%~ and the performance is sufficient for my use case (two users can stream without any stuttering issues at all).

 

[Mirfster]

True there most certainly would be a point of diminishing returns somewhere. Keep in mind that the recommendation is a general one. To be honest I don't think that FreeNas would not work if you had 16gb and 200tb of storage. It would just be how performance would be impacted and the actual use case. If it was a ton of reads and writes with small files vs just mainly reads, that would affect the scale as well.

Disclaimer: I don't have 200tb to validate with. :D

 

[djdwosk97]

True there most certainly would be a point of diminishing returns somewhere. Keep in mind that the recommendation is a general one. To be honest I don't think that FreeNas would not work if you had 16gb and 200tb of storage. It would just be how performance would be impacted and the actual use case. If it was a ton of reads and writes with small files vs just mainly reads, that would affect the scale as well.

Disclaimer: I don't have 200tb to validate with. :D

Ugh, I really wish I knew what the answer was.....

I'd like to go Haswell and save the $150; on the other hand, I'd like to go Haswell-E and get the extra cores as well as support for 10 drives without the need for a sata controller card; but on the other other hand, I'd be going with a less tested configuration (x99) than if I went skylake.

 

[Mlovelace]

The E5-v3 xeons are in production with freeNAS, most people don't use x99 because the desktop class MB. Look at the 2011-3 server boards from supermicro and asrock if you want to use hardware that is currently in use.

 

[djdwosk97]

The E5-v3 xeons are in production with freeNAS, most people don't use x99 because the desktop class MB. Look at the 2011-3 server boards from supermicro and asrock if you want to use hardware that is currently in use.

If I were to use a supermicro board then it would end up costing quite a bit more than the Skylake i3. The only reason the Skylake i3 build is the same cost as a Xeon E5 build is because the X99 board is only $165 due to a nice rebate.

Is X99 able to fully take advantage of all of ECC's features? I know there are some Q87/H87 consumer boards that support ECC memory but they don't take advantage of any of its features even if the CPU supports ECC memory (i.e. it allows you to use ECC memory, but it's basically just expensive non-ECC memory).

 

[Mlovelace]

I can tell you that another person on the forum has a 10x8TB pool with 32GB of ram has made comments that he is probably short on memmory for the system .

I tend to opt for the option that allows for future growth, which the x99 build will allow for. That would be my pick, but its not my money ;)

 

[djdwosk97]

I can tell you that another person on the forum has a 10x8TB pool with 32GB of ram has made comments that he is probably short on memmory for the system .

I tend to opt for the option that allows for future growth, which the x99 build will allow for. That would be my pick, but its not my money ;)

Is X99 able to fully take advantage of all of ECC's features? I know there are some Q87/H87 consumer boards that support ECC memory but they don't take advantage of any of its features even if the CPU supports ECC memory (i.e. it allows you to use ECC memory, but it's basically just expensive non-ECC memory).

I think that's honestly my main concern with X99 right now.

Here's the motherboard I was planning on: http://www.newegg.com/Product/Product.aspx?Item=N82E16813157544
and here's the RAM: http://www.amazon.com/gp/product/B00KK8XNLA/?tag=ozlp-20

 

[TheKiwi]

I can tell you that another person on the forum has a 10x8TB pool with 32GB of ram has made comments that he is probably short on memmory for the system .

I tend to opt for the option that allows for future growth, which the x99 build will allow for. That would be my pick, but its not my money ;)

Yeah, 80tb raw might be a bit too much for a 32gb system to handle well.

 

[Jailer]

Is X99 able to fully take advantage of all of ECC's features?

I think that's honestly my main concern with X99 right now.

And unless you can get AsRock to verify that in writing I'd stick with a 2011 board from Supermicro.

The question you should be asking yourself is, "how valuable is my data and what hardware am I going to trust it with?"

 

[TheKiwi]

Get a server board, don't buy X99.

 

[djdwosk97]

And unless you can get AsRock to verify that in writing I'd stick with a 2011 board from Supermicro.

The question you should be asking yourself is, "how valuable is my data and what hardware am I going to trust it with?"

An X99 board isn't exactly likely to put my data at risk assuming it supports ECC memory. Whether or not there's some weird issue with FreeNAS is something else entirely, although X99 has been around long enough and has probably been used in enough FreeNAS builds to know whether or not it would have any incompatibilities with FreeNAS 9.3.x

 

[Mlovelace]

One more thing to throw gas on the fire of indecision for you, the E5-2603v3 is a very underwhelming cpu with it 1.6Ghz clock. You are likely to find yourself cpu bound for file transfers and other tasks.

 

[djdwosk97]

One more thing to throw gas on the fire of indecision for you, the E5-2603v3 is a very underwhelming cpu with it 1.6Ghz clock. You are likely to find yourself cpu bound for file transfers and other tasks.

file transfers aren't a big deal, and from what I've read plex is able to take advantage of multiple cores and so the extra cores should outweigh the slower clock speed.

Although I'm still completely undecided as to which route to take :/

 

[Jailer]

An X99 board isn't exactly likely to put my data at risk assuming it supports ECC memory.

And therein lies the problem.