Need your advices & recommendations for a FreeNAS build (updated: 2017/01/23)

[DeaDSouL]

Hi,

I'm about to build a FreeNAS box that will serve the following:

• Backup
  • Computers & laptops (6 devices)
  • iOSes devices via iTunes on a virtual windows box (9 devices)
  • Timemachine (AFP) for OSX devices (2 devices)
• PLEX media server: Storing & streaming 1080p media
• Games library ( for 3 PCs) [will be on the SSDs 5x500GB in RaidZ1]
• ownCloud: for all family members
• File server: (CIFS & NFS)
• Transmission

The parts I've chosen so far, are: (last update: 2017-01-23)

Motherboard: Supermicro X10SRH-CF ATX LGA2011-3 Motherboard
CPU: Intel Xeon E5-1650 V4 3.6GHz 6-Core Processor
Boot: 2 x 32GB SanDisk Ultra Fit USB 3.0 Flash Drive (SDCZ43-032G-G46)
Storage: 7 x 10TB WD Gold in RaidZ3 & 5 x Samsung 850 EVO-Series 500GB 2.5" Solid State Drive in RaidZ1
in future will expand the Z3 pool and adding the second vdev (7x10TB)
Chassis: Supermicro 3U SC836BE1C-R1K03B
NIC: Chelsio T520-BT High Performance, Dual Port 10GbE Unified Wire Adapter
CPU Cooler: Supermicro SNK-P0048AP4 heatsink
RAM: Crucial 64GB Kit (2 x 32GB) ECC DDR4-2400 RDIMM (CT2K32G4RFD424A)
UPS: Cyberpower PR1500ELCDRTXL2U 1500 VA / 1125 W

That chassis supports E-ATX & ATX motherboards.. I'm not sure if Micro-ATX is supported too. That's why I've emailed supermicro support team, and I'm waiting for their reply. In case it wasn't supported I guess I'll go with Norco RPC-4224 4U with a good redundant power supply. Although the 24-bays is much better than the 16-bays in terms of future-proof. Or do you suggest something else?

This build might look like an overkill?
Well, yes it might. But I might use many plugins and I don't want to find myself unable to. Besides I'll enable the compression. as well, as this built should be future-proof 5~7 years at least.

Removed: (So, what do you recommend for CPU Cooler, ECC RAM, UPS and HBA?)
So, what do you recommend for UPS ?



Updates:

[2017-01-11]:

1. The Supermicro technical support team, just confirmed that the X10SRM-TF will fit into SC836BE1C-R1K03B.
2. They said that I'll need a RAID control or HBA to connect all 16 hard drives. And as far as I know that ZFS wants the full control and direct access to HDD in order to be able to do its thing. So, I think my only option is the HBA... Any good HBA recommendation please?
3. A nice friendly helpful guy called "Cpuroast" in the #FreeNAS channel on FreeNode. suggested X10SRH-CF instead of X10SRM-TF in order to avoid getting an HBA since that board already has the needed 3008 in order to connect those 16bays in SC836BE1C-R1K03B.
4. Since X10SRH-CF has a dual 1Gb instead of 10Gb ports, I've decided to go with the Chelsio T520-BT.

[2017-01-13]:

1. Choosing Crucial 64GB Kit (2 x 32GB) ECC DDR4-2133 RDIMM (CT2K32G4RFD4213) based on the #FreeNAS channel's users recommendation. Again, specially "Cpuroast".

[2017-01-17]:

1. I noticed E5-1650 is the most recommended CPU when it comes to NAS.. So, I've chosen E5-1650 V4 instead of E5-2620 V4.
2. Since E5-1650 V4 supports 2400MHz ram, I've chosen Crucial 64GB Kit (2 x 32GB) ECC DDR4-2400 RDIMM (CT2K32G4RFD424A) instead of Crucial 64GB Kit (2 x 32GB) ECC DDR4-2133 RDIMM (CT2K32G4RFD4213).
3. I've been advised to use 7 HDD in Z3 instead of 6 HDD in Z2. Because of the huge stress time the drives will be in when I'm rebuilding the array if one drive fails because of the huge size of 10TB, which may causing another drive to fail in that period, and if that happens I can't afford losing another drive or I'll be in a serious trouble. So, I'll be using 7 drives in RaidZ3.

[2017-01-18]:

1. Choosing APC Smart-UPS 1500VA LCD RM 2U 120V (SMT1500RM2U) as the UPS.

[2017-01-19]:

1. I totally forgot that the standard voltage in my country (Kuwait) is 240V and the standard frequency is 50Hz. Which means the APC Smart-UPS 1500VA LCD RM 2U 120V (SMT1500RM2U) will not work since it's 120V/60Hz.
2. I was talking to Supermicro support team, and they recommended SNK-P0048AP4 as the CPU-Cooler for this setup.

[2017-01-23]:

1. Choosing Cyberpower PR1500ELCDRTXL2U (1500 VA / 1125 W) as the UPS.

any suggestions or advices would be greatly appreciated

Thanks

 

[Quebecman]

Wow, that's a big project. Do you mean to run games directly from the NAS? I assume it would be wiser to store games backups and installation files on it rather than run games from it, especially in the case of recent AAA titles, in which case SSDs would be a moot point. Your bottleneck could be your network, so you might want to mention the router you plan to use, or would that be (plural) routers.

 

[DeaDSouL]

Hi,

Wow, that's a big project. Do you mean to run games directly from the NAS?

Yes, this is the idea. Storing all installed games on FreeNAS (SSD) and running them as if they were installed locally.

I assume it would be wiser to store games backups and installation files on it rather than run games from it, especially in the case of recent AAA titles, in which case SSDs would be a moot point.

Of course all none-steam installation files will be backed-up on the Z2 pool.

Your bottleneck could be your network, so you might want to mention the router you plan to use, or would that be (plural) routers.

The pfSense will be the router, and as for the switches, I'm going to use two of them
First one is the 10Gb switch: NETGEAR ProSAFE 8-Port 10-Gigabit Ethernet Smart Managed Switch (XS708T-100NES)
Second switch is the 1Gb switch: NETGEAR ProSAFE S3300-28X 28 Port Gigabit Stackable Smart Managed Switch with 2 x 10GBASE-T and 2 x10G SFP+ Ports (GS728TX-100NES)

According to calomel.org,

5x 256GB raid5, raidz1 931 gigabytes ( w= 817MB/s , rw=610MB/s , r=1881MB/s )

Which I believe it's going to be fine for 1 ~ 3 PCs to read the games from those SSDs.
And yes I know those numbers were generated on a tuned optimized FreeBSD 10.2 which is not the one the current FreeNAS is based on. But the numbers should be in that range, specially that it's a pure SSD without any compression.


Any suggestions to complete this build are very welcome

Thanks

 

[Quebecman]

I can't imagine 3 PCs accessing the exact same game files remotely at the same time in a same install folder, that's bound to cause problems. You would need three different install folders per game for it to work in which case you would be better off installing on each machine locally.

In any case, gaming evolves fairly quickly and you are lucky when the hardware lasts 5 years. SSD technology is also evolving fast and costs keep coming down.

I don't think you gain anything significant for gaming, but rather are putting a lot of pressure on the server if say three machines game at the same time while another machine plays a movie.

My advice is that you could however pursue this route experimentally with one SSD at first. Unless this is alredy documented... But it's adding a layer or two I say (server+router+switch), and gaming is best executed as efficiently as possible.

And then there is 4k to consider. I can't imagine 4k gaming going well with what you are proposing. Then again the graphic card handles that.

Have a read at the sticky post "Will it FreeNas", there's a pdf that is not long to read for hardware recommendations. Perhaps you did already.

So yeah, gaming as you plan it probably raises your requirements and technical difficulties by a significant margin I think, and that money is potentially better spent on the gaming rigs rather than the NAS.

 

[MatthewSteinhoff]

You'll get better performance by putting those SSDs in the individual PCs and playing your games from there. Less latency for sure.

That said, I'm not sure if it will really matter. You have a beast of a machine and can probably handle whatever you throw its way.

RAM... 32GB ECC to start. If that doesn't meet your performance requirements take it to 64GB. Not sure you'll see much benefit above that point.

CPU Cooler... Whatever Supermicro recommends.

UPS... do the math and give yourself room to add drives later.

Cheers,
Matt

 

[DeaDSouL]

Hi @Quebecman,

I can't imagine 3 PCs accessing the exact same game files remotely at the same time in a same install folder, that's bound to cause problems. You would need three different install folders per game for it to work in which case you would be better off installing on each machine locally.

right now on a very bad 1Gb network (with 2~4MB transfer rate), I've tried playing the same game from 2 PCs from one shared steam library folder on the network using NFS protocol the only problem I had was the huge loading time because of the bad transfer rate i'm having, other than that there was no issues whatsoever except of configuring the Steam settings on those PCs to point to that shared folder as the steam library.. I won't even call that an issue.
I'm pretty sure the 10Gb switch will significantly reduce that loading time, even if it's going to be similar to the time is needed to load a game from a normal mechanical drive, I would call that a success.
in worst scenarios, if it didn't turn the way I was wishing for, I'm sure I still can find a good use for those SSDs at least using them as a local disks for gaming on those PCs, as you suggested.
does it worth the shot? I see no reasons why I shouldn't try it as it might actually really work.

I don't think you gain anything significant for gaming, but rather are putting a lot of pressure on the server if say three machines game at the same time while another machine plays a movie.

The way I see it, if each game is going to need 1 ~ 2GB to be transferred from the server to each client in order to load that game level/scene.. it means about 6GB even 10GB in total of a transfer for those 3PCs which sounds like a normal copying file from the server to the client, so, I don't think it's really that bad.
As for streaming the movie.. yes it will use some of the resources.. but isn't that why I'm building this box in the first place?

My advice is that you could however pursue this route experimentally with one SSD at first. Unless this is alredy documented... But it's adding a layer or two I say (server+router+switch), and gaming is best executed as efficiently as possible.

I will test and compare the results and see whether or not it's really worth it.

And then there is 4k to consider. I can't imagine 4k gaming going well with what you are proposing. Then again the graphic card handles that.

Why? Please correct me if I'm missing something, but as far as I know, the graphics card and CPU are the ones behind rendering and processing the game scenes and graphics.. so, if it was a 4K or not, the server should not really care, as long as all its job is transferring the game files to the client who will take care of rendering the game. maybe the server will need to transfer more files (in size) which of course will increase the loading time, like a few extra seconds.. but again, all the 4k thing should be done and taken care of from the client side.. I guess this is the way it works.

Have a read at the sticky post "Will it FreeNas", there's a pdf that is not long to read for hardware recommendations. Perhaps you did already.

Yes, in fact I'm spending most of my time these days reading everything useful I could find on this forum, just to avoid any regret situations :p

So yeah, gaming as you plan it probably raises your requirements and technical difficulties by a significant margin I think

Indeed it is :D
Thanks for your points and advices


Hi @MatthewSteinhoff,

You'll get better performance by putting those SSDs in the individual PCs and playing your games from there. Less latency for sure.

Indeed it will be much better using them natively.. but again, it's an experiment and might just works :)

That said, I'm not sure if it will really matter. You have a beast of a machine and can probably handle whatever you throw its way.

Thanks, hopefully it remains that way for the next few years

RAM... 32GB ECC to start. If that doesn't meet your performance requirements take it to 64GB. Not sure you'll see much benefit above that point.

Any advices on which memory type I should be using? RDIMM or LRDIMM, or it doesn't really matter.

Thanks

 

[Quebecman]

You have this well in hand. It sounds like a very exciting project.

So no confusion for save files and loading gamer profiles then for the games. That was also one of my concerns.

I imagine 4k streaming is becoming more and more common, both incoming for watching videos, and outgoing if you, say, stream your gaming sessions in 4k. It's more that which I had in mind than the computing in the end. But I wonder if it will also impact game files in the future, as games keep getting bigger and bigger in GB numbers. 4k might demand more and more detail I suppose. Just a theory.

Let's see a quote from the hardware recommendations:
"Registered DIMMs alleviate load on the memory bus by using registers to drive the address lines of the DRAM chips. LR-DIMMs extend this to the data lines as well. These techniques allow for greater capacities, at the expense of some added latency."

I'm no help here other than that. I imagine you are aiming to go for 64GB +.

Oh wait! The specs of your selected motherboard above says:

• Up to 1TB ECC 3DS LRDIMM
• Up to 512GB ECC LRDIMM
• Up to 256GB ECC RDIMM

 

[MatthewSteinhoff]

Go with RDIMM unless you expect to need more than 256GB.

Cheers,
Matt

 

[DeaDSouL]

I imagine 4k streaming is becoming more and more common, both incoming for watching videos, and outgoing if you, say, stream your gaming sessions in 4k. It's more that which I had in mind than the computing in the end. But I wonder if it will also impact game files in the future, as games keep getting bigger and bigger in GB numbers. 4k might demand more and more detail I suppose. Just a theory.

I agree. streaming a 4K is a nice feature that i would like to have specially from this box i'm building. but if it wasn't capable of 4k streaming specially in future, I still can solve that by letting any strong PC runs the plex server to stream those 4k movies stored in the nas box. although I don't see this happening in near future. but I'm only saying that there is a solution for it. as for the games. oh man, yes they're evolving faster than how they used to..

I'm no help here other than that. I imagine you are aiming to go for 64GB +.

that is what I was thinking to start with :D

Go with RDIMM unless you expect to need more than 256GB.

that's what I ended up choosing :D


Thanks guys

 

[rickygm]

Tell us how was your project, when you finished it.

regards

 

[DeaDSouL]

Tell us how was your project, when you finished it.

regards

Of course :)

 

[DeaDSouL]

Sorry, I was busy the last few days..

regarding to:

UPS... do the math and give yourself room to add drives later.

this my approximately watts needing, although I'm not that sure of the calculations I made. Hopefully someone correct them for me or tell me how to do it the right way..


35 : 5 (7000rpm fans) x 7 watts
320 : 16 (hdd 10tb wd gold) x 20 watts (I'll use only 14 of them, but just in case I may need to use them all)
50 : 5 (ssd 500gb 850 evo samsung) x 10 watts
145 : 1 CPU E5-1650 V4
80 : X10SRH-CF (not sure, but the highest used watts in mobo is ~80)
24 : 8 ram slots x 3 watts (not sure) (i've calculated all slots. although I'll use only 2 of them in the present)
200 : extra watts, just to be in the safe side
---- ----------------------
854 : total needed watts (max. load)


854 wattage x 1.6667 = 1423.361 VA

So, I think I need some UPS with ~1500VA

is it correct?

Thanks

 

[MatthewSteinhoff]

I think I need some UPS with ~1500VA

You wattage looks a bit high. In any case, yes, there isn't much of a cost savings going 1,000VA so you might as well go up a size and give yourself plenty of room for expansion.

Cheers,
Matt

 

[Ericloewe]

16 (hdd 10tb wd gold) x 20 watts

More like 30W each at spinup.

854 wattage x 1.6667 = 1423.361 VA

Yeah, you're never going to see such atrocious power factors on a halfway modern PSU. Just make sure the model you pick can handle the real power that you need - marketing several hundred VA more than the actual real power output of the thing is a rather shady practice in the days of .99+ power factors.

 

[DeaDSouL]

You wattage looks a bit high. In any case, yes, there isn't much of a cost savings going 1,000VA so you might as well go up a size and give yourself plenty of room for expansion.

right, it's better than finding my expanding options are very limited in future ... thanks :)

More like 30W each at spinup.

thanks for correcting my knowledge about HDDs..
is the 30 wattages because of the RPMs or the storage's size?

Yeah, you're never going to see such atrocious power factors on a halfway modern PSU. Just make sure the model you pick can handle the real power that you need - marketing several hundred VA more than the actual real power output of the thing is a rather shady practice in the days of .99+ power factors.

I was looking at APC Smart-UPS 1500VA LCD RM 2U 120V (SMT1500RM2U) it's about $567.95 on bhphotovideo.com
What do you think about it? is it a good option? fully supported in FreeNAS?

Thanks

 

[Ericloewe]

is the 30 wattages because of the RPMs or the storage's size?

Neither. It just matches the observations for WD Reds. You might ask if your drives won't need more power since they're 7200RPM models. Possibly, but it has worked for everyone so far and we only have limited data.

2U 120V (SMT1500RM2U) it's about $567.95 on bhphotovideo.com
What do you think about it? is it a good option? fully supported in FreeNAS?

Sure, but keep in mind that its real power output is "only" 1kW.

 

[DeaDSouL]

Sure, but keep in mind that its real power output is "only" 1kW.

1kW is not bad.. right? I don't think I'm going to need more than 1kW even in full-load

 

[Ericloewe]

1kW is not bad.. right? I don't think I'm going to need more than 1kW even in full-load

Yeah, but note that the critical part is spinup, not full load (for most servers).

 

[MatthewSteinhoff]

I was looking at APC Smart-UPS 1500VA LCD RM 2U 120V (SMT1500RM2U) it's about $567.95 on bhphotovideo.com

It'll work but you can get a larger unit for $200 less. I like APC but I like saving $200 even more. (We've had good luck with CyberPower units in wiring closets at smaller loads. Our main UPSes are still APC.)

Cheers,
Matt

 

[tvsjr]

Another happy Cyberpower user here. After having to sledgehammer swollen batteries out of some SmartUPSs, and having a few others halt-and-catch-fire, I'm not really big on that product line. My one issue with Cyberpower (out of warranty) was happily and quickly corrected with a brand new unit, at no cost to me.