24x4TB drive FreeNAS build - not cheap!

[timb_yyc]

Hello,

I just finished this new build, thanks to all the information I gathered from these forums and the Internet, and thought I would introduce myself and the project.

My plan is to replace the existing servers. Currently I have 1 Windows 2003 server (2x2TB mirrored), 2 FreeNAS7 boxes (4x2TB, 1 RAID5, 1 ZFS1) and 1 Netgear ReadyNAS Ultra Plus 6 drive RAID6 (6x3TB) - All of these will be replaced by this new 24 drive server to store business data, personal files and media (music & movies)

24x4TB Server Configuration
Seagate 4TB Hard Drives x25 (one cold spare)
Norco RPC-4224 24 Bay Case
Norco 120mm fan bracket
Norco RL-26 Sliding Rail Kit
IBM M1015 LSI SAS Controller x1
INTEL RES2SV420 SAS Expander x2
120mm Case fans Delta - 113cfm
120mm fan cover/protectors
Motherboard - SuperMicro X9DR3-F-O
CPU-Xeon E5-2609 v2 2.5GHz x2
RAM - KVR16R11D4/8 x8 (64GB ECC)
Power Supply -750watt eVGA Gold
APC Back-UPS Pro BR1300G (edit May 4,2014)
8GB USB boot drive + spare for FreeNAS

This is my first FreeNAS 9 build and so far I am very pleased with the results!

Pic1

Pic2

Pic3

 

[ser_rhaegar]

Awesome build!

 

[jgreco]

Glad to hear it is working out for you. I don't like the E5-2609 too much, it's a contemptible CPU compared to its more capable brothers. On the other hand, they're not that expensive so two of them might do the trick.

 

[timb_yyc]

Thank you for the input jgreco. Is there anything specific that I should be be watching out for with the existing CPU? I do see that PassMark doesn't exactly give it good marks for speed (http://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2609+@+2.40GHz ) - Is there a better CPU you would recommend for this system? So far, when accessing the new server from a Windows 7 computer, once in a while if I go into a large folder there can be a noticeable pause/delay. I don't recall having this problem when accessing any of my old servers (Windows Server 2003/ Freenas 7/ ReadyNAS).

Yes, everything has been working out pretty well. I have finished moving all existing data over to the new server and all was well until I applied the last Freenas update (9.2.1.4.1) and it blew up all my permissions. It's time to roll up the sleeves and learn more about cifs/share/permissions.... I didn't need this right now!

 

[jgreco]

No, really, if it's making you happy, then it is simply doing more for you than it did for me. The E5 stuff is all mostly ridiculously expensive to get similar performance to E3; an E3 at 3.4GHz is, what, $250, while an E5 at 3.5GHz is probably around $1000 (E5-2637 v2). The thousand dollar part, or better a pair of them, will make for the best possible system, but if your existing part is fast enough - what's the benefit? None.

As for large directories, is the pool simply getting pegged by I/O when that happens? Keep an eye on it. More RAM or the addition of an L2ARC are the likely fixes, and probably not a faster CPU.

My unhappiness probably comes from the fact that I've been working heavily with E3-1230's (threaded, faster) and the E5-2609 was a sucktacular sideways move while I waited for the E5 Ivy's to be released. Running ESXi there was a lot of contention and unhappiness, in part because the CPU was simply not up to the task, even if it wasn't doing much.

 

[timb_yyc]

Yes, according to the Freenas reporting, the CPU isn't really doing much. I do find that the "Physical Memory Utilization" chart is difficult to understand (I'll try to post it here) but if more RAM or an L2ARC will help then I'd be open to giving it a try. Will see if I can track performance when the actual problem happens. Thanx again for your input!

 

[trionic]

Intel do not list E3 Xeon processors for a socket 2011 mobo.
http://ark.intel.com/search/advanced?s=t&SocketsSupported=FCLGA2011

Intel do not list E3 processors with support for more than 32GB RAM.
http://ark.intel.com/search/advanced?s=t&FamilyText=Intel® Xeon® Processor E3 v3 Family&MaxMemMBMin=65536

So if you want >32GB RAM on your FreeNAS machine then you cannot use an E3 CPU. Given the 1GB RAM per 1TB disk rule then an E3 is out of the picture for any FreeNAS build with more than 32TB of disk (eg: 10 x 3TB or 8 x 4TB).

Right?

 

[jgreco]

Um, well, everyone else in this thread pretty much knows the tech and has built systems with it.

 

[trionic]

I don't see anyone in this thread with >32GB builds on an E3 nor E3 S2011 builds.

Just trying to get the facts straight for my own understanding.

 

[cyberjock]

Yes and no. Strictly speaking yes.

That thumbrule is just a thumbrule. It is not a guarantee of any kind of performance, just an estimate of where to go. You might be able to use 64TB of storage with 32GB of RAM. On the other hand, there's lots of documented cases where someone had 10TB of data but needed 128GB of RAM because less RAM was too slow. Likewise if you decide to enable dedup, your RAM needs will explode out of control *extremely* rapidly.

The thumbrule should be seriously considered when building servers and until you have built lots of servers and have experience with how much RAM you "think" you might need there's a high risk you might not have enough RAM if you go too low. It's meant to give you an idea of how much RAM you might need for your server. Sometimes you need less, sometimes you need more, sometimes you need many many times more.

The E3 CPU and accompanied chipset are limited to 32GB of RAM. If you want more than 32GB of RAM, you are instantly talking socket 2011, which also means a more expensive motherboard and *much* more expensive CPU for the same processing power as the E3. The E3 is really sold as a "small to medium server" while the socket 2011s are billed as "large and very large servers"

 

[timb_yyc]

Oops. sorry about that, I thought Trionic was referencing my CPU but now I see he's talking about the E3 - I'll just go back to playing in the snow... yea, we're still dealing with snow in Calgary in May! I have got to move to someplace warmer!!

 

[trionic]

The E3 CPU and accompanied chipset are limited to 32GB of RAM. If you want more than 32GB of RAM, you are instantly talking socket 2011, which also means a more expensive motherboard and *much* more expensive CPU for the same processing power as the E3.

Thanks. That was the understanding I'd built-up from creating a spec for my first FreeNAS build :)

Digressing slightly, it seems strange that Supermicro do not list 32GB DIMMS (other than seemingly unavailable LRDIMMS) on the S2011 X9SRH-7F's HCL despite the spec detailing a max memory capacity of 256GB when using DDR3 ECC RDIMMs.

 

[jgreco]

I don't see anyone in this thread with >32GB builds on an E3 nor E3 S2011 builds.

Just trying to get the facts straight for my own understanding.

You quoted the relevant Intel pages. E3 is intended for smaller servers and is correspondingly featured and priced. I see no one in this thread confused about the tech except you...? Nobody here has suggested >32GB E3's or E3 2011's.

 

[trionic]

I see no one in this thread confused about the tech except you...? Nobody here has suggested >32GB E3's or E3 2011's.

Your previous answer...

Um, well, everyone else in this thread pretty much knows the tech and has built systems with it.

...was ambiguous and I interpreted it as:

Um, well, everyone else in this thread pretty much knows [E3 CPUs and S2011 mobos] and has built systems with it.

(My emphasis)

Glad that's cleared up.

OP: Nice tidy build. What drive temperatures and power consumption are you measuring? Did you consider redundant power supplies or a UPS?

 

[timb_yyc]

Your previous answer...
OP: Nice tidy build. What drive temperatures and power consumption are you measuring? Did you consider redundant power supplies or a UPS?

Thx. I replaced the original (4x80mm) cooling fan partition in the RPC-2424 case with the optional 3x120mm unit, then added high capacity Delta 113cfm 120mm cooling fans. The hard drives are currently at/or around 26 C - At the moment the server is in my garage with a temp of 17C. I will be moving it to the basement server room soon, the temperature there should be a little warmer. The power consumption reads 240watts with the server idle.

No need for a redundant power supply as I always have something in stock or I can replace with a similar model in a couple hours from a local supplier. I have listed a UPS (in my signature line), it's the APC Back-UPS Pro BR1300 and it works well with FreeNAS9.

 

[Fox]

I also have the E5-2609.. I bought it for the reasons mentioned in this thread, expandability past the 32GB limit. As hard drives get bigger and bigger (and prices fall) I can see potentially going past 32TB in a few years, and I want to be sure that I can expand it without having to replace everything.. I ran the numbers, and the increase in cost was negligible compared to everything else. The 24 drive supermicro case was the biggest expense.

About the only thing I don't like is the power consumption, although I don't think it is the processor or MB. I have the redundant 1200watt redundant power supplies (came with the case), and with all the fans it's really sucking power. According to the power supply data, it is taking in about 190watts right now (95watt each power supply). Heck, when it's shut down, it uses 28 watts! I think I am going to start pulling some things I don't need out. Though, I'm lucky, I'm in one of the top three US states for cheap power, so I really can't complain too much.. I like what I have..

 

[timb_yyc]

Before starting this project I built a spreadsheet and calculated that the 4 existing old servers, combined, consume 327watts. I've gone from 19.6TB to 57TB with better redundancy (RAIDZ2) and using less power. Also, should the need arise I can double my HD capacity, adding another box/power supply and another 2x Intel SAS Expanders + misc cables...oops, and more ECC RAM!

 

[Hyperion]

It looks like a cool build.
But why would you want so much storage?
My first 1TB is porn, then the rest is movie ISOs.
That doesn’t even come close to my 11TB.

 

[timb_yyc]

It looks like a cool build.
But why would you want so much storage?
My first 1TB is porn, then the rest is movie ISOs.
That doesn’t even come close to my 11TB.

It's funny that you ask but I have several requirements. I remember buying my first 10MB (yes megabyte) hard disk drive for my Radio Shack Color Computer and saying "I'd never be able to fill that".... Data will always expand to fill the existing hard drive!

Currently backing up Blu-Ray movies (for the 120" theater - seen here: http://www.blu-ray.com/community/gallery.php?member=tim_yyc&folderid=2442 ) but in the near future we'll be looking at 4k media / also I use the server for client backup images of systems using Acronis and I also do some professional photography, shooting raw files and I never seem to delete anything so it all takes up space. My previous NAS was 6x3TB using RAID6 and it filled up in close to 2 years so I decided that instead of buying over priced, under-powered NAS boxes I would take the bull by the horns and build something that will manage all of my data and make it future proof (expandable) as well. Also, the total power consumption is less for this box versus the combined power needed to operate the previous 1xW2k3/3xNAS boxes.

So far, I am very happy with the build. I don't think that FreeNAS9 is as nice/easy to use as FreeNAS7 but that's something I can work on learning.

Tim

 

[Hyperion]

Ok so, some things take up 4K, that shit takes up a shit load of space.
But how many times can u watch “Sharknado” in a life time?