First FreeNAS Build - 6x 6TB RAIDZ2 on ASRock C2750DI

[Darren Myers]

This, this build needs to be somewhere for everyone who makes a FreeNAS server to see

 

[TrevorX]

I am humbled. Thank you, Darren, that's very kind of you to say

 

[TrevorX]

This is more or less normal. To reduce the vibrations you can add weight to the case and/or add silent-blocks between the case and the shelf and/or use some vibration isolation system between the drives and the case (but be careful because that can lead to less vibration dampening on the drives and reduce their lifetime) ;)

I have some sorbothane feet on their way, hopefully that will prove effective ;-)

 

[Couto]

I'm a total noob, but your build is quite interesting, and I'm using it as a starting point for my own build.
I have however one question (please remember that I'm a totaly newbie with computer parts and so on...).
Since I don't have a choice but to have my computers exposed on my living room, aesthetics are a factor of importance for me, and I know that tastes are relative, but...
What would I have to change by using the BitFenix Phenon Mini-ITX case instead of the Lian-Li PC-Q35B? The main difference that I can find is that it doesn't support 5.25" drive bays so I guess that I won't be able to use the hot-swap drives, am I right? (I'm probably wrong, but nevertheless I would like to hear your opinion)

Thanks :)

 

[TrevorX]

Hi Couto,

The Phenom can support up to 5 3.5" drives natively, plus one with an adapter (included). So you could fit that many drives in there... A couple of 2.5" drives should likewise be no issue. But yes, you completely lose all ability to hotswap - drive replacements are manual, fiddly and time consuming. As long as you're ok with that then great

What drives are you considering using? I'd install a second front-mounted 120mm fan above the one included to ensure adequate cooling of the upper disks. I have a feeling that the top-most disk sits in the cage at the top of the case, too, which will get precious little airflow in the default configuration. You can install two 120mm fans in the top (not included) so I would definitely do that. I'd even replace the rear 120mm with a 140mm - more airflow for less noise. Unfortunately you can't use a fan controller with this puppy, so you're either limited to locked speed or get yourself PWM fans that can be controlled by the mainboard properly, otherwise six 120mm fans running full tilt will get pretty annoying.

I also don't know if you're personally familiar with those cases, but they are big. Way bigger than you expect for mini-ITX - they're actually larger than many micro-ATX cases due to their width. If you're ok with that then fine - it's just something to be aware of ;)

There is one thing I will say as a caveat here - if you're a total newb as you state, this may not be the best idea in the world. Most off-the-shelf NAS appliances will do all sorts of things with a few clicks that you will need to spend days to weeks learning how to do with FreeNAS (or risk losing all your data). Now if you are doing this with the intention of using it as a sandpit to learn in, then by all means, go right ahead! But if you primarily need a reliable storage solution and you don't have a few months to get to grips with it before you 'deploy it to production' (ie start trusting it with your actual data) then right now you should start with something with a much shallower learning curve, and maybe build a FreeNAS development box with some old parts to learn on for a while, with an eye to building a proper system once you've fully tested your intended configuration and you feel very confident with the environment. I'm not trying to dissuade you here - we all need to learn somewhere, and FreeNAS is a hell of a lot easier than a console-only box! I'm just going by your admission that you lack extensive experience, because there's a big difference between 'playing around learning FreeNAS' and trusting your important, irreplaceable data to a device you can't competently administer.

I hope I've been of some assistance - good luck! :)

 

[Couto]

I'm a programmer so, fiddling around for months until I reach the perfect point for production is not a problem, however I've never messed with hardware before and that's my major obstacle here.

I'm literally using your set as a starting point, so at the moment I'm using the same disks as you ;)
Personally I don't find the ability to hot-swap disks something terribly important, since I don't expect to move them around that much, and I can afford to shutdown the machine for an hour or so in case of need.

I'm just starting around with all this stuff, since me and my wife have started to outgrown our owncloud instance on a VPS, so we both decided that it might be cheaper/more interesting to store the data in house, however I'm currently in exploratory mode, and checking prices/parts/consumptions...

Thanks for the help, and the tip about the fans, I didn't notice that lack of controller and that's an interesting point!
The box size itself might be a problem, but I know of a local store selling those cases, so I might jump in there to take a look at the size in real life! :)

 

[TrevorX]

Cool :) Yes, the fastest way to learn is to jump in with both feet - I just didn't want to go encouraging someone who admitted being 'new to hardware' to get stuck in when there's a real possibility that they could make one simple error and find their data wiped out.

Hardware is easy, but I suppose it depends on your background - thinking about it, there's a hell of a lot you'd need to know about architectures, integrated circuits and physics to have a good appreciation for the subject. I usually think of it like Lego bricks because of the simple and interchangeable nature of how things fit together, but knowing 'why' is far more useful and far more complicated than 'what'. I recommend reading articles on sites like Anandtech, TweakTown, TechReport... god the list is pretty massive - an aggregator like the splash page of Overclockers Australia is useful as it lists summaries of to-the-minute hardware reviews from all over the web a few times a week. The new Intel Xeon-D (Broadwell-DE) has just been launched, and as various parts trickle out there will be a tide of reviews that not only cover the new architecture, but will link to backgrounders on many of the major architectural components. It will take a fair bit of effort and time to get through that and come out the other side understanding it all (as you'll have to do lots of supplemental reading) but believe me it will be the best way to break the ice. You will be a far better programmer knowing how the underlying hardware works, too (even if what you're writing is so abstracted you don't have to actually deal with hardware hooks, memory management, etc).

Yes, if there's a PC shop that stocks various cases I'd recommend checking that out - only you know your tastes and the physical space your gear will be housed in. I have a new UPS on order that I'm either going to use myself or ship off to a client site, depending on noise level. Most rack UPS's have stupidly loud (but very high CFM) fans that are impractical in a home, but I have no 'showroom' to visit that stocks gear like that (you purchase it based on specs, not aesthetics, after all). But I get exposed to a heck of a lot of hardware in my job so I get to play with lots of stuff and get to know it without having to wear the financial commitment ;)

Hey if you follow my hardware specs you're not going to go far wrong, assuming your use will be similar - I wouldn't build something like this for a DB server or VMDK/VHDX image host! Horses for courses - for a home file server, this is solid gear.

Best of luck! :D

 

[philious77]

This, this build needs to be somewhere for everyone who makes a FreeNAS server to see

Darren,

What GBICs are you running in the Chelsio T420-CR?

Trevor,

Can you confirm physical space in the Lian Li case for a 10GB NIC when the power supply is installed? Thanks

 

[Darren Myers]

I got picked up the recommended ones from Cheliso http://www.ebay.com/itm/391075659499?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT i got them from that seller who accepted my best offer of 25$ each (i got 3) , very quick shipping and the SFP+'s work flawlessly that i can tell (its been a 2-4 months now) with no issues.

However i can't speak on if the card will fit, but it should have good airflow the chipset gets quiet warm

 

[TrevorX]

Trevor,

Can you confirm physical space in the Lian Li case for a 10GB NIC when the power supply is installed? Thanks

That case supports full size PCI-E expansion cards, so I expect there will be sufficient room. The PSU actually sits above the card location - see this rear photo: https://picasaweb.google.com/m/view...25031/5983116412587702209/5983116431438106290

 

[cynod]

Just popping a quick note in here from another forum n00b - GREAT post. Thanks so much for posting it.

I've been a lurker for a year or so now and have followed FreeNAS since its inception. Amazing piece of software that puts many (MANY!) commercial NAS solutions to shame. I've finally come into some funds and, TBH, got fed up with the nasty state of my home mix of PC, multi laptop, multi tablet/phone, open-wrt router with crappy USB HDD shoved on to hold the odd video.. sigh. It's just horrible.. and I was once an DC ops manager for a nasdaq listed company. I should know better.. and indeed I do and hence I'm spending some cash and doing this right (finally) :)

So, thanks for this post as this ASRock is amongst the options I'm considering.

I'll be posting over the weekend to a new thread with my likely build :)

 

[TrevorX]

Thanks Cynod :) Glad you've found it helpful. As I wrote elsewhere, the new Xeon-D is a very exciting platform, but the entry cost is quite onerous - I expect Avoton to be the best option for a home FreeNAS until it gets a proper replacement in the same market segment, so these ASRock boards are likely to remain a good buy for another 12 months or so at least. In my opinion ;)

 

[blahhumbug]

The Denverton follow on to Avoton will hopefully show up next year. Should be 16 airmont cores (14nm) but I haven't found any other details published yet.

 

[TrevorX]

The Denverton follow on to Avoton will hopefully show up next year. Should be 16 airmont cores (14nm) but I haven't found any other details published yet.

Not looking like launch before Oct to Dec given how little detail there has been thus far. If improvements are anything like Xeon-D over previous generation (or Avoton was over Atom) then it should be a big step up, but we won't know until first silicon is spun and engineering samples get a viewing, which could be as late as June 2016. Hey, it's not like Intel are under real pressure from AMD or anyone else at this point, so they have the luxury of time to develop fabrication processes properly and go in with good yields from day one. That extends release cycles a bit, but we get more consistent (and cheaper) products as a result. Enough competition to keep them innovating without the pressure to release products prematurely to stave off competition - it's a model that has been working pretty (surprisingly) well ever since Intel moved away from P4 clock wars and into the first Core series.

Blah blah blah sorry, I'm being boring - Denverton should be brilliant, but no need to wait when Avoton does a pretty stellar job today ;-)

 

[TrevorX]

As you may be aware, there were two potential flaws in this design. The eSATA card (yes, you were right, Eric), and potential heat build up from the drives in the 5 disk hotswap pod.

The first initially seemed to not be an issue - I was able to connect the external drive, partition it and run some test backups without issue. Until I rebooted the NAS, whereupon the drive disappeared. Not so great. It turned out, however, that the drive hadn't disappeared, but the volume had. Weird. So I recreated the volume and managed to regain access. And run some backups successfully. And all was good, until I rebooted. And then the volume wouldn't show up. Wonderful. Further testing revealed the same issue after shutting down and restarting the drive (while the NAS was left running). Excellent.

The second issue had me a little more concerned. How hot were those drive running? FreeNAS provides no ability to determine that, out of the box. There are numerous ways to extract that, including CLI commands and scripts, but I found the simplest to be running FreeNAS Temp Report - I could see the drive temps with the click of a mouse.

Unfortunately, the drives were running hot - high 50's to mid 60°C temps, constantly. That wouldn't do - I wanted sub-50°C, if possible. I looked at potential solutions, working within my design limitations (low noise enclosure, because it sits in my home office). Disassembling the enclosure to see if I could design a new shroud to allow attaching a larger fan, I realised nothing I did would get around this:

This is the circuit board that the drives attach to within the enclosure. The fan sits on this side and sucks air through those holes. As you can see, they're not exactly large. In fact, they're extremely restrictive. On the other side of the circuit board you can see electrical traces close to those holes, so there's no scope for increasing their size.

So the only way I was going to get more air moving through that would be to move it faster, which meant more noise. Even with a larger fan somehow hacked onto it, it would be noisier.

It seemed the only way to go was to replace the pod altogether. Initially I started looking for five disk enclosures, but then I realised that removing the single drive bay above actually gave me four 5¼" bays to play with, which opened up a whole different possibility - two three-bay enclosures. Three bay enclosures use the same size fan as five-disk ones, but only cooling three drives. I'd be able to reduce drive density and increase fan-to-drive ratio. The downside was, of course, cost - looking around at available options, I settled on the Vantec M3500 - apart from easy availability, it was the only enclosure I could find that was actively promoted as being 'low noise'. Each enclosure was more than the original cost of the five disk enclosure I was using, but that doesn't matter if it isn't capable of doing the required job.

I also decided to try replacing the eSATA card with a simple pass-through backplate that allows connection of eSATA ports to standard internal SATA connectors. I'd be using the Marvell ports on the mainboard as the Intel ones were all doing duty for the ZPool drives, but it would probably be an improvement over the eSATA board.

Queue rebuild:

Unfortunately I was being a bit lazy/pressed for time, so I didn't go to the effort I could have tidying up cables:

Here we are, all re-assembled. I'm looking forward to seeing how doubling the airflow affects drive temps:

Locked and loaded. Somehow, I think it looks a lot better like this, like it has finally been done properly.

The result? Well, on the first issue, I had immediate access to the external drive and could see everything on it. Brilliant. Backups ran without a hitch. Rebooting either the NAS or the external drive worked fine, with the drive coming back online correctly. Solved!

Drive temps were likewise much improved - I've seen a 10-12°C drop in temps across all drives, with them now running from mid 40's to low 50's. That's also with the enclosures set at the lower speed setting - I haven't tried them at the higher setting yet, but I may test it out as we head towards summer.

So I'm very happy with the upgrades as they have successfully achieved my goals. The NAS now synchronises important data folders every fifteen minutes, running nightly backups of everything else (basically only excluding movie folders). I'm using RSync tasks to a localhost target, so I've even managed to avoid any custom scripting for backups. The drives are now running within a much healthier and safer temperature range, despite the whole thing being quiet enough to not be annoying or distracting (well, if you ignore the regular pulses of data access to the drives, which don't bother me, but took my wife some getting used to).

 

[TrevorX]

Sorbothane Feet

I did have a slight unforseen issue with my NAS - I have it mounted on a shelf due to space restrictions in this room, and it was causing a vibration along the wall, which is the same external wall as my bedroom. The vibration caused a noticable hum from the wall at night which kept my wife awake, so I was having to turn the NAS off at night. My solution was sorbothane feet:

These are possibly overkill - you can get thin disks of sorbothane that would probably have been fine, but I opted for 40mm diameter, 20mm thick hemispheres and they have completely eliminated the vibration. So if you find you have a device (NAS, air conditioner, pool pump, compressor, engine) that's causing an annoying vibration, you may find that completely isolating the equipment will eliminate the noise, in which case you can't go past sorbothane. Highly recommended.

 

[TrevorX]

Enclosure power connector clearance

I'd like to touch briefly on the Vantec EZ Swap M3500 series of mobile racks. Unfortunately I couldn't find any images on Vantec's website of the rear of these units, so it was very much a case of pay and hope that these would be right for the job. Fortunately, as it's turned out, they've been brilliant for my needs, but that doesn't mean to say the product is perfect. Take a look at the position they've used for the two SATA power connectors:

I've since been able to find an image of the four drive model, which actually has worse issues:

Now, given the amount of plastic dead space there is back there, I really don't think it would be too difficult for Vantec to separate those power connectors (double-stacking as they've done with the four disk model is atrocious) and give them sufficient space (2mm extra would be sufficient on my ones). Or, if they don't want to redesign the units, at least provide a pair of short SATA power extensions that will mount straight.

While they're at it, why not figure out how to incorporate a 92mm (or 120mm) fan - a custom break-out connector would make that possible. But I know I'm living in dream-land with that one ;)

So look, these do work ok with the power connectors plugged in at an uncomfortable angle. I'm glad I don't have the four disk model, though, as I definitely would have needed to use extension cables there. I'm just a bit sick of lazy design decisions, so I thought I'd call them out on it. You can do better, Vantec.

 

[Ericloewe]

Unfortunately, the drives were running hot - high 50's to mid 60°C temps, constantly. That wouldn't do - I wanted sub-50°C, if possible. I looked at potential solutions, working within my design limitations (low noise enclosure, because it sits in my home office). Disassembling the enclosure to see if I could design a new shroud to allow attaching a larger fan, I realised nothing I did would get around this:

This is the circuit board that the drives attach to within the enclosure. The fan sits on this side and sucks are through those holes. As you can see, they're not exactly large. In fact, they're extremely restrictive. On the other side of the circuit board you can see electrical traces close to those holes, so there's no scope for increasing their size.

Those are luxurious openings when compared with your typical rackmount's openings. It is a bit hard to cool five drives in one of those things, but I've been pleasantly surprised by how cool my drives run in 3-in-2 bays with an 80mmx25mm fan. They're relatively quiet, too.

 

[Robert Trevellyan]

I was being a bit lazy/pressed for time, so I didn't go to the effort I could have tidying up cables:

Man, those cables are such a mess!</sarcasm>

Take a look at the position they've used for the two SATA power connectors:

Would this work better?
http://www.amazon.com/dp/B00BBDL17G/?tag=ozlp-20

 

[TrevorX]

Those are luxurious openings when compared with your typical rackmount's openings. It is a bit hard to cool five drives in one of those things, but I've been pleasantly surprised by how cool my drives run in 3-in-2 bays with an 80mmx25mm fan. They're relatively quiet, too.

Perhaps, but that doesn't make it a good situation. There's absolutely no reason the circuit board needs to impede airflow so much. They could easily design a board that is 80% smaller while retaining all connectors and circuit traces.

The Vantec is a significant improvement - the ventilation holes are around twice the size, but I didn't get a photo before I assembled it, sorry. But that still leaves a hell of a lot of room for improvement.

The drive caddies in the original enclosure wouldn't have helped, either - they used plastic between the cartridge and the enclosure and the cartridge itself was very thin. The Vantec unit uses thicker, solid aluminium cartridges that have direct contact with the chassis, so much greater heat dissipation capability.