First FreeNAS Build - 6x 6TB RAIDZ2 on ASRock C2750DI

[TrevorX]

EDIT 15 July 2015: Update here.

Hello :)

I've just built my first FreeNAS system and I thought I'd say hi. I wrote the build up and shared it on OCAU as I've been an (inconsistent) member of that community for about 16 years. This is a shorter version, as lots of stuff I wrote over there is really rather redundant here ;)

This is a project I've been planning for years, but have only recently had the cash to go ahead (and a pressing need to solve a problem). I've been running out of storage for a long time, but I haven't had the funds to create (what I consider to be) a good, reliable storage solution until now, so I've been biding my time and reading. Following a recent screwup of Windows that damaged the file system, now it's time to upgrade. Currently I'm running a workstation with a Samsung 850Pro SSD for the OS, with an 8-port Areca RAID controller running 8 1TB HDDs in RAID5. There is no user data stored on the SSD - everything is on the RAID array. I could completely lose the SSD without breaking a sweat - I can have Windows running on a replacement drive in less than an hour if I have to install from scratch (or about 10 minutes restoring from a disk image). The Areca controller scrubs the array against parity data, so I thought I was fairly well protected against common causes of data loss. The size of the HDDs means rebuilding the array doesn't take too long, limiting exposure due to a failed drive. And for a workstation, it's not a bad system (RAID6 would be better, but the original array was set up with five disks and I added three extras later - you can't add a second parity disk without trashing the array and starting from scratch).

What I've come to realise, though, is that keeping data on your PC introduces significant risk. As I mentioned, Windows screwed up for an unknown reason late last year - I booted the PC up and chkdsk popped up to tell me there were errors on the drives. I didn't think too much of it, so allowed it to scan them. About 18 hours later, I was back up and running, but within a few minutes I had noticed a number of 'zero size' files. I told chkdsk to run another thorough scan and left it for another day. When it came back up, most of the files I cared about had been restored, but quite a lot hadn't. Fortunately all my personal data was recently backed up, but I didn't have enough storage for all the movies I had on there; I estimate about 5% was lost. Thorough sector scanning tools were unable to recover more than a handful of photos - no large files were recoverable.

With my business clients' systems I push for full, live data replication between physical boxes (offsite where possible), incremental quarter-hourly backups to internal backup drives and nightly backups to external storage or offsite systems. That's on top of mirrored arrays, of course. I haven't had the luxury of adequate disposable funds to afford 7TB of external backup capacity, so I had to live with the tradeoff of backing up the much smaller amount of data that was truly irreplaceable and relying on the array never failing to keep the movies secure (which can always be ripped again anyway, it's just annoying). The lesson I've had here is that data storage must be separated from the PC - if Windows glitches and it affects the file system, you could have significant data loss, no matter how robustly you design your storage subsystem. Basically I should follow my own advice to clients - pony up the cash to do it right because the risk of data loss is simply unacceptably high otherwise.

I've been following various NAS trends for a few years on the sidelines and have been impressed with how far FreeNAS has come. I'm a Microsoft technology professional, so am very familiar with everything Windows, but ReFS isn't mature enough for my liking (and performance is terrible for 'parity' systems, so you need 100% redundant mirroring to avoid a performance bottleneck) and I want self-healing features such as provided with ZFS (a long time ago I had a faulty drive that caused slow, unnoticeable bit rot which wasn't picked up until long after the drive had been replaced. Backups had copied the bad data too. Fortunately it was nothing I couldn't live without or recreate, but I've never trusted a single HDD with data since). For similar reasons to ReFS, BTRFS is also off the table due to immaturity.

Now I'm new to FreeBSD - in fact, I have very little Linux knowledge or experience generally - what I do know has been garnered from riding shotgun while other professionals do their thing (usually interfacing with Microsoft systems I am somehow involved with). So I am far from an expert in this area!

As to my particular build requirements, I need at least two disks of redundancy, it needs to be highly available and extremely reliable, it needs to be of adequate size for the data I have and at least three years of growth, I want it to be reasonably fast for the use I will put it to, fairly quiet and relatively low power.

So here's my component list:

ASRock C2750D4I Mini-ITX (Avoton Atom C2750)
Lian-Li PC-Q35B case
Crucial 8GB DDR3 ECC 1600MHz 1.35v RAM x4 (32GB)
HGST Deskstar NAS 6TB HDD x6 (RAIDZ2)
WD Caviar Green 6TB (external backup drive)
Transcend SSD370 64GB x2 (OS drives)
Seasonic SS-660XP2 F3 80Plus Platinum 660W PSU
HDD Mobile Rack SATA/SAS 5 x hot-swap drive bays in 3 x 5.25"
Welland EZStor ME-751J Trayless HDD Rack 5.25" Bay for 3.5" SATA
Welland EZStor ME-240 Dual 2.5" SATA mobile rack
Vantec NST-330SU3-BK NexStar 3.5" eSATA external enclosure
Astrotek eSATA PCI-Express card
5.25" to 2.5" bay adapter

You may be wondering what the Caviar Green is doing in there. I'm intending to do partial backups to this as an external drive; I'm not intending to use snapshots of the whole zpool. Instead, I'm going to script rsync to backup and synchronise specific folders (essentially everything other than the Video folder).

And now for some photos (or it didn't happen, right?) Yes, my photos are lame, sorry - don't bother flaming me over them - I don't care :p

Here's the collection of parts:

And stripped bare of their packaging:

Assembled, but prior to PSU installation:

With PSU (not much to see here):

Drives ready to go:

All locked up and ready for action!

Thanks for reading, and building such a fantastic community - I've already learned so much from lurking the past couple of years! Also thanks to Ericloewe, Fraoch, Robert Trevellyan and JoeVulture for your help and suggestions :)

 

[HoneyBadger]

Research? Compatible parts? Gobs of memory and non-Green drives? Even using the Intel ports on the ASRock? A well-documented decision process with lots of pictures? Excellent cable management?

That's just pure awesome. Welcome!

 

[TrevorX]

Thanks HB :)

Hmm I just realised most of my rationale has been left out by deleting all the stuff I thought would be a bit superfluous over here... Do you think I should add it back in? I don't want to come across as preachy, but if that sort of thing would be useful I can always replace it...?

 

[HoneyBadger]

You'd be preaching to the choir if anything, so I don't think there's any harm in seeing the decision-making process. Can always put it in a spoiler tag so that it can be expanded on-demand for those who want to see the background story.

Very nice system. The only thing I might suggest is keeping an eye on drive temperatures in those hotswap bays - it looks like you're relying on the PSU and top-mounted fan to pull negative pressure, so make sure there's enough airflow across them.

 

[TrevorX]

Yea I'm trying to get freenas-tempreport running, haven't spent more than a few minutes on it yet though. Don't worry, I'm very aware of the potential problem of heatsoak on storage! A little frustrated there's no simple builtin way of extracting the SMART temp result, actually... The hotswap bay does have an 80mm fan built in, so I'm not just relying on the PSU and top 120mm, but I am relying on them to get the heat out pronto ;) I wish I had a 3d printer and CAD skills to design a replacement shroud, then I could bolt on a 120mm fan instead of the pokey 80mm.

 

[Bidule0hm]

You can see my useful scripts link in my sig if you want a simple way to see the temps ;)

Very nice build BTW ;)

 

[Ericloewe]

One problem, though: That PCI-e eSATA card looks mighty incompatible and unreliable.

 

[TrevorX]

Yes it is by far the weakest part of the whole thing, but I spent several weeks looking for alternatives and there were very, very few options - pretty much every design is at least five years old (some as old as 2005!) and they are very thin on the ground. I fully expected it to not work, to be honest, but it seems to be communicating with the drive fine. I am yet to push any real data through it (so far all I've done is check communication and formatted it), so that and the external drive itself remain the unqualified components of the build (I don't have a great deal of time, so I do bits and pieces here and there when I can find 15mins - this writeup took six weeks to put together).

If you have any alternative PCI-E ESATA suggestions I will most certainly be interested in looking at them - Vantec did such a card, which I did manage to find on eBay, but it would have cost nearly $100 by the time I got it here.

 

[TrevorX]

You can see my useful scripts link in my sig if you want a simple way to see the temps ;)

Very nice build BTW ;)

Thanks Bidule0hm I'll definitely take a look at your scripts when I get a chance.

 

[Ericloewe]

Unfortunately, SATA controllers other than Intel's are a crapshoot. Some Marvell controllers work reasonably well with non-RAID firmware, but the only solution that does work for sure is SAS, which naturally brings quite a few complications (unless someone actually makes SFF 8088 to eSATA breakout cables).

 

[TrevorX]

Thanks Eric I'll be able to start testing in a few days so I'll be able to qualify data reliability. I assume if it works OK initially there should be no issues with it going forward, should there? Assuming of course that the hardware doesn't just up and fail one day, of course ;-)

The other thing I could use is an internal SATA to eSATA backing plate connected to one of the internal Marvell ports. With the controller set to AHCI with hotswap support enabled, would that be considered a more certain/reliable solution?

 

[Robert Trevellyan]

If you have any alternative PCI-E ESATA suggestions I will most certainly be interested in looking at them...

I have no trouble using a Rosewill RC-219 with my FreeNAS. It looks like it may be discontinued - no longer available on Newegg - but still available today on Amazon. It's based on the SiI3132 controller, which is on the FreeBSD hardware compatibility list, and an Amazon search for the controller comes up with several alternatives.

I've also had no trouble with my HighPoint Rocket 640L HBA, so maybe their Rocket 622 is worth a look too.

If I remember correctly, this is for periodic backups to your Vantec enclosure, right?

 

[TrevorX]

I have no trouble using a Rosewill RC-219 with my FreeNAS. It looks like it may be discontinued - no longer available on Newegg - but still available today on Amazon. It's based on the SiI3132 controller, which is on the FreeBSD hardware compatibility list, and an Amazon search for the controller comes up with several alternatives.

I've also had no trouble with my HighPoint Rocket 640L HBA, so maybe their Rocket 622 is worth a look too.

If I remember correctly, this is for periodic backups to your Vantec enclosure, right?

Thanks for the suggestions, Robert I'll certainly take a look at those.

Yes, it is for periodic backups to the external Vantec enclosure with the WD Caviar Green, probably using rsync.

 

[joeschmuck]

Very nice and clean build. The 5 drive bay 80mm fan is likely a 80mm x 20mm, just ensure you size it before ordering a replacement. Mine lasted about 1 year before I had to replace it. My 5 drive bay also alarms during a fan failure so you need a fan with a tach output (FYI). If you were good with a dremel tool you could examine your case, see what the largest fan you could connect to across the back (doubt it would be 120mm), remove the 4 screws holding on the plastic backing, cut off the protrusion and then fit a fan to it. Now that is sounding like a nice project for me to do the next time I have to replace my fan.

 

[TrevorX]

Thanks Joe. Actually, two minutes after the first time I turned it on I dismantled the whole thing, replaced the (fairly good) Lian-Li fan with a Noctua (had several sitting on a shelf, as you do), fully disassembled the 5 disk bay, determined I didn't have a decent replacement on hand, so ordered a PWM 24x7 guaranteed Arctic Cooling model (from a supplier I was going to the following day anyway), hitched in a temporary fan controller to reduce the noise of the stock unit for a day, then got it back up and running with the sides off. The Noctua is slightly, though still noticeably, quieter than the Lian-Li 120mm. The Arctic Cooling 80mm is about 10% of the noise for 90% of the CFM! It is also reporting back to (and fully controllable by) the mainboard, which is more than I can say for the annoyingly limited three pin fans...

Oh and yes, you're right - hotswap units like this do indeed usually have fan fail alarms - in my experience, they trip even if the fan isn't within a predefined nominal RPM range. But oddly this one has no alarm at all. To be perfectly honest I'm not even sure where it came from - it was another component that came off a shelf after sitting there for a couple of years. It's also not the type I would normally have used - I usually spring for ones with individual power buttons and failsafe locks. It must have been purchased for a project that never went ahead, but it saved me a couple of hundred dollars, and to be honest with the front fascia of the Q35-B closed it is well protected from accidental ejection (and I can't see the fancy activity LEDs of the nicer units anyway).

Having had the back off the hotswap chassis (several times now) it wouldn't be too difficult to fabricate a shroud to fit, but it would be pretty ugly - instead of the shroud narrowing towards the fan, it would be almost straight as a 120mm fan is almost exactly the same dimensions as the rear of the whole unit. I wouldn't bother modifying the existing shroud as I'd lose about 40% of my surface area going to a 120mm. Thinking about it, I could probably design it in something like SketchUP easily enough... Maybe I should see if someone local could print it for me... It would definitely give me a lot more confidence, and that's what this is all about!

Now if only I could run an air-conditioning duct into the front of it...

 

[joeschmuck]

Now if only I could run an air-conditioning duct into the front of it...

I'm sure you will figure that one out too.

For my fan in the 5 driver bay I ended up taking the fan and running it off 5VDC vice the full 12VDC. The alarm remained silenced. Since you do not have an alarm you could run your fan from 7VDC (+5 and +12 lines) and then call it a medium speed fan.

 

[TrevorX]

No need - the Arctic Cooling Fan is sufficiently quiet already Going to 120mm would allow a significant increase in cfm with no increase in noise, so I'm keen to look at that, but if I have to stick with 80mm the one I have in there now is fine

 

[joeschmuck]

Near the end of a scrub or high activity check the drive temps to see where they are hovering, that will tell you if you should upgrade the fan. If temps remain below 40C (depending on your ambient temperature of course) then you should be fine.

 

[TrevorX]

Thanks for the suggestion Joe. Hoping to get a chance to look at drive temp analysis in the next few days, just had a lot on.

I do have one question, and I realise I'm being lazy writing it here, but what the hey - I'm writing this reply already so I may as well pop it in... Last night my wife made me get out of bed and turn the NAS off. Not because of fan noise - that is barely audible outside that room. No, you see the NAS sits on a shelf, our bedroom is next to the home office room and the vibration from those six HDDs is causing a sympathetic vibration through the wall that creates a 'hum' that sounds similar to having an outdoor air-conditioning unit on the other side of the wall. Now, from the reading I've done, it looks like the NAS never powers down the drives because it writes fairly continuous logs to it. To be perfectly honest having them running constantly isn't that terrible in terms of longevity anyway, as parking heads and then spinning back up places substantially more wear on a disk than running it constantly anyway. However, it would be nice to know if this is in fact precisely what is happening and if this is 'normal' behaviour that everyone else is seeing also?

I'll get some vibration dampening feet to sit it on, but if that doesn't work it will just have to go on the floor somewhere... It's a shame 'cause it fits really nicely on that shelf :-/

 

[Bidule0hm]

This is more or less normal. To reduce the vibrations you can add weight to the case and/or add silent-blocks between the case and the shelf and/or use some vibration isolation system between the drives and the case (but be careful because that can lead to less vibration dampening on the drives and reduce their lifetime) ;)