3ware and FreeNAS and Mirroring?

[rustynail]

I have been using 3Ware cards for some time now. They have been very reliable and fairly easy to replace defective disks without down time or disruption.

Typically these have been under an Ubuntu install running as a LAMP or or Vmware Server. But now I am looking at consolidating my storage into a single device.

Looking at setting up 12 disks in a raid 10 under a 3Ware SATA controller. Then do an install of FreeNAS on another disk that is connected to the computers motherboard.

Here are my questions:
1) Am I going to be able to monitor my 3Ware controller from within FreeNAS as I have under Ubuntu? I know not exactly the same, but what are my options for monitoring a 3Ware controller under FreeNAS? SSH to tw_cli ok?

2) I have done the this setup on a smaller scale with success, but not with this many disks. Are there any comments on what I am proposing? Better and economical methods?

3) I think that I have read that FreeNAS has the ability to mirror itself. Right now I will have only 1 up but want to get a mirror going later next year. Are there any suggestions or recommendations on the first one that I setup so that a mirror will work? What I have done in the past is an rsync between a source and destination. Not exactly a mirror, but is fast and effective.

Thanks.

 

[jgreco]

With ZFS, you're much better off NOT using the RAID features of the 3Ware controllers. ZFS offers great recovery capabilities and is tightly integrated with FreeNAS.

FreeNAS expects to be installed on a flash drive, not a disk.

Using RAID 10 with 12 1TB disks, you wind up with 6TB, and the ability to withstand a single disk failure (yes, you can theoretically lose *up* to six of them, but you have to get very lucky as to pairwise selection).

Using ZFS RAIDZ2 and 12 disks, you wind up with 10TB, and the ability to withstand two disk failures.

With ZFS, you can use snapshots and replication to mirror one ZFS filesystem to a backup on another filesystem. That's a nice plus for ZFS.

 

[ian351c]

From all the research I've done, here's what I can tell you:

1) Short answer is yes. Command line (tw_cli) is built in. Daily status reports include alarms from the 3Ware controller (if there are any alarms). No support out of the box for the 3Ware Web GUI.

2) FreeNAS supports ZFS. All my research tells me that ZFS provides superior recoverability with comparable performance to using the 3Ware RAID. One note: (and I can't verify this yet as I'm still in research mode...) you'll want to either get a small SSD as a "separate device" for the ZIL (ZFS Intent Log, similar to a journal, but on steroids) or use your 3Ware card in Single Drive mode (similar to JBOD, but you still get to use the 3Ware on board RAM for write caching). Using the 3Ware card with either a BBR module or a decent UPS should allow you to keep the ZIL on your RAID (RAIDZ in ZFS terminology) and still have good write performance. You'll want a 9650 or newer card to support Single mode. See: http://forums.freebsd.org/showthread.php?t=15176

3) I haven't looked into this, but my understanding is that FreeNAS can be set up to mirror to another device (even another FreeNAS box) via rsync. Not exactly High Availability in the Enterprise sense, but close enough for SMB or Home Office.

 

[rustynail]

Thanks for the responses.

It never occurred to me about using only software raid. What kind of performance hit is there with a RAID-Z2 in comparison to a hardware RAID-10?

The 3Ware controller I am considering is the 9650SE-12ML with the BBU.

As for the install of FreeNAS, it definitely won't be on a compact flash, but a small 2.5 SSD. Long story short, on my home NAS, something went completely wrong with a compact flash setup and I lost all my data. Fortunately I have a backup of my 4TB via rsync and nothing was lost. With that experience I am very hesitant on using compact flash for OS installs.

Doing a lot of research right now on the RAID-Z2. But what I can't find is a performance comparison to a RAID-10.

 

[Milhouse]

Even an Atom-class CPU can saturate a GigE connection with RAID-Z2. The advantage of software raid of course is that you don't need an expensive card to manage the array, or find exact same card on eBay when your original controller bites the dust as it's the only hardware that now understands the way your data is stored.

SSD is overkill for the FreeNAS install - you only install the OS on the CF/USB/SSD, never any data. Even if the CF/USB/SSD/HDD became corrupted, it's hard to see how FreeNAS could trash your ZFS data unless it accidentally executed the command required to destroy your pool. I appreciate you had a bad experience once, but you're wasting money on an SSD just to host FreeNAS.

 

[b1ghen]

Hi,

You can use a RAID10 like setup with ZFS as well if you want, in your case setup 6 mirror 2-drive vdev's, then create the zpool out of those 6 vdev's and voila you have a "RAID10" with ZFS. This should net plenty of performance for most uses. The best way is to actually test things out since you seem to have the hardware already, try different variants of mirrors, RAIDZ1, RAIDZ2 and benchmark them. Then factor in the redundancy you think you need and make a decision.

And most people boot their Freenas of a USB key, with backups of the config a crashed USB key shouldn't cause any dataloss from my understanding. I haven't had one crash yet though but since most USB keys are MLC based it's more likely to crash than a SLC based one. If you can find a SLC based USB key of 2GB+ I would go for that as my boot device.

 

[rustynail]

I did not blame the corruption to FreeNAS. Other than the fact the CF to SATA card becoming very unstable, leaving I think the entire SATA system unstable, thus me possibly making a mistake in the solving. I also have setup 12 KIOSK's with 8 GB CF to SATA cards and they all seemed to out of nowhere fail. Probably poor quality hardware.

An 8 GB SSD is $40 from a major manufacturer versus my confidence in CF, I will definitely squander the $40 in this project :) - It will help me sleep at night.

The key on the hardware is RELIABILITY, not $$$. I have the utmost confidence in 3ware controller cards whether they are used in raid or the combining 12 SATA drives into FreeNAS for a ZFS Raid.

 

[ian351c]

Many people encourage the use of USB sticks because they're cheap and they don't take up a SATA port. But then they stick out of the back (usually) and if you have a RAID card, you're not using your built-in SATA ports anyway. Different strokes and all that...

As for performance, google tells me that RAIDZ and RAIDZ2 performance is either a little better or a little worse than RAID10. Rebuild times are a usually a lot faster with RAIDZ. Either way, you should have enough performance to saturate a 1Gb link. Given the flexibility that RAIDZ gives with the performance of the 3Ware card in write back mode (which I already had lying around), my road will take me down the RAIDZ on 4x2TB Single Mode disks on the 3Ware controller. YMMV, Void Where Prohibited, No Deposit No Return, etc...

 

[jgreco]

"saturate" a 1Gbit ethernet may be optimistic. For large sequential access files, with caching and maybe a tailwind, yeah, okay. It'll be fine for most things.

An SSD has reliability characteristics similar to a HDD. See http://www.zdnet.com/blog/storage/ssds-no-more-reliable-than-hard-drives/1483 etc. You have no unusual reliability in a SSD, unless you've got them in a RAID. If you want to sleep comfortably at night, either put your SSD's in a RAID, or have a cold spare SSD, or use USB flash and keep a spare. That last one will be cheapest by far, allows for trivial FreeNAS upgrades, and may actually be more reliable than a SSD, because - at least from my observation - flash usually suffers from infant mortality more than failures down the road, if treated well, and flash is so cheap that you tend to just keep rotating in new modules periodically as part of your upgrade strategy.

 

[ian351c]

Well, that's exactly my use case (except maybe for the tailwind, since my NAS is in the basement . But yeah, maybe "saturate" was a poor word choice. From what I've read, it will be hard to tell the difference between RAIDZ and Hardware RAID10 for most workloads if the connection to the NAS is a 1Gb ethernet link.