ZFS with or without Hardware RAID

[lecrucious]

Wasn't sure if this was the best place to post but since I'm new to FreeNas, thought it was a good try.

So here's what I am trying to accomplish:
I want to create a NAS, running FreeNas. This guy is gonna be my big honking storage device for everything. AFP, iSCSI, NFS, CIFS, Snapshots, Replication, etc. I want to make it virtual, ESXi. The existing host has plenty RAM and a good number of empty bays for later expansion.

What I can't decide is Hardware RAID with Raw Device Mapping (RDM). Or do I go straight RDM per disk and use software RAID.

I have a good hardware RAID controller so this is not a concern. I am also aware that HW vs SW RAID can be a bit of a touchy subject. Let me start by saying my question is not which is better in general. Depending on the application I have used both and am comfortable with either. My biggest concern is, from a ZFS perspective, which will give me the best long term stability and flexibility? This wouldn't even be a question if the underlying filesystem was something else like NTFS.

In the past I have usually used HW RAID and just let the controller handle the disks. But reading through the FreeNas docs I saw a phrase that said "ZFS prefers direct, exclusive access to the disks, with nothing in between that interferes." The document essentially dedicates a paragraph to say, "Let ZFS do it's thing and don't try to help."

Since I am new to ZFS I didn't want to take any chances and am hoping to get it right the first time. I understand using RDM has the added complexity of properly identifying failed disks but with proper labeling and documentation...not really worried.

So to re-cap. I'm new to ZFS. Is it really the uber-filesystem that I should let it have the disk directly and let it handle data integrity/fault tolerance? Or do I use "old trusted RAID controller" who, even with some horror stories, is mostly reliable?

 

[paleoN]

This guy is gonna be my big honking storage device for everything. AFP, iSCSI, NFS, CIFS, Snapshots, Replication, etc.

So to re-cap. I'm new to ZFS.

Then you need to read up on ZFS some more. Unless you usually entrust everything to something you don't understand?

• 
  

  When you have some time I would encourage you to read the obligitory wikipedia link:
  
  http://en.wikipedia.org/wiki/Zfs
  
  and take a look at this presentation:
  
  https://blogs.oracle.com/video/entry/becoming_a_zfs_ninja

I want to make it virtual, ESXi.

I wouldn't.

The existing host has plenty RAM and a good number of empty bays for later expansion.

That's relative. The actual numbers?

What I can't decide is Hardware RAID with Raw Device Mapping (RDM). Or do I go straight RDM per disk and use software RAID.

My biggest concern is, from a ZFS perspective, which will give me the best long term stability and flexibility?

"Software RAID". You will want to let ZFS handle all the RAIDing. Doing so will give you the best long term stability and flexibility.

Since I am new to ZFS I didn't want to take any chances and am hoping to get it right the first time. I understand using RDM has the added complexity of properly identifying failed disks but with proper labeling and documentation...not really worried.

I would be. Perhaps it would be worth testing this out before you stuff all of your data on it.

Is it really the uber-filesystem that I should let it have the disk directly and let it handle data integrity/fault tolerance? Or do I use "old trusted RAID controller" who, even with some horror stories, is mostly reliable?

It's uber enough and better at data integrity.

 

[cyberjock]

Not to sound disrespectful, but RTFM. Your questions are best answered by you. The manual is VERY well written. I had never used FreeBSD before(or FreeNAS) and knew nobody to ask questions or help figure it out. In less than 30 days I had a fully functional system and avoided all of the common noob mistakes....because I read the manual. That's all you need to read.

My only hiccup with my first setup was I had a RAID controller that was compatible with FreeBSD, but wouldn't work with FreeNAS. The manual claimed it was compatible. Later I found out Highpoint's manual was in error and it wouldn't work.

 

[lecrucious]

noobsauce80, thanks for your reply and no biggie. No disrespect taken.

Perhaps I didn't make it clear in the first post but I've been through the manual, probably five times, ok maybe four, and I agree it is very well written. Thanks to the manual, and a few other resources, I was able to resolve a lot of initial issues on my test box without posting to the forum. For example, one noob mistake that is incredibly easy to avoid is allocating enough memory, but I just had to low ball it and learn the hard way. Hey that's what testing is for. However, with my being new to ZFS, some of the terminology and underlying details are not absolutely clear in my mind even with all the reading. I find forums can be a good place to fill those gaps. Most of the time I have read the answer but one of you guys may word it just a little bit different that helps me grasp it better.

Thanks again.

 

[cyberjock]

Check out the presentation in my sig. That may help too. If you have Microsoft Office the powerpoint is very helpful with the animations.

 

[rickwhois]

If you're new to FreeNAS, I'd definitely go through the manual more, make some practice vm's...etc, get familiar with it.

I'll say that, what you are looking to do is definitely possible. I just went through a migration to esxi 5. Freenas 8.2 runs great in this environment. As for freenas 8.2 standalone, my major beef was monkeying with plugins, I really love freenas 8.2, but dread the plugin system at this time. ( services like serviio and minidlna are pretty bad). I needed something that I could use more services like mediawiki, sick beard, "PLEX"!!! etc. You know, something more like linux. But I also wanted to continue using freenas for storage and sharing.

So I built the esxi system with freenas(storage & sharing), ubuntu server(sabnzbd, cps, etc...application server), lion server, win2k8 r2, etc... They all run smooth. So I raw mapped my hdd's to esxi datastore and then imported my existing zpool software raid (6x2tb zraid 2) right into my freenas vm and it worked flawlessly. (but backing up data is always recommended).

this link here is useful for rdm. (I went with rdm b/c my sata's would not pass-through in esxi) otherwise you might be able to pass-through your sata controller to the freenas vm. http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/

I am not by any means a master at esxi or freenas. I've been using freenas since ver 7 & I've only used esxi for 4 months. But there are a few threads on the webs regarding esxi & freenas so you'll have some resources. I'll echo what others have said on this thread and say that reading the manual and cheat sheets will save you more time in the long run.
*one good thing about rdm, is that if my esxi were to go down for some reason, I could still import my zpool from a standalone fn8.2 install.
*if you go esxi, get ready to buy memory!! freenas & my app server eat up lots of ram

 

[lecrucious]

paleoN, thanks for your reply as well,,,thought I replied back this morning but looks like it got lost in the shuffle. I'll definitely be taking your advice on furthering my understanding of ZFS before going Prod. I've had a test box running for about a month now and successfully deployed most of the services I am planning for in Prod.

ESXi, because I already have the infrastructure set up and the hardware so it will make it easier to get started.

16GB of RAM, only three of which is currently in use and six empty drive bays of which four would be dedicated to FreeNas for data. I am not looking to exceed 4TB in this setup. If my needs grow then I'll likely move to a dedicated box.

I'm ok with RDM because virtualization and datacenter administration is my day job. So RDM is just another day at the office.

Sorry for the bullet points, my original post was more thorough but I'm pressed for time.

Thanks to you and rickwhois for the additional info. Looks like I'll be going software RAID and let ZFS work it's magic.

 

[paleoN]

16GB of RAM, only three of which is currently in use and six empty drive bays of which four would be dedicated to FreeNas for data. I am not looking to exceed 4TB in this setup. If my needs grow then I'll likely move to a dedicated box.

In that case I concur, you will have enough capacity. 8GB for the FreeNAS VM plus the 3GB existing still leaves you with another 5GB. What's your planned zpool config, 4 2TB drives as a pair of striped mirrors?

I'm ok with RDM because virtualization and datacenter administration is my day job. So RDM is just another day at the office.

I didn't expect you to change your mind. There have been a couple of other people who decided to virtualize & use RDMs and got burned. I imagine you can handle it assuming you actually test for failure.

 

[lecrucious]

I haven't decided yet. I won't have all drives at once. I will likely start with 2x 1TB drives and add two more later. These are small form factor so I may be limited to 1TB each. As for the zpool config, not sure there yet either. I was considering, like you said, a pair of striped mirrors but capacity is more important to me at the moment than performance so maybe a RAIDZ1.

I still need to get my head wrapped around the various options to figure out which is best for me. Oh and BTW, that ZFS ninja series was awesome. The guy really makes it easy to grasp the concepts.