Pulling the trigger on FreeNAS, mind answering a couple of questions?

[Danger]

Hey everyone, sorry for typing a book here, but I have been researching building a FreeNAS server at work and need some advice before I get my boss to pull out the credit card. I am hoping I get quality responses by providing a well thought out post instead of posting a bunch of threads on individual questions and problems.

I work for a small, but quickly growing, medical device manufacturer. We basically manufacture glorified Windows computers with some custom medical hardware attached. We have around 20 employee PCs in our main office right now, and then employees all over the country working out of their homes. I built a SBS 2011 Server a couple of years back that handles our domain, permissions, and especially full disk backup for everybody in the office. Trust me when I say that most of us are lazy and unorganized enough with data that we need regular incremental backups, and need backups retained for a while. The owner will gladly keep paying for more disk space if it means he doesn’t have to organize the files on his desktop and laptop, haha.

This server I built was great 2 years ago, but we (and I) have quickly outgrown it. It’s a 6-core AMD desktop CPU with 16GB desktop RAM and a bunch of random hard drives. A while back I converted it to run VMWare ESXi 5.0 so I could make better use of the hardware and RAM to play around with other stuff, including a separate VM for our extremely undemanding database program we run just to keep it safe and separate.

With a new product line we have begun using WDS and MDT to deploy Windows 7 to our new machines. I now have a Windows Server 2008 R2 VM handling that. I have already begun to cringe as I attempt to deploy a machine while the SBS VM is running a mid-day backup. It literally takes 4 times as long to deploy a single Windows 7 machine as when the server is “idle”.
To top that off, we are moving offices and need to set up 5-6 IP surveillance cameras. Why not store full 1080p at 30 fps, am I right?!

I already got my hands on 3x 3TB WD Red HDDs and have been messing with FreeNAS in my office on a spare PC as a proof of concept. I have it up and running, and my boss is approving what I say I can do with it. Now I just need to make sure I can actually do it…

I am looking at running a total of 6x 3TB WD Red HDDs right now. Yes, I know they aren’t enterprise grade, but my listed expenses for this project are already making my boss’s eyes pop out. Most of the data stored on this NAS is not going to be extremely crucial, I fit everything we consider important on an external 2TB drive and take it home with me on a regular basis to have an off-site backup. Anyway, 6x3= 18TB storage. Subtract 6TB for parity and I’m looking at 12TB of total storage space. We can work with that for a while, I think.

I read some hardware info around here and I think I am sold on a SuperMicro motherboard and a Xeon E3-1230 V2 CPU. Since the cheaper SM motherboards only come with 4 RAM slots, I would love to start off with 16GB RAM (2x8GB) and leave the other 2 slots for future expansion. If this CPU/MB/Case combo can last us 3+ years it will go a long way in convincing my boss to pull out the credit card. The IBM ServeRAID M1015 looks well within the budget in regards to a storage controller.

Will 16GB RAM be adequate for 18TB-6TB parity = 12TB storage? I would like to eventually max out this setup with 36TB-12TB parity for 24TB storage and 32GB RAM before we have to purchase an entirely new system. Feasible or bad idea? I know more RAM = Better, but I’m trying to keep costs as low as I can.

I know this isn’t a ton of help without hard numbers, but am I looking at overwhelming the CPU or HDDs with simultaneously storing 10MB/s of video, performing PC backups, and deploying a Windows machine at the same time? Is it obvious I will need a couple of SSDs, or too hard to tell?

I read in the slideshow in the Noobs forum that it is not a good idea to use iSCSI and ESXi datastores, but why? I was really excited about this project until I read that. I definitely planned on using it for both of those. I have searched around and found no shortage of people using FreeNAS for iSCSI and ESXi datastores. Is it THAT bad? What are my alternatives? Unfortunately SBS 2011 refuses to use a mapped network drive for backups, but iSCSI does work.



I know I have a lot more to learn, I just hit a point this afternoon where I had to decide if this was worth my time pursuing or if I need to cut away and find a different solution. I really appreciate the help from everybody.

 

[cyberjock]

Howdy. I wrote that presentation. Glad it helped.

RAM:

2 things to consider for RAM- Having 8GB means your system won't crash; and more will almost always mean better performance. So to answer your question, your server won't crash but performance will depend on how much loading your server has, the type of loading it has, the CPU's performance, etc. Too hard to tell if you'll need more or less, but I'd expect it to be "good enough" for your purposes(ignoring the fact that you mentioned iSCSI and ESXi... more on that later). I can promise you that if you go to 20TB+ of total disk space you will almost certainly need more than 16GB of RAM. And you'll know if you need more, the server will start sucking performance-wise. Mine did. 10MB/sec total transfer speeds told me I needed more RAM. Upgraded and performance was immediately over 100MB/sec again.

My system is an E3-1230V2 with 32GB of RAM and a Supermicro X9SCM-F. Great combo and if I had to build a new server for myself today I'd go with those again in a heartbeat.

iSCSI and ESXi, that's a very very long story. I really don't feel like writing a book about the technical problems, but I'll provide some insight into some of the possible solutions...

I'd recommend you search the forums and read some of the tickets at bug.freenas.org if you want to know all the technical stuff(there's a ticket 1500-something written by jgreco that has some stuff about the issue). The bottom line is if you want to use iSCSI and ESXi datastores with ZFS you almost certainly will have to do some ZFS tuning(which is non-trivial and could take weeks to get right assuming you can get the experience and knowledge off the internet to even get it right). ZFS tuning is very much a personal thing. What works for you won't work for me or the guy down the block. You have to know your hardware, your needs, and how to tune it to work for you. If you aren't willing to dig deep and be willing to put 20+ hours into research before you even begin trying things you should probably not attempt it. Also you will almost certainly need to buy some high speed SSDs(yes, plural). You will probably need a ZIL and perhaps an L2ARC.

Note that before you go with an L2ARC you will need lots of RAM. Probably 32GB to start. L2ARC eats RAM to store the index, so you will need more RAM just for the index. So now you are talking about something that is perhaps more than what a socket 1155 board handles and you are into the high end expensive socket 2011 stuff or 1366 with 48+GB of RAM. There's no way to guess how much RAM you may need because again it will have to be personalized by you for your setup. The other option is to use UFS for iSCSI and ESXi datastores.

I'd guess that about 90% of new users to FreeNAS that want to use iSCSI and/or ESXi datastores with ZFS will have given up on it within a few weeks. Most will give up on iSCSI or the ESXi datastore entirely or they will have switched to UFS because the tuning thing is damn near impossible for a newbie to do. It will require many many hours of reading, experimenting, and pulling your hair out. If its any consolation, I haven't tried to use either because I know the fight is uphill, against the wind, in the snow, etc.

How much your server can/will do is dependent on how deep your pockets are and how dedicated you are to learning ZFS. Willing to spend some money and alot of time on research, and you will get it working. On the other hand if you don't have either, then walk away and save yourself the trouble. ZFS is an amazing file system. If your data is very very important to you it is worth learning about. But it's not for everyone. Plenty of IT guys have gotten their butts handed to them trying to figure this stuff out, and non-IT are really in trouble if they think they can pick this stuff up in a week or two.

 

[Danger]

Thanks for the response, Cyberjock. I think you misunderstood because I was talking about starting with 16GB of RAM (2x8GB) and eventually expanding to 32GB. Looking at the cost of the rest of the build, I'm thinking about pushing for 32GB right off the bat due to it only costing less than 10% of the entire build. However, then the motherboard is maxed, which sucks.

I found the ticket you were talking about (https://bugs.freenas.org/issues/1531 for anybody curious) but I haven't read the whole thing yet. I'm brainstorming how I can keep the VMs living on local storage in the ESXi server and only use the NAS to back them up. Although I suppose I could use that as a last resort if I can't get the ZFS tuning figured out.

How big would the SSDs need to be?

Thanks again.

 

[cyberjock]

ZIL is strictly writes, but bigger SSDs mean faster SSDs, so generally bigger is better.

L2ARC is similar, but you don't want to oversize it. Then you use too much RAM for no benefit.

It's really a personal thing based on your usage patterns.

 

[gpsguy]

I wouldn't use a VM for these deployments. Do it on physical hardware with local storage. Consider putting it on it's own VLAN to separate the traffic from your production network.

With a new product line we have begun using WDS and MDT to deploy Windows 7 to our new machines. I now have a Windows Server 2008 R2 VM handling that.

When I heard you say medical, PHI and HIPPA came to mind. Do/will you need to encrypt the backups you take off-site?

...I fit everything we consider important on an external 2TB drive and take it home with me on a regular basis to have an off-site backup.