Virtual Machine Best Practices...

[Chris Weir]

Hello,

I've been reading through a few of the stickies and a few other things. Right now, I'm only really "experimenting" with FreeNAS, ultimately with the idea of using it in a production environment - a small office.

Reading through the form so far, virtualised FreeNAS appears to be a bad idea. Right now, and initially in production, I would really like to have this running in a virtual machine.

For testing purposes, the machine I'm currently messing about with is a Sun Ultra 20 M2 with the following specs:

Dual Core AMD Opteron 2.3GHz,
8GB DDR2 ECC Memory,
HP P400 Hardware Raid controller with 512MB cache and battery backup, currently running in Raid 5 with 3x 200GB Hard Drives,
SUN Workstation/low end server motherboard, with Dual Gigabit networking, all based on nVidia 3600 series pro chipset (If memory serves),
Running Windows 2012 Server. Full license.

This machine has been running two virtual machines, headless. One is a dedicated in use MySQL server, and the other a LAMP server for testing purposes. Both of these are CentOS 6.5 VM's that at their very peak uses 512MB of RAM (I'm using Dynamic memory allocation for these, which CentOS 6.5 supports). I know I could run these services directly on Windows - but I find dealing with VM's and future upgrades will be far easier using VM's. It's easier to administer them too, without having to worry about affecting the other machines.

In addition the machine has power conditioning and protection in the shape of an APC 1400VA UPS. This is another reason why I like the Virtual Machine setup - Windows Server automatically saves the state of the machines and shuts down the system safely during a blackout. The price and time needed to configure another box is currently prohibitive.

This whole setup is rock solid and has, touch wood, never crashed or missed a heartbeat once in the last 3 months of being online.

This is all used in a tiny two PC office, with the MySQL server being used with custom written software that I have written, and is used purely to manage inventory, customers, eBay listings, and orders. It's under fairly low load, which is a good thing considering the age of the hardware - all decent high quality kit, it's just not going to break any speed records.

This is a family business, and the LAMP is essentially just being used to test and create the online store. Once it's up and running, the LAMP will be unused and the website dumped onto dedicated hosting. We are unsure whether to do this with the MySQL database, considering the offices internet connection is not the best. This would, however, allow us to have the store website hook right into our main database, and the redundancy that comes with professional hosting could save a future headache. Anyway, that's another question for another time.

So, I hope you get the just of the situation. Sorry for the story - I get a bit carried away here.

Right, back to FreeNAS. The main reason I want to use it is for a) a file repository for both the networked PC's and any future additions - I don't like individual file shares on these client PCs, it's just messy and not very elegant. b) I would like to avoid using Windows Server for this - I purely want to use that to manage and host the VM's, nothing else. c) From looking around, the dedication of people (and, I use this in the most complimentary way possible, there are some seriously anal people here who like things done right - my sort of people!) make this an attractive platform to use with decent support. d) I want to avoid Active Directory - it's not needed, and I'm really not familiar enough with it. However, due to real life constraints and the fact I have to prove this before proper implementation, there are some limits on how I would like to have this "done right".

Currently I have it up and running, serving a 200GB share. All fine and dandy. CPU load is low, and the RAM usage is low. Currently it is using only UFS as the hardware limitations prevent the use of ZFS.

Once funds are available, the machine will be replaced with concurrent technology, preferably using a hardware RAID 6 array with battery backup cache and the same UPS, and Dual (redundant) PSU's. I would, however, like to be running only a single machine and use VM's. I have been eyeing up a HP DL380, but I can't spend right now until the business grows.

So, now for the questions:

Reading from other forms, direct access to the hardware is required in order to safely and reliably run ZFS volumes. I understand it's for SMART access and that it is designed to have raw access to the hardware as it does all the RAID implementation, as opposed to how I currently have it done via hardware. I appreciate this. But I would like to know: a) What is the major advantages of ZFS over my current hardware RAID 5 setup with Battery Backup Cache? b) Are there any other issues with using FreeNAS in a VM that I should be aware of? and c) Would a UFS volume, backed by RAID 6, be a reliable enough alternative to a ZFS volume?

I have read http://forums.freenas.org/index.php...nas-in-production-as-a-virtual-machine.12484/ and still not entirely sure of the bottom line.

The NAS will be used primarily to store the Windows volume images of the 2 PC's for backup purposes and used as a central repository for work files. The said PC's also have volumes that are currently used by Windows Server 2012 for full system image backups (including the VM's and their data disks). Think of it as distributed redundant backup.

I have two, 320GB, and one, 1TB, external hard drives that can be used for additional backup.

Anyway, sorry for such a long post, I have just read other posts asking similar questions, and I wanted all the information out on the table so there's no room for misinterpretation, and that my limits can be seen.

Cheers,

CW.

 

[Chris Weir]

Here's a couple of Pictures of the current office and the SUN "Server"

 

[cyberjock]

If you want anal you've come to the right forum. We're very anal... keep that in mind with the answers you're about to get...

Reading from other forms, direct access to the hardware is required in order to safely and reliably run ZFS volumes. I understand it's for SMART access and that it is designed to have raw access to the hardware as it does all the RAID implementation, as opposed to how I currently have it done via hardware. I appreciate this. But I would like to know: a) What is the major advantages of ZFS over my current hardware RAID 5 setup with Battery Backup Cache? b) Are there any other issues with using FreeNAS in a VM that I should be aware of? and c) Would a UFS volume, backed by RAID 6, be a reliable enough alternative to a ZFS volume?

a. ZFS has bitrot protection, superior rebuilding through the use of ZFS as a file system and logical volume manager, and much higher performance demands. You shouldn't mix hardware RAID with ZFS, period. There's a list of reasons I don't want to get into here, but just don't do it. You touched on one with SMART, but only one. If you want to do hardware RAID you are going to want to go with UFS. Unfortunately 9.2.1.x is the last branch with UFS. So you are looking at a dead end if you want to go that route. So be smart and go with ZFS and don't even consider the RAID controller.
b. It's broken. We don't know why. We don't have much information, and we really don't care. It's something that just isn't worth time to investigate because it's clearly something to do with the virtualization layer which we don't have access to. So we simply tell people not to use it, and leave people high and dry if they decide to ignore our warnings.
c. There is no substitute for ZFS. Period. Some people would consider UFS + RAID6 to be "suitable". After all, it's been accepted practice for years. But why consider UFS if it's already dead to FreeNAS current testing releases?

In short.. use ZFS, don't use a RAID controller, and don't try to virtualize.

 

[Chris Weir]

Thanks for the quick reply. I don't particularly want to go down the route of using end of life and effectively end of support software. In addition, the hardware controller, although reliable and a decent card with the 512MB cache and battery backup, is getting on and only supports SATA1 (1.5gbps) and 2TB volumes (I think the latest firmware I flashed might have resolved this). But I digress, ZFS will be what I'll use.

I did consider the issues with Hyper-V, as I had to look around and use a botched build in order to get FreeNAS to see the virtual networking adapters. Not good. But it is up and running for currently messing around with, with working shares and mounted volumes.

Ideally it would have been good to have everything on one machine for such a small setup. The SUN machine is of very decent quality as well. What I might end up doing is using that on it's own for FreeNAS, and use the raid card from it in another build for the VM's. It currently only has 8GB of ECC ram though, but as I don't envisage needing more than a TB of storage for now, it can stay at 8GB. (The motherboard is maxed out and can only take 8GB maximum). The constraint right now, surprise surprise, is money. But I also do not want to end up with a half arsed system that might give up the ghost when it matters most. I have a load of old (read, late Pentium 4 era based Xeon servers), they are in the first picture sitting at the back ontop of the black box. But they're no good for this, and noisy as hell.

For now I'll need to just be happy with Windows shares using Windows server, and the RAID 5 with the "distributed backup". In a few months, I'll migrate that server installation to a new machine, and turn the SUN machine into a FreeNAS box, or more likely, leave the server where it is and build a FreeNAS box. Another alternative is to use the FreeNAS VM as it is now, using UFS on the virtual disk that is on a RAID5 array, until I build a dedicated box.

I've downloaded your PDF guides. I'll have a look at these and sit on the issue for now until I have a budget to play with. If things work out, we may have a fibre to the cabinet and a cable internet connection, for redundancy and speed, and have everything but the SUN machine, as a FreeNAS box, hosted online.

Btw, the top two pictures are old pictures of the office. It now has blinds on the windows, and racking at the back where those (very old) rack mount servers are sitting. Also, there are now filing cabinets sitting where those black drawers are sitting. Yes, it is a converted garage, but it's well insulated, has cable internet connectivity (only 20MB for now) a phone line, a fax machine, sarcastic posters and even a whiteboard! Just waiting for a water supply for the coffee machine :).

 

[cyberjock]

Whoa.. a Hyper-V user. You are definitely on an island with that choice. You might be the second or third user to even try using FreeNAS on Hyper-V. Nice to see that it actually works since it wouldn't work at all 6 months ago.

Nice little office. That lock reminds me of the locking device we had for rooms containing classified information in the military. ;)

 

[Chris Weir]

Yeah, I got a license for Windows Server 2012 data centre edition for free, so may as well put it to good use. Whenever I've played with VM's in the past, I've always used Virtual Box. I'm not a MS hater, but I really didn't expect much from Hyper-V - but really, I can't fault it. It's been a smooth ride so far - I suppose it really should be considering it's a professional commercial product.

Here's a screen shot of it running in the VM. The right side is my second monitor that I'm using to show a remote desktop of the SUN machine running Hyper-V. The left screen is my desktop that I'm transferring some data to the shared volume. You can see that I'm averaging 36MB/s, with the transfer, while CPU on the VM is at 44% and RAM usage at ~700MB all in. Not really that bad considering the overheads with using a VM and the RAID 5 penalty (The P400 card has a poor reputation for RAID5). I'm also using a dynamically expanding volume so that would be another overhead. Btw, just before you freak out, this is purely experimenting. It's not going to be implemented.

The SUN machine is currently here at my home away from the office while I do some upgrades on it. The battery for the write cache was one of them - it won't enable the write cache without a healthy battery, so that was a job for the old solder iron (HP no longer manufacture the battery, so new NiMH cells were needed - replaced with a PP3 connector and a 4 cell AAA battery box for easier replacing in future).

Reminds you of the military? Working side by side with my father is like being in the military! What part did you serve in? I was a reservist in the Royal Navy myself until they decided that having psoriasis was an issue. Didn't see much or go many places in my short stint though.

 

[cyberjock]

I was US Navy.

 

[jgreco]

I have read http://forums.freenas.org/index.php...nas-in-production-as-a-virtual-machine.12484/ and still not entirely sure of the bottom line.

The bottom line is that you should not run FreeNAS in production as a virtual machine, particularly if you are a N00b, which is why that post is prominently posted in that forum.

Getting FreeNAS to work well is tricky enough on bare metal, and adding the virtualization layer along with the plethora of terrible choices that it allows you to make increases the number of moving parts and opportunities for things to go way wrong.

Despite having written that article and a whole bunch of other stuff that strongly discourages virtualization, I am not a virtualization naysayer. However, it requires expertise - someone sufficiently arrogant to be able to walk through the list of points I've offered and to counter them, or to tackle them understanding the finer points. Plus for bonus points you get to risk your data if your arrogance exceeds your excellence. There are people here who are sufficiently competent, but so far I'm the only one who has actually documented a path that I can explain to another user why it works safely.