BUILD YA - X10SL7 (for FreeNas + VM Lab)

[Charlie Fineman]

Getting ready to replace an older Proliant running VortexBox. Been pretty happy with the appliancy feel of VortexBox but less keen on the storage options they've provided out of the box. I'd also like to place some additional VMs on this box... some utilities (and windows 'desktop', perhaps an MB3 server, etc) plus some experimental VMs (I'm a software guy and I frequently use VMWorkstation to create sandboxes for playing around with different stacks).

FreeNAS/ZFS + ESXi feels like it might be the right choice for me. I've read some of the caveats about using FreeNAS on top of ESXi but it looks like folks have successfully done this so I'm interested in giving it a go. I'll poke around more (not to mention experiment before laying down the final config) to see what pitfalls folks have encountered but if anyone feels like warning me off, now would be a good time!

Having said that, this is a hardware forum so let's get on to the sand-based aspects of this.... primarily I want to be comfortable this box will be able to run a small number of VMs (<10) comfortably without unduly harming the performance of the FreeNAS VM. Specifically it's ability to

• run/serve ZFS
• plex video to a couple clients
• stream music
• media ripping

I will be surprised if the answer were "no" but before I sink the money I want to be sure I have appropriate expectations.

• SuperMicro MBD-X10SL7-F-O
• e3-12X1-v3 (where X=3,4, or 7... not clear the 10% boost from 3->7 is with the 30% increase in $!)
• 32GB ECC (might scale back to 16GB for now and see if I need more... will probably stick with ECC since I'm not that $ conscious and some portion of the data I'll be managing is 'precious')
• For the time being will be reusing a
  • big-arsed thermaltake case
  • Either a 400W or 600W PSU (2 or 3 years old but I just bench-tested them and they seem fine)
  • A bunch of random disks (will look into this separately... one concern I have with ZFS is pool management esp for heterogeneously sized disks... but that's a topic for another post)
• Eventually (maybe sooner rather than later given my parity needs) will add on a SAS extender (the RES2SV240 one of the posters referenced looks pretty tasty)

Look forward to your feedback/warnings as I contemplate my move into the FreeNAS neighborhood.

 

[mjws00]

Heh. This is a scenario I understand well. You are now evil and and an outcast to be shunned. ;) You have to survive the "warnings off" and be prepared for the fact that it's tough to get experienced guys to mention vm's at all. It gets botched badly, then there is crying over lost data. The challenge I see is that the variables are so great that it is almost impossible to determine if it was user error/panic that induced loss, a glitch, a driver problem. The folks with the experience to troubleshoot it, can't typically duplicate the issue and lack the time to go "deep". Burned users even if they claim "expert" status often document poorly and provide very little real data to address fixing the problems. They don't test thoroughly and move "real" data early.

I ain't jgreco, he has bigger toys. But I'll tell you that following every best practice I could find. Reading every relevant post. Doing what you want is working perfectly on my hardware. It is also 30 seconds to install a boot usb, and run or recover on baremetal. In addition the pool drives work perfectly on any other box. I've done every nasty thing I can to the box, pulling drives yanking cables, cutting power. It is trivial to recover ZFS. I can find no better way beyond NOT doing it. Frankly the esxi box is more important in my environment than FreeNAS. There are many ways to skin the storage cat, this one is just nice.

Read jgrecos sticky on "absolutely must virtualize" and understand the "do not virtualize" thread. On that hardware vt-d should work perfectly. Go slow, read lots. You have to promise never to bitch about data loss. Make sure you have backups that can't be corrupted. Test like a mofo... walk on eggshells.

Here be dragons. Good luck.

 

[TXAG26]

Getting esxi setup the right way with FreeNAS is challenging and everything has to work perfectly or the wheels fall off very quickly. I'd say keep your current box and dedicate it to one or the other roles (esxi OR FreeNAS).

On esxi, 32GB ECC ram is not enough for FreeNAS and other VM's. I'm using ALL of the ram on my box and hindsight 20:20, probably should have gone with a higher classed Xeon and Intel 600 chipset series board with support for at least 64gb REG. ECC. Just my $0.02 though.

 

[Charlie Fineman]

Thanks for the thoughts.

I started reading through some of the recommended threads and the advice there seems sensible. To paraphrase: keep the configuration as closely aligned to the physical machine as possible. Dedicate HBAs via passthru to the FreeNAS VM seems to be the primary point being made. Also the bit about setting things up on bare metal before bring ESXi into the picture seems like solid advice. In fact both of these points (which, hopefully, I correct captured/conveyed the spirit of) ring true regardless of whether I use FreeNAS or some other solution.

The more I read about it however, I think it's probably not the right critter for me.... primarily because of the way ZFS wants to deal with disk pools. I've got a fair-size collection of disks that I've gathered over the past ~5 years. They seem to be running well but there are of various sizes and I'm loath to get rid of them. I may need to look elsewhere :-S

 

[jgreco]

Looking elsewhere isn't the solution really. The big risks with large scale virtualized storage are actually similar for other NAS products when virtualized, it is just that we're not all that into enabling risky behaviours without at least the courtesy of letting you know.