Resource icon

"Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

IT_Architect

Cadet
Joined
Mar 20, 2012
Messages
8
Until I read this thread, I was thinking about using a couple FreeNAS VMs as NFS targets. LOL! Our last bare metal server was decommissioned in the fall of 2008, and we have NO desire to ever return to what life was like prior to that. We are currently using Windows Server 2003 VMs in that capacity, and we require transparent compression, so it looks like we'll be giving Mr. Bill his due again to upgrade to Windows Server 2012.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
The problem is that a NAS is not your average server. Your DNS server takes a few gigabytes of virtual disk. Your web server might have a few dozens or even hundreds of gigabytes of data, but that is all easily held on a single virtual disk. FreeNAS is managing potentially dozens of drives with terabytes of nonredundant storage, and to aggregate that into a more reliable whole.

Your typical VM environment is simply not designed to handle this cleanly. VMware is pretty good when it is running on a datastore that is residing on a RAID array. The RAID controller handles failures, and as any VM admin knows, if your RAID itself fails, your ESXi freaks and your VM's are screwed and you are probably going to have to recover from backup.

So if you really wanted, you could probably create some virtual disks on redundant RAID storage and then run FreeNAS on those. The problem is that then people don't want to give FreeNAS any further redundancy. Despite the underlying storage subsystem theoretically having a second copy of the data if there's a corruption, the RAID controller does not allow FreeNAS to access that. ZFS can detect but not correct that. So you then need to build a RAIDz1 (or other redundant ZFS vdev) out of your virtual devices, and then you have that needed redundancy.

But nobody wants to do that because the performance will suck if you do it in RAID5+RAIDz1, and it'll be very expensive if you do it in RAID1+mirrored vdevs, and you have to be so careful about the overall system design. You could do it though. You'd be married to a bunch of potentially difficult assumptions but it works.

But what people want to do is to create JBOD datastores on nonredundant HBA storage and then pass that through as virtual disks or RDM's to their FreeNAS VM. And that's a recipe for failure. Because what's eventually going to happen is that a disk is going to fail (or go fail-ish) and ESXi is going to freak and the virtual disk driver will do nasty things and some of your ZFS components will suddenly stop working. And then your pool is corrupt.

And nobody actually wants to listen to the gory details, all they hear is the answer they want to hear.

There's a good reason that we tell you to run FreeNAS on the bare metal. It's your RELIABLE STORAGE. It can't be reliable if you can't recover from a problem. It can't be reliable if it can't tell when problems are developing with the drives. It can't be reliable if it cannot replace a drive live. Why would you run your RAID array as a VM? It is a system that has legitimate reason to be working with the actual hardware.
 

IT_Architect

Cadet
Joined
Mar 20, 2012
Messages
8
It is a system that has legitimate reason to be working with the actual hardware.
What I was hoping to do is use ZFS like I do UFS2, to take advantage of the compression. I don't need or want FreeBSD to provide any redundancy or healing.

Thanks!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
You can do that, but you'll probably want to turn off ZFS checksums then too because if ZFS cannot recover data it has detected to be in error, it returns zeroes. I assume, reading in between the lines, that your RAID controller will handle redundancy and that you'll only use a single virtual disk. That'll work fine but it loses you lots of the best features of ZFS. (And please don't come back in six months and tell us that you actually created a frankenmonster with half a dozen vdevs and one day you had a disk problem, bad things happened, and now you need help recovering your pool. That help simply isn't available.)
 

IT_Architect

Cadet
Joined
Mar 20, 2012
Messages
8
You can do that, but you'll probably want to turn off ZFS checksums then too...
Sounds right. I don't have much any experience with ZFS, and didn't know I could do that.
I assume, reading in between the lines, that your RAID controller will handle redundancy
Yes, ESXi is on a RAID-10.
...and that you'll only use a single virtual disk.
I was thinking UFS2 for the OS and ZFS+NFS for the data. The two vmdks allows me to backup the OS and not the data. I don't want to backup the backups.
That'll work fine but it loses you lots of the best features of ZFS.
Most of those advantages are native to ESXi.
(And please don't come back in six months and tell us that you actually created a frankenmonster with half a dozen vdevs...)
That won't happen. LOL!
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Most of those advantages are native to ESXi.

ZFS has features that don't exist in any other file system anywhere. That's why ZFS has carved out a new market at the high-end for file servers. That's also why ZFS was created from the ground up to work in the magical ways that it does. It was easier to create a new file system and LVM as a single package than to try to modify existing file system.

So I'm sorry, but either you are expecting things from ESXi that don't exist or you don't realize how awesome ZFS is. Or both.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I was thinking UFS2 for the OS

You do not get a choice of how to install FreeNAS; you will be creating a small virtual disk that emulates the USB stick FreeNAS expects to be installed on.

Most of those advantages are native to ESXi.

I wasn't aware that ESXi offered protection against data corruption, periodic integrity scanning, automatic repair of corrupted data, etc. The only useful ZFS feature ESXi offers in my opinion is Flash Read Cache (L2ARC) but you kinda have to pay a lot to get that.
 

IT_Architect

Cadet
Joined
Mar 20, 2012
Messages
8
You do not get a choice of how to install FreeNAS...
I started thinking that I'd probably have to just install FreeBSD to do what you suggested.
I wasn't aware that ESXi offered protection against data corruption, integrity verification, automatic repair of corrupted data, etc.
It doesn't of course, and you're right, those are good if it has bare metal access. I do have snapshots and can easily clone the whole shebang and start it on other ESXi instances, and it's all free.

You've been most helpful. It's just that to implement them I should be looking at adding ZFS to a couple of my existing FreeBSD VMs.

Thanks!
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
Until I read this thread, I was thinking about using a couple FreeNAS VMs as NFS targets. LOL! Our last bare metal server was decommissioned in the fall of 2008, and we have NO desire to ever return to what life was like prior to that. We are currently using Windows Server 2003 VMs in that capacity, and we require transparent compression, so it looks like we'll be giving Mr. Bill his due again to upgrade to Windows Server 2012.

Don't be afraid to run FreeNAS in a VM, but read this thread and ask questions. The key is you need to give ZFS(be it FreeBSD, Solaris, etc) access to the physical disks and design the disk sub-system to support ESXi running off it. This all will require some old fashion roll up the sleeves and learn something new approach. I will say that WS 2012 will be the cheaper of the 2 options, but FreeNAS is the better of the 2 from the point of safe keeping of your data. Can't beat checksums on every block that get checked periodically automatically, RAID-10 only detects unreadable blocks - it does nothing about disk corruption that still returns a block.
 

IT_Architect

Cadet
Joined
Mar 20, 2012
Messages
8
Don't be afraid to run FreeNAS in a VM, but read this thread and ask questions. The key is you need to give ZFS(be it FreeBSD, Solaris, etc) access to the physical disks.
I don't know how I can do that with ESXi already using all of the RAID-10, and VMs where the data is stored in a .vmdk file. People will probably shoot me, but this is what I have. 90% of our servers are FreeBSD. I decided to install PC-BSD 10 to learn about it, and I was already familiar with the command line end. It was interesting in that I haven't had to install anything yet. I just check a box, and I have it, and that includes, rpc, nfs, etc., and all I needed to do was edit the configs, and it even did a lot of that based on my checkboxes. It installs very fast and easily. It looks like it comes with a VMware tools installed, and the interface is quick, even remotely through tightvnc. The only thing that doesn't work that I can tell is the Windows 7 NFS client where I get a 53 error for some reason, but the local freeBSD client has no issue, so I need to track that down yet. It has a lot of useful app packages that just click-of-a-button install, and maintains what is installed like Windows "Programs and Features", and installs and uninstalls are just that easy. I see it keeps track of security updates as well, and automatically installs them the ones you approve, just like Windows server.

There are a lot of switches to ZFS, and setting it up takes some planning if you use it as designed, but the concepts are very logical to me, and in my case, where I have two pools, tank and nas-pool, and one virtual drive/vmdk in each, so you can't get simpler than that. This arrangement enables me to backup the OS and not the DATA, since I don't want backups of backups. Because of this arrangement, I figured this FreeBSD or PC-BSD looked to be a better route than butchering FreeNAS. If I lose the farm, it doesn't matter. I can restore the OS and simply make a new DATA. It will be interesting to see how well this compares with 2003. The compression seems to be decent. I don't care if it isn't as fast as long as it's reasonable.

After I'm confident of the NFS, I'll prototype on a local ESXi to make sure everything is still good, and then upload it over the VPN to the active area. I'll let you know how it performs and maybe learn a few tweaks from people here. As with ESXi, the VM will only live on a VPN, so there is no direct public access. If it doesn't work out, at least I'll know why I bought 2012 server.
 

pbucher

Contributor
Joined
Oct 15, 2012
Messages
180
I don't know how I can do that with ESXi already using all of the RAID-10, and VMs where the data is stored in a .vmdk file.

That is fine, but no ZFS on any OS with that setup, as least if you want any support around here. The one bonus with FreeNAS is if you backup the config file and you "lose the farm" you just reinstall the OS and upload the config file.

I've experimented with servers where I have no choice but to use the ESXi provided virtual disk and the results haven't been pretty performance wise(though the RAID subsystem on the box was less then optimal) and I certainly wouldn't store anything important that you don't have a backup of on it. But at the end of the day I don't know if it risk wise there a really a big difference between a Win 2012 server with a big vmdk holding it's data drive vs a FreeNAS ZFS vol on a big vmdk. Both VMs are depending on the raid subsystem to keep the data safe, ZFS certainly isn't designed to work well on this setup but outside of killing most of ZFSs redundant projection of your data. At a minimum ZFS at least can tell you if a block got corrupted by the RAID subsystem, bad news is it won't have it's usual redundant block to replace the corrupted one with and life isn't going to be good.


The key to making a good SAN with ZFS & ESXi is to pass through a disk controller and let the OS drive that card and the drives attached to it. It's best done with a LSI card, because that's what 90% of folks who do this use and it's known to work. Not all HBA's work well went passed through, I've gotten an adaptec card to work because I have no choice(rented server in a data center a few hundred miles away), but I wouldn't recommend it simply because 1 box that works isn't nearly enough to prove a solution is viable.
 

IT_Architect

Cadet
Joined
Mar 20, 2012
Messages
8
I've experimented with servers where I have no choice but to use the ESXi provided virtual disk and the results haven't been pretty performance wise...But at the end of the day I don't know if it risk wise there a really a big difference between a Win 2012 server with a big vmdk holding it's data drive vs a FreeNAS ZFS vol on a big vmdk...At a minimum ZFS at least can tell you if a block got corrupted by the RAID subsystem, bad news is it won't have it's usual redundant block to replace the corrupted one with and life isn't going to be good...The key to making a good SAN with ZFS & ESXi is to pass through a disk controller...It's best done with a LSI...I've gotten an adaptec card to work because I have no choice(rented server in a data center a few hundred miles away)...
My backups of these backups are the running servers and multiple copies. In this particular application I have two rented servers a thousand miles away connected gigE, and with the compression, it nearly maxes out their drive's theoreticals. It also drives the CPUs pretty hard with the compression.

We've been using mostly ESXi since 2008. It is antithetical to normal server thinking. Under heavy load, ESXi's vmfs under ufs, will kill FreeBSD and ufs on bare metal on the same hardware. OTOH, I lose ~25% of the CPU performance under extreme load, but not noticeable during normal heavy loads, and cores are cheap. IMO XEN isn't in the game. Hyper-V has been flop after flop, but is now getting our attention. We'll have to play with their latest to see if they have anything.

As CTO for a couple large automotive suppliers, I used about every enterprise controller and drive out there, and I worked with their engineers. Bustec, a DEC mutiny, had a tremendous story, but I their performance never panned out for me in the real world and they had buggy firmware an drivers. Adaptecs have a great IOS, and solid as a rock. They bought a company that made caching controllers who I was buying from because the Adaptecs didn't perform. Time went on and Dell started shipping their own PERC controllers. I was buying Dell's and replacing their controllers with the $3600 caching controllers. One day I decided to test the cheap Dell PERC controller against it so I could gloat and ended up mum. Eventually I learned a no-name, LSI, was behind them. They've since come into their own. The LSIs don't have the standardization and sophistication of the Adaptecs, but they are solid, simple, and fast. I can see why Adaptecs are popular with DCs, but the only Adaptecs I bought afterward were for SCSI peripherals and backup systems. LOL!

The VM is finished, I'm about to test it local ESXi, and then 7zip it up to the production environment to stand on it. If it works, it works. If it doesn't it doesn't. I won't know until I try. Either way it will be interesting. If I have to buy a couple 2012 servers, at least I'll know why I did, and it won't be for lack of researching other options. This is my first time with FreeBSD 10, and ZFS, and I have plenty of FreeBSD VMs that need upgrading, so the experience won't hurt me there either.

Thanks huge for your help!
 
Joined
May 15, 2014
Messages
1
Hi,
I read this thread and other threads of freeNAS forums and am confused. It seems like the instructions are aimed at larger business.
I was planning to use freeNAS for a home backup scheme (described below) and from what is explained here, not only would it not work but would require investment of many hundreds to thousands of dollars.
My original plan was to have two freeNAS virtual machines, one on my home server and one on my parents home server (by server I mean a relatively cheap computer running windows 7 which currently simply has windows share, a printer and some simple programs such as web browser and microsoft office).
The virtual machine would host a windows share over zfs and i will use a standard synchronization program (e.g. bittorrent sync) to sync computers in the house to both shares. I would then use zfs snapshot utility so that I would have previous versions. I was even considering generating the virtual hard drive on an external usb drive (simply because I don't have an extra bay in the servers).
I figured this would cost me some time but no new hardware.

The instruction here practically say this would be bad because a. I am using a VM on top of an OS, b. the computers do not have ECC memory, they have crappy motherboard which probably does not support PCI-passthrough (which won't be relevant anyway because of the OS) and will have limited memory/cpu.

I am trying to figure out what would be the meaning of this. After all, most home NAS solutions (including appliances) do not have such strict requirements. I could have used a simple linux solution and used for example rsync + hard links to do the version or use bittorent sync own versions but the use of snapshots seemed like a better solution.
I do not understand the high requirements.
Specifically: I have had extremely weak servers serve as NAS. For a simple backup (which would hold maybe 1TB of data with changes of <100MB per day, backed up nightly), would the performance hit from such a configuration really be such a problem? I even done movie streaming on such weak machines in the past.
As for the memory corruption. Beyond the fact that I plan to have two copies. I do not understand why would the entire disk go bad. Assuming I do snapshots and the memory was bad causing a bad file to be saved, wouldn't later sync fix that (if only because they see the block written does not match the original block) thereby at most ruining a file in a single version? Even if the case is that the entire file system gets corrupted, wouldn't I be able to know it from the internal consistency of the filesystem (telling me to restore from the other machine instead)?
What am I missing here?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
They aren't "aimed" at larger businesses, but it is expensive to make it work, as well as require expert knowledge to make it work without throwing your data to the proverbial wolves.

For many home users FreeNAS is going to be excessively expensive and is going to be over the top. Storage aficionados will gladly spend the money as they may consider it part of their hobby to be "elite" and protect their precious data.

Hate to break it to you.. but most NAS appliances(your words, not mine) are just that.. appliances. They aren't meant to be run in a VM, and in some cases use proprietary hardware that makes running it in a VM impossible.

As for the RAM questions, I've explained this all in my ECC vs non-ECC thread.

This thread has been locked by someone. It wasn't me, but I think it is appropriate to close this thread as we've provided the information and there's no point in discussing it further. This thread is to discuss the threats and convince you to think twice(or at least try to do it the 'most right' way). We're not really here to give anyone an education in virtualization. If you want that I'm sure VMWare sells classes for that kind of thing.

What we are here to do is explain the serious risk to your data and how you can mitigate some of that risk(but not all of it).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I am trying to figure out what would be the meaning of this. After all, most home NAS solutions (including appliances) do not have such strict requirements. I could have used a simple linux solution and used for example rsync + hard links to do the version or use bittorent sync own versions but the use of snapshots seemed like a better solution.


Good morning and welcome to the FreeNAS Rail Freight Yard. We offer help and assistance with your diesel or electric locomotive. We offer locomotives in the size of eight to 128 ton varieties, capable of pulling trains ranging from a single car up to several hundred.

As you note, our trains have wheels. And yes many of the people here have estates large enough to be able to use a train in their backyard. We would note that the average person parks trucks, cars, or bicycles at home. We will note that having wheels and being parked at a residence does not make our trains into trucks, cars, or bicycles. It does not make the trains magically lighter; the trains are still way heavier than your average commonplace wheeled vehicle. You cannot park a train on your average concrete driveway.

You have noticed that our trains are constrained to working on tracks and you are wondering why they are so heavy. As you note, other vehicles "do not have such strict requirements." You are correct. The difference is that our vehicles are much bigger and simply cannot be compared; when one of our trains happens to meet up with a car:

giphy.gif


The train wins by default. There are good reasons to use a train. It's more likely to get your freight from A to B without an accident. It's able to transport a lot more freight than your average truck. You can add more cars to the train later if it turns out you need to carry more freight later.

Do not let that deter you from driving a truck, car, or bicycle if that's more appropriate to your task. Those are also wheeled vehicles and each serves a legitimate purpose. It sounds to me like you are in need of a bicycle. The bike shop is --->> thataway, and you are certainly welcome to go get one. We won't be offended. Trains aren't for everyone.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I realize I haven't updated this in too-long so I've added some comments about specific PCH's known to be strange or problematic.
 

Gudeen

Cadet
Joined
Mar 13, 2018
Messages
6
Hi,
I currently have a FreeBSD server with some jails, bhyve vms and samba with a 6 disk raidz2 pool as file server. It has been working fine for several years, but I would like to have a web interface to manage users, shares, backup, snapshots etc.
So I was wondering, would it make sense to set up freenas in a bhyve vm using zvols (raidz2 on host, no raid in freenas) on the raidz2?
The other choice would be to completly reinstall the system using freenas as host, placing everything else in bhyve VMs.

In either way, it would be a (non raid)ZFS VM on a (RaidZ2/ZVOL) ZFS host, in such a case what should the zfs configuration look like? e.g. RAM requirements, ARC etc. Probably most RAM should remain with the host for managing the raid/zvol instead of allocating it to guest vm?
 

MikeHartman

Cadet
Joined
May 21, 2018
Messages
4
Older boards utilizing Supermicro X8* or Intel 5500/5600 CPU's and prior are expected to have significant issues, some of which are fairly intermittent, and may not bite you for weeks or months.

Can anyone elaborate on this or point me to more info? What kind of issues? Performance? Trashed data? It sounds like setting up FreeNAS in a VM and verifying it can see and interact with the drives on the passed-through controller isn't sufficient to prove pass-through is working correctly, so are there specific tests that can be run to determine if my particular system is prone to these problems? Or is it always going to boil down to "run it for months and see if it blows up"?

I have a ThinkServer TD230 I'm hoping to use as a home NAS. Until I came across this post I thought it ticked all the boxes - dual Xeons, plenty of threads, 32GB of ECC RAM, LSI 9240-8i, 4 gigabit NICs... But I was hoping to run FreeNAS in a VM under ESXi so I could run related stuff alongside it. Crashplan, Plex, maybe a couple other light things. As soon as I read the above I checked, and yes of course it has a 5500 board. Specifically S5500BC.

Given that this is just for home use occasional performance problems are not a show-stopper. If the referenced issues make it take an hour to boot up sometimes, or cause it to freeze for a couple hours every few weeks, or force me to take a consistent 10% performance hit, that's the kind of thing I can probably live with.

Everything will also be backed up off-site, so while I certainly wouldn't want to be in the situation, if my pools were corrupted I could redo everything locally and then redownload it. It would take quite a while and be pretty damned inconvenient, not the kind of thing I would want to do even once a year, but it does make data loss bugs survivable if they're rare enough.

Am I just out of luck? Or are the expected problems something I can potentially live with or even avoid? I can run FreeNAS on the bare metal if I must, but that means tracking down/putting together another server for ESXi and the other VMs, more rack space, more noise, more heat... Not a problem in a data center but a factor when the rack is right next to my desk.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
Can anyone elaborate on this or point me to more info?
I virtualize FreeNAS on ESXi (6.0 U2) using Dell C2100/FS12-TY Servers. They use Intel 5500/5600 CPUs. That being said, you should know that it is more involved to properly virtualize FreeNAS to ensure things work properly.

I would recommend starting a new thread so others can reply there. ;)
 
Top