Is there a ZFS virtualization problem?

[jixam]

I read all the advice against virtualizing TrueNAS and it never seems to apply to ZFS in particular.

If one wants a file server and is set on making it virtualized – for benefits like easy backup, less hardware, live migration, etc. – TrueNAS will work as well as anything. Right?

Lots of RAM, ECC memory, SMART data, plenty of redundancy – ZFS requires none of that. Each one is a benefit but it will be an improvement no matter which file system is used.

Reordered writes can destroy a ZFS pool, sure. They can corrupt any file system. So modern hypervisors do pass flushes through and ZFS on VMDK works just fine. Right?

 

[jixam]

Apparently I didn't actually read it all. The forum now suggests this thread which seems to agree with the idea that there is no inherent ZFS virtualization problem.

My concern is that people will read the advice and think "oh, I'd better not virtualize TrueNAS then". So they end up with a virtualized Windows/Linux file server that will have all the same issues.

 

[Davvo]

Please read the following resources:

Please do not run FreeNAS in production as a Virtual Machine!

[---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file sharing (small office, documents, scratch space). THIS post is aimed somewhat more at people wanting to use FreeNAS to manage lots of storage space. ----] [---- 2022/11/19: It's...

www.truenas.com

Resource - "Absolutely must virtualize TrueNAS!" ... a guide to not completely losing your data.

[---- 2024/01/16: Still relevant. Virtualization really doesn't change much. Updates made as appropriate. ----] [---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some...

www.truenas.com

 

[Ericloewe]

for benefits like easy backup, less hardware, live migration, etc.

But none of those apply to a NAS. You can't live-migrate dozens of disks and even a handful of disks would be painful beyond any practical application, there's little impact in terms of hardware, and there's nothing on the OS side to backup beyond the config database.

 

[Etorix]

I read all the advice against virtualizing TrueNAS and it never seems to apply to ZFS in particular.

You appear to have read everything backwards. ZFS has a specific issue with virtualisation: ZFS operates atomically between states which are known to be correct. Any intermediate layer (RAID controller and/or hypervisor) interfering with what ZFS considers to be committed to permament storage is a MAJOR concern.

Lots of RAM, ECC memory, SMART data, plenty of redundancy – ZFS requires none of that.

Wrong on all counts but SMART data. ZFS needs RAM, and strives to make most use of it. ZFS assumes that RAM content is valid; an undetected bit flip in RAM is an issue. And ZFS without redundancy is of limited use—"file X is corrupted and cannot be corrected, go and get that tape archive…".
Monitoring SMART data is good practice, but this is not done by ZFS itself.

My concern is that people will read the advice and think "oh, I'd better not virtualize TrueNAS then".

That would actually be a correct conclusion: Better not virtualise ZFS. Especially if you do not fully understand the consequences.

So they end up with a virtualized Windows/Linux file server that will have all the same issues.

Well, unless the hypervisor happens to corrupt a top-level metablock it cannot thrash an ext4 filesystem with the same deadly efficiency it can damage a ZFS pool, and there are repair/recovery tools for non-ZFS systems.
The power of ZFS also means that when things go wrong, they go REALLY wrong.

 

[Ericloewe]

Wrong on all counts but SMART data. ZFS needs RAM, and strives to make most use of it. ZFS assumes that RAM content is valid; an undetected bit flip in RAM is an issue. And ZFS without redundancy is of limited use—"file X is corrupted and cannot be corrected, go and get that tape archive…".
Monitoring SMART data is good practice, but this is not done by ZFS itself.

Little bit of a pessimistic take, since all those are indeed important, but not because ZFS will absolutely die without them. More a matter of it being silly to tow a train with a pickup truck with a Browning machine gun on the bed - not strictly impossible, but there are much better uses for the technical and much better ways of powering the train.

 

[jixam]

Please read the following resources:

I did read those posts. They are the ones with claims like "the hypervisor might be reordering data on the way out (which makes the pool at least temporarily inconsistent)". If that is even valid today, it is not particular to ZFS. It is bad news for any file system.

But none of those apply to a NAS. You can't live-migrate dozens of disks [...]

A virtual NAS can make sense. Maybe I do not have a dozen disks, maybe I just have 1TB or 0.1TB of documents and pictures. Building a dedicated server for that is a bit excessive and requires an additional backup strategy.

Incidentally, though I realize that this is the TrueNAS forum, my question is really about virtualizing ZFS in general and not just NAS applications.

ZFS operates atomically between states which are known to be correct.

Sure, but again: that is not special. All modern file systems (and databases) have this requirement.

Well, unless the hypervisor happens to corrupt a top-level metablock it cannot thrash an ext4 filesystem with the same deadly efficiency it can damage a ZFS pool, and there are repair/recovery tools for non-ZFS systems.

With ZFS I can do a scrub and be fairly sure that my data is safe. With ext4 I might have partial corruption that I will not notice until my backup schedule has wiped out all the good copies. The latter scenario is arguably more catastrophic.

 

[Scharbag]

You really cannot "virtualize" ZFS unless you do something really lame like pass virtual disks into a VM and THEN use ZFS.

If you use a good HBA and pass that through to the VM with VT-d, then the VM has complete native access to all of the HDD hardware. This does eliminate a number of VM features, like vMotion and snapshots, but it will work. I ran my system that way for a very long time (better part of a decade IIRC). It is a great way to consolidate things on a single box. You will have to remember that the HBA disks cannot be used by the host or other VMs so you will need other disks to boot the host and store the TrueNAS VM.

It works, it is fast, it is stable but it needs to be on the proper hardware.

Cheers,

 

[jixam]

The question is not whether the setup should be virtual. The question is: given a virtual disk, is it more lame to use ZFS than to use ext4?

IMHO, ZFS will not be used to its full potential on a virtual disk but it will still be better than ext4.

 

[Patrick M. Hausen]

Due to the copy-on-write nature of ZFS you cannot use a thin provisioned disk. Well, you can, but it will just expand to the full capacity. That might be a thing to consider.

I use TN CORE as my preferred hypervisor so with the virtual disks on ZFS anyway on the host system, I prefer EXT4 or UFS for the guest.

 

[jgreco]

The question is: given a virtual disk, is it more lame to use ZFS than to use ext4?

The answer is "probably yes". The copy-on-write wreaks havoc with thin provisioning, and the usual ZFS rules for space allocation ("don't fill your pool more than X%") mean you overconsume what is quite likely to be very expensive hypervisor-grade storage space.

ZFS will virtualize just fine if you follow certain rules and understand what's going on. Perhaps what you're looking for is more along the lines of

Virtually FreeNAS ... an alternative for those seeking virtualization

[---- 2014/12/24: Note, there is another post describing how to create a full-sized FreeNAS VM directly managing a bunch of disks. THIS post is aimed at people wanting to use FreeNAS to provide a little basic file sharing off of virtual disks on shared VM storage. ----] There's been a lot of...

www.truenas.com

So the answer to your question has been known and posted here for many years... it's just that most of the time, that's not actually what people are looking to accomplish with virtualization. At the time it was written, many virtualization hosts topped out at 32GB per host, and the 8GB memory consumption of FreeNAS combined with other factors made it mostly unattractive.

 

[jixam]

The linked post literally claims that it is insane to use ext4 in a VM (as ext4 violates Key Concept 2 which is an "actual VM rule").

Anyway, I think this is the confirmation I was looking for:

ZFS will virtualize just fine [but ...]

So there is no inherent virtualization problem, just a lot of trade-offs. Thanks.

 

[Davvo]

So there is no inherent virtualization problem, just a lot of trade-offs. Thanks.

Rephrasing this a bit we get:

There is no inherent ZFS problem, just a need to set up things as ZFS requires.

 

[jgreco]

The linked post literally claims that it is insane to use ext4 in a VM (as ext4 violates Key Concept 2 which is an "actual VM rule").

As the author of the linked post, I am reasonably certain it says no such thing. Key Concept 2 discusses the storage system requirements for keeping redundancy available and visible to ZFS. It says nothing about ext4.

ext4 works fine in a VM, of course. I have no idea where your insanity claim comes from.

So there is no inherent virtualization problem, just a lot of trade-offs. Thanks.

This is incorrect. There can be problems with virtualization when it isn't done correctly on hardware that fully supports what is required. Do not expect me to bless virtualization on some godforsaken pile of hardware or using a strategy that violates the guidelines I've outlined numerous times over the years. I've done the hard work for you. It's up to you to take the advice.

 

[HoneyBadger]

Due to the copy-on-write nature of ZFS you cannot use a thin provisioned disk. Well, you can, but it will just expand to the full capacity. That might be a thing to consider.

Well, you can present the ZFS VM with a thin virtual disk that's tagged as supporting discard operations, and periodically zpool trim it if the hypervisor processes the guest-issued SCSI UNMAP command down through the virtual disk to the underlying physical storage ...

ZFS will virtualize just fine [but ...]

This is a big "but" (and I cannot lie) so you have to be aware of the sharp edges that you can expose yourself to. ZFS expects that it isn't being lied to when it requests things like cache flushes to ensure a stable state on disk, and if you have a layer of abstraction that isn't 100% honest with it, you can have "undefined behavior" as a result.

 

[jgreco]

if you have a layer of abstraction that isn't 100% honest with it, you can have "undefined behavior" as a result.

Which is why the preferred and safest method is still to pass the disk controller through to the VM via PCIe passthru, whether that's an LSI IT mode HBA, PCH SCU, SATA AHCI controller, etc. If you rely on VM virtual disks, you add the risk of whatever storage system backs that up. If you have a million dollar enterprise SAN array managed by a SAN guru, you'll probably be fine. If your hypervisor only has an LSI 9260 running a RAID5 and you put a bunch of virtual disks on it, it may seem to work but you've got the recipe for a future screw job (have your resume handy if there's anything important on the array).

The matter of why your NAS should have hardware access to an LSI IT mode HBA or one of the others I mention boils down to the proven track record that they work correctly for all the edge cases. Which is the long way of saying what @HoneyBadger said above.

 

[Scharbag]

I am still lost as to why someone would bother to run TrueNAS on virtual storage - what is the point? Just run Ubuntu and format a big XFS target.

Seems like a very silly thing to do...

 

[Davvo]

I am still lost as to why someone would bother to run TrueNAS on virtual storage - what is the point? Just run Ubuntu and format a big XFS target.

And get tangled with the Linux Kernel? Nay!

 

[jixam]

ZFS expects that it isn't being lied to when it requests things like cache flushes to ensure a stable state on disk, and if you have a layer of abstraction that isn't 100% honest with it, you can have "undefined behavior" as a result.

Several responses make it sound like ZFS is picky about flushing – but ext4/XFS/SQL Server/whatever expect the same things. As the risk has already been taken by using virtualization in the first place, I don't see how adding ZFS on top is an extra risk.

If you rely on VM virtual disks, you add the risk of whatever storage system backs that up.

If you rely on pass–through, you add SPOF hardware and additional backup procedures. Those are also risks.

I am still lost as to why someone would bother to run TrueNAS on virtual storage - what is the point? Just run Ubuntu and format a big XFS target.

It is not a bother. TrueNAS is easy to install and it has a decent UI for configuring file sharing. For example, Windows shares with AD integration and shadow copies ("Previous Versions") via ZFS snapshots is fairly easy to set up.

 

[jgreco]

I don't see how adding ZFS on top is an extra risk.

Biggest one? Because ZFS has no "fsck"/"chkdsk" type tools to fix your pool when you manage to corrupt your pool. ZFS is heavily reliant on being able to trust your storage to commit blocks in the manner which has been arranged by ZFS. If other stuff starts going on under the hood, you rapidly develop a nonzero chance of (for ex.) a RAID controller's "write cache" caching data and then losing it. No write reordering. No elevator sort. Etc.

If you rely on pass–through, you add SPOF hardware and additional backup procedures. Those are also risks.

Sure, but you can easily replicate data to a second (on-site standby) machine, and a third (off-site disaster recovery) machine. Nothing's foolproof. If you don't believe me, allow me to introduce you to my ten pound sledgehammer... but it makes sense, to me at least, to avoid unnecessary SPOF's and provide an appropriate environment to run the thing.

I am still lost as to why someone would bother to run TrueNAS on virtual storage - what is the point? Just run Ubuntu and format a big XFS target.

@jixam posted a reasonable answer:

decent UI for configuring file sharing. For example, Windows shares with AD integration and shadow copies ("Previous Versions") via ZFS snapshots is fairly easy to set up.

but for a little more insight:

During the '90's and '00's I got tired of Samba rendering the office fileserver useless after an "upgrade"; the FreeBSD Samba port was basically useless because it would get rewritten every year or two, needing new dependencies, OS upgrades, research into which options were needed, and often nearly total reconfiguration. At a certain point you just want stuff to work reliably and with a consistent usability. Samba has NEVER been any of that, even today on TrueNAS, I occasionaly hear about eye-crossing issues, and iXsystems has @anodos (a Samba guru) on staff just to help keep the thing on the rails. I would rather benefit from the vast knowledge of @anodos and the iXsystems team to maintain my Samba server, and I've mostly gotten my wish, because in the last 12 years with FreeNAS, most of my "Samba maintenance" has been "install new version of FreeNAS, get on with life" rather than me trying to play SMB guru and build or upgrade my own. This is really one of the biggest reasons I use FreeNAS, and I know I have been saved endless amounts of anger and frustration, which I can now spend working on other projects.