Is ZFS as frail as posts make it out to be?

[LordKitsuna]

Everywhere i read people make it sound like if ZFS loses power even once everything is fucked forver and dead and ill lose everything. Its like i need a enterprise grade power backup to be safe. However maybe they just dont have the right hardware? here is my hardware, i do sometimes get power outages how likel am i to just have everything go away?

2x dual core Xeon procs
28GB ECC DDR2 RAM
2x 4-Port SATA controllers
8x 3TB HGST HDD
AFAIK that is all the relevant hardware, freenas is installed on a 16GB flashdrive. Are all these warnings just hugely over exaggerated?

 

[nightshade00013]

The problem comes in when the system is writing to a drive. ZFS holds the data in RAM and then writes in as large a continuous space as possible. Power goes off data goes up like a fart in a windstorm.

I had a test system up and running without any battery backup and it lost power a couple times and there was no major problem. The thing is that you never really know when it will be writing something important. Not to mention that just letting something get power off without being prepared for it is not a good idea in general when it comes to data integrity.

As far as power outages a simple battery backup can solve any issues, set it up and plug it in then make sure the monitor is setup correctly. Power goes off it goes on battery and then gracefully shuts down. No data loss no worry.

 

[danb35]

If anything, ZFS's copy-on-write design means that it's safer than most other filesystems in the event of a power failure. Data not yet written would be lost, of course, but data on disk should be pretty safe. However, "safer than most other filesystems" is not the same as "perfectly safe." If power dies while writing the uberblock, there could indeed be harm to your data.

People here assume that you're using FreeNAS because you care about your data. If you care about your data, you'll want to use reasonable steps to protect it. Those steps include a redundant storage pool, server-grade hardware, ECC RAM, and a decent UPS. Even though the risk of data loss due to a power outage may be low, it's easy to mitigate.

 

[DrKK]

@LordKitsuna Right, I think danb35 has hit it on the head here. We assume that what has brought people to the FreeNAS universe is a desire for full data protection and integrity; after all, FreeNAS represents a learning curve for most people and an expense (in new hardware) for most people.

As danb35 says, ZFS is not brittle at all, quite the opposite. But, there is a risk of data loss if power is lost, and it's just simply an overall bad idea to lose power on a server suddenly. I use cheap APC/Cyberpower UPS's (consumer grade)---the $50 ones---to perfectly excellent results in FreeNAS.

 

[SirMaster]

As others have stated, power loss on any file server is not a great situation. However due to the atomic nature of transactions in ZFS it should be more resilient to power loss than other file-systems.

You could stand to lose up to about 5 seconds of writes though due to the way ZFS caches writes, but ZFS itself is far from frail and has been designed from the ground up to be extremely robust and it takes many extra steps and measures to protect your data all along the way. I can't really think of a more robust file-system.

 

[LordKitsuna]

Alright, thanks guys. I will look into a cheap UPS to keep things going. It just seemed odd to me my desktop with its plain ol EXT4 never had issues with the power losses and everyone made it seem as tho one power loss would result in some unrecoverable state. I was like wait shouldnt the atomic nature make this insanely less likely. Thanks for the info guys.

 

[SirMaster]

Alright, thanks guys. I will look into a cheap UPS to keep things going. It just seemed odd to me my desktop with its plain ol EXT4 never had issues with the power losses and everyone made it seem as tho one power loss would result in some unrecoverable state. I was like wait shouldnt the atomic nature make this insanely less likely. Thanks for the info guys.

I've done plenty of power loss testing on my test ZFS box and have never experienced data corruption. Not that it can't happen, but no, it's certainty far, far from common or likely to happen.

I think that you also have to keep in mind that a dedicated file server is assumed to have far more load and data activity going on than your EXT4 desktop and thus the opportunity itself for corruption from power loss would be greater.

You really need to compare a busy EXT4 fileserver to a busy ZFS fileserver to accurately compare the frailty when doing testing and comparisons.

 

[jgreco]

Sure, ZFS is "frail" but most other things you'd be likely to use is even more frail.

We assume you care about your data, and to that end, treating your filer like you care is an integral part of the data protection strategy. When you start randomly dropping the filer on the ground, baking it in a closet, or randomly de-energizing it, you're increasing the chance for hardware failure.

 

[LordKitsuna]

You really need to compare a busy EXT4 fileserver to a busy ZFS fileserver to accurately compare the frailty when doing testing and comparisons.

Fair enough, the NAS is basically just taking over for my desktop. I mostly just mass store media and occasionally stream it via plex. My use is write once read many.

We assume you care about your data.

I do i promise! its just that my wallet cant care as much as i do right now :<

 

[jgreco]

Yes, it costs a little. On one hand, you can have a very cheap desktop SoC-style NAS unit (QNAP, Synology, etc) with several disks in there in a stripe, which gives you about as much space as you can ask for at a low cost, but any disk failure results in disaster. At the other end, you can have a big filer like a NetApp, which will cost you four arms and three legs for a modest amount of storage, but which won't fail, and NetApp will periodically ship you hard drives to insert into the unit because they've determined that a disk is going wonky and they want to proactively deal with that.

ZFS is much closer to the NetApp solution in terms of reliability, scalability, and performance, while being a lot closer to the QNAP/Synology unit in terms of price and space. You can absolutely "go cheap" on a ZFS platform, but what you end up doing is compromising away some of the reliability for a cheaper price. The forum encourages that users do some prudent things such as using a decent mainboard in a cool-running chassis, ECC RAM, RAIDZ2, a UPS, SMART monitoring, etc. You do what's suggested and your chances of the filer storing your data successfully goes up dramatically, for what turns out to be very modest increases in cost.

 

[cyberjock]

I think that it is better to compare ext4/ntfs/whatever to ZFS in terms of "brittleness" as follows:

1. Until ZFS, there was no definitive way to prove/disprove if/when corruption happened on the fly.

How many times have you rebooted a desktop/laptop/server and then things were totally jacked up and you made up some excuse with little or no actual evidence as to what happened. "The RAID controller locked up" or "The system was unstable" or "maybe it was a momentary power spike"? Every had done it. We make up some justification based on little or no actual evidence and go on our merry way. If it happens every week, then we actually start looking for the real cause and fix it. Otherwise we run chkdsk or fsck and life goes on.

For the first time in our IT lives, we have a file system that is supposed to be "incorruptible". Of course, that assumes you have sufficient redundancy to recover from any corruption that exists for virtually any reason (and excluding things such as CPU errors and RAM errors).

So we went from a situation where we ran a chkdsk or fsck and basically "got what we got" to a file system that tells us flat out what is wrong (if anything). You've never been able to get that kind of information before, so now it looks like ZFS is terrible and ext4/ntfs is awesome. The reality is that your expectations for ext4 and ntfs have been expected to be so low for your entire IT career that the fact that you can suddenly find out exactly what is wrong in just a few CLI commands is incomprehensible and hard to quantify in terms of "how much better is it?" and our brains are instead assuming that ZFS must suck. The reality is that everything else has sucked until ZFS and now that we are around to see how awesome ZFS is we don't believe how amazing it actually is.

 

[russnas]

I've had my unit lose power a few times while idling,only the USB stick got corrupted just use a new stick and restore ssettings and check over with offline backup,no issues so far.

I was gonna say hardware is quite resilient but I've had good luck with data and drives,

wouldnt want to lose power while writing to disc, if cifs share is read only should be fine.

 

[Z300M]

Fair enough, the NAS is basically just taking over for my desktop. I mostly just mass store media and occasionally stream it via plex. My use is write once read many.



I do i promise! its just that my wallet cant care as much as i do right now :<

If you have a Costco membership and a store near you, you may find a CyberPower UPS for $100 -- I think it's a 1300W model. I don't see that model at costco.com, but our local warehouse had them a few months back.

 

[David Dyer-Bennet]

There *shouldn't* be a risk of data loss on power failure, and there isn't in the Solaris version of ZFS. Has FreeBSD managed to break *that*? That's really horrible.

It's a *transactional* file system. If an update is partially written due to power failure, it should roll back to the last consistent state when power is restored. This all works on Solaris.

 

[hugovsky]

If an update is partially written due to power failure,

I would call that data loss. What do you call it?

 

[jgreco]

There *shouldn't* be a risk of data loss on power failure, and there isn't in the Solaris version of ZFS. Has FreeBSD managed to break *that*? That's really horrible.

It's a *transactional* file system. If an update is partially written due to power failure, it should roll back to the last consistent state when power is restored. This all works on Solaris.

It all theoretically works on FreeBSD as well, but the reality is a lot more complicated. Sun had the ability to certify specific hardware. If I create Grinchy's Spinning Platter Of Doom Storage Device which caches several gigs of writes without any strategy for persistence, and hook a dozen of them up to a Sun box, then I've managed to break that "guarantee" on Solaris as well. The difference is that your Sun FE will look at that and say "but we never certified that hardware as compatible, your system design is broken, oh, and by the way, I just voided out your hardware warranty."

With generic PC hardware, there is no field engineer watching what the chumps^Wusers do with their machines. This means that it is easier for chumps^Wusers to make mistakes like that. This doesn't mean that ZFS is broken, or that ZFS-on-FreeBSD is broken, but rather that the chumps^Wusers are broken. Classic PEBCAK.

If you then take some time to look very carefully at what I've been advocating here in the forum for many years, you'll notice that there's this creepy thing going on where I suggest very specific things to do in order to build a successful NAS. For example, server-grade boards, ECC memory, Intel or LSI HBA SATA/SAS ports. I push people to avoid huge swaths of the "stupid" and "junk" prevalent in the PC world. Why? Because it gets us closer and closer to that highly reliable computing platform model that Sun assumed was present when they created ZFS. My ideal FreeNAS box more closely resembles a Sun server than it does your desktop PC.

Also note that your Sun FE would get very frowny faced if you plugged your Sun right into the wall. A UPS is *always* a good idea.

Also note that your Sun FE would probably look on in horror if you pulled the cord out of a running Sun. Power loss is a bad thing.

 

[Bidule0hm]

Also note that your Sun FE would get very frowny faced if you plugged your Sun right into the wall. A UPS is *always* a good idea.

Also note that your Sun FE would probably look on in horror if you pulled the cord out of a running Sun. Power loss is a bad thing.

But... it's certified...

 

[nightshade00013]

But... it's certified...

And costs how much to have sitting there with it's nice certified label. I have no clue but I can bet it's a good amount more than FreeNAS would be.

 

[jgreco]

But... it's certified...

Maybe not after the Sun FE watches you pull the plug and voids your warranty

Geez, I sound so cynical.

 

[jgreco]

And costs how much to have sitting there with it's nice certified label. I have no clue but I can bet it's a good amount more than FreeNAS would be.

Well, that was actually what caused me to ditch Sun for "my" stuff way back in the Sun 3 era and move to FreeBSD. The problem was, you could still get good work doing systems hacking with Solaris, so I suffered under their interpretation of SVR4 for many years after.

I've still got some IPC's, IPX's, a SPARCstation 5, and some Sun 3/50's and 3/60's around for whenever I feel like retrocomputing.