ZFS import runs out of swap & hangs OS

[brentaar]

Several days ago, the ZFS disk decided to stop responding and I've been trying to import it since. I tried the Auto-Import on GUI and it gave me an error.
Reading various posts I've learned quite a lot, but nothing has worked for long. I'm relatively new to ZFS and could use some help from others with more experience.

using tmux

• zpool import says that the volume is online and that it can be imported
  
  Code:

  mount -uw /mnt; zpool import -R /mnt ${zvol id#}

• gstat shows various things are happening on the disk
• top shows that zpool is doing vary little CPU% wise

This all runs for a while and then it runs out of swap that causes pretty much everything to stop responding.
I've gotten access to the data once and then it stopped responding.

What are the steps needed to diagnose and then fix this?

Events that led up to the issue:

• took snap-shot of volume
• realized my entire volume was set to deduplicate with verify, and turned that off
• removed SSD ZIL
• exported zfs volume to import it uder a new name
• took the disk that was the ZIL
• started scrub from GUI
• stopped scrub from CLI
• started the deletion of snap-shot, pressed cancel (i think) and then closed web browser. And this is the point things went wrong

Running FreeNAS-8.3.1-BETA2-x64

 

[cyberjock]

Before you thank me(or hate me) for replying I'd appreciate it if you could provide some information. You're the first person to have experienced what I nicknamed "lockout" of your data thanks to dedup. Could you post your hardware specs and approximately how much data you had in your zpool?

So here's what you want to see(or not want to see). This is the dedup paragraph from the manual:

Deduplication: the process of eliminating duplicate copies of data in order to save space. Once deduplicaton occurs, it can improve ZFS performance as less data is written and stored. However, the process of deduplicating the data is RAM intensive and a general rule of thumb is 5 GB RAM per TB of storage to be deduplicated. In most cases, enabling compression will provide comparable performance. In FreeNAS® 8.3, deduplication can be enabled at the dataset level and there is no way to undedup data once it is deduplicated: switching deduplication off has NO AFFECT on existing data. The more data you write to a deduplicated dataset, the more RAM it requires, and there is no upper bound on this. When the system starts storing the DDTs (dedup tables) on disk because they no longer fit into RAM, performance craters. Furthermore, importing an unclean pool can require between 3-5 GB of RAM per TB of deduped data, and if the system doesn't have the needed RAM it will panic, with the only solution being to add more RAM or to recreate the pool. Think carefully before enabling dedup!

And from the release notes:

ZFS v28 includes deduplication, which can be enabled at the dataset level. The more data you write to a deduplicated volume the more memory it requires, and there is no upper bound on this. When the system starts storing the dedup tables on disk because they no longer fit in RAM, performance craters. There is no way to undedup data once it is deduplicated, simply switching dedup off has NO AFFECT on the existing data. Furthermore, importing an unclean pool can require between 3-5GB of RAM per TB of deduped data, and if the system doesn't have the needed RAM it will panic, with the only solution being adding more RAM or recreating the pool. Think carefully before enabling dedup! Then after thinking about it use compression instead.

If you add an L2ARC you may be able to get access to your data. Keyword: may. You will almost certainly be forced to upgrade your RAM. If you follow the thumbrule of 5GB per 1TB of data you might be able to access your data. Remember that there is no upper limit so you may not even be able to get access despite meeting that 5GB per TB of data. I'd try maxing out the RAM on your motherboard before you try adding an L2ARC. The L2ARC is only helpful in some ways and is less likely to get you out of your pinch than having more RAM.

You're the first person to be locked out of your data in this fashion. Honestly, I'm surprised it hasn't happened sooner. Too many people just don't heed warnings when they start playing with FreeNAS and don't realize how nasty things can get if you haven't taken the time to do your homework.

I'd appreciate it if you post back and let us know how this goes or even if you get access to your data again. We were just discussing this a few weeks ago at http://forums.freenas.org/showthread.php?10673-Anyone-using-Dedup

Good luck and I hope you are lucky enough to get your data back.

 

[JaimieV]

Cyberjock, you might be firing at the wrong angle: the setup was working okay *with* dedup enabled, the problems happened after dedup was turned off.

Now, I've got no idea what turning off dedup actually does to the data on-disk. If ZFS does a trawl and 'pads out' all the files with the removed dupe data, then the subsequent actions of removing the ZIL and exporting the pool might have been terrifically badly chosen. The timeline Brentaar posts looks like a sure-fire recipe for killing off a pool to me, out of sheer confusion... and for making people wish there was a fsck_zfs.

 

[brentaar]

Thanks for responding.

The hardware spec:

• Platform: ASUS AMD Zacate E350/Hudson M1/SATA3&USB3.0/A&V&GbE/Micro ATX Motherboard s E35M1-M PRO
• Memory: 7773MB (maxed out for the motherboard
• OS is on a jump drive (Build: FreeNAS-8.3.1-BETA2-x64 (r13015))
• 2 * WD Red 3 TB in a stripe zpool
• 60GB OCZ ssd as L2ARC (was ZIL)

The amount of data is 3TB to 4TB. One to two of which was created after depulication was turned off.

I've got a laptop with about 16GB and better 64-bit processor. I've held off on using that, because I don't know what using SATA for the SSD, eSATA for one WD Red and SATA doc connected through USB3 would do. If there was a way i could let the zpool know that everything was still there I'd setup that mess. I have another zpool that could hold enough of the data to allow me to wipe the current disks, though as usual I'd prefer just to fix this mess.

Again thank you for responding,
brentaar\

 

[cyberjock]

Cyberjock, you might be firing at the wrong angle: the setup was working okay *with* dedup enabled, the problems happened after dedup was turned off.

Now, I've got no idea what turning off dedup actually does to the data on-disk. If ZFS does a trawl and 'pads out' all the files with the removed dupe data, then the subsequent actions of removing the ZIL and exporting the pool might have been terrifically badly chosen. The timeline Brentaar posts looks like a sure-fire recipe for killing off a pool to me, out of sheer confusion... and for making people wish there was a fsck_zfs.

From the manual(I quoted it above):

In FreeNAS® 8.3, deduplication can be enabled at the dataset level and there is no way to undedup data once it is deduplicated: switching deduplication off has NO AFFECT on existing data.

The problem is the only way to "undedup" blocks after they've been deduped is to turn off dedup, move the files off of the zpool, then back.

This is why enabling dedup is a recipe for disaster. Once the DDT table has gotten big enough that you can't use the ZPOOL with your installed RAM you are in hot water. At that point you've realized you are in error, but you are stuck with the deduped files forever unless you undedup them by moving them off the zpool and back after you disable dedup.

Dedup works just like compression. Once enabled any files that use that feature will forever need that feature until that file is deleted.

Brentaar-I would look at trying to move your zpool to a machine with at least 16GB of RAM. I know that's not close to what you need with 5GB of RAM per TB of data, but you don't have much choice as far as I can see.

 

[brentaar]

Will ZFS be smart enough to find the disks on the different machine? And in a different location?
Or can it be told where the new disks are?

 

[warri]

ZFS should be able to figure out that there is a pool available for import. In FreeNAS, AFAIK the disks are not bound to certain SATA ports, but identified by some unique IDs. Usually a 'zpool import' should get you running again on another system.

 

[cyberjock]

If you just connect the drives and the controller's drivers are part of FreeNAS the drives should just work, ZFS should auto-find the drives, and your ZFS pool should auto-mount. In your case it likely won't auto-mount, but you can fix that with the "auto-import" options in the GUI or from CLI as zpool import.

Brentaar, another comment I figured I'd add. If you DO get your zpool to work by doing an import on a machine with more RAM you will have to go through some process to dedup the data before you can move the drives back to the old machine with 8GB of RAM. So you should start figuring out your plan of attack. My recommendation would be to plug in a big drive to your machine, make a zpool of 1 disk and copy folder by folder to the big drive, delete the data on the zpool that you've copied, then copy it back. That's the only way to remove the dedup from the files. Otherwise if you have a backup(I'm thinking not because you are going to such lengths to get to the data) then you should just delete the zpool and recover from backup.

 

[brentaar]

After connecting everything, I plan to copy everything i can off the zpool and then start fresh without dedupe.

Hopefully there enough space on my desktops zpool to accomedate all of the transfers I want to make (there is a lot of duplicate data, thus the uncertainty).

There is an average of 1GB free of the 16GB total availabe and the transfers are progressing at a decent pace.

To me my data is important. And knowing it was there, but inaccessible gave me enough of a reason to try to figure it out.


Not related to the problem

What the setup looks currently:

• One WD Red is in the external dock
• One WD Red is being powered by the desktop using an adapterand is using the desktop's SATA female to eSATA male to an eSATA cable that is connected to the laptop sitting on top of the running destop
• the L2ARC is in the laptop taking one of the HD bays
• the OS is on a jump drive
• All of this being ran off of a System76 Gazelle

(I'm very surprised I got it all running last night after drinking.)

 

[JaimieV]

From the manual(I quoted it above):

Derp! Thanks. :o

 

[cyberjock]

Derp! Thanks. :o

No prob. I figured you were tired or something.

 

[brentaar]

And just when I think i'm on the home stretch, things are pushed too far. It was working for almost 24hrs, and now the zpool isn't being recognized by zpool import.

About the only thing i have to go on is that shutting down the system gives this error

I'll have to troubleshoot this after work.

 

[cyberjock]

You may need more than 16GB of RAM. Perhaps a drive is failing in addition to your other problems.

Now do you understand why dedup is borderline irresponsible? LOL. I'm thinking you'll never ever make that mistake again. Dedup has a VERY limited subset of uses, and an even smaller subset of machines that it will work exceedingly well on. If I were a betting man I'd bet my life savings you don't fall into either of those ;)

 

[brentaar]

Quick update...

Almost of the data has been recovered; slow process due to having a smaller, slower zpool, and manually deduplicating the data on the desktop.

yay progress, and hopefully a more full explanation to come this weekend.

 

[brentaar]

Solved

The issue arose from having deduplicate on, and trying to delete a dataset snap-shot, but stopping it in the middle.
The system would run out of memory and would become essentially non-responsive; some of the keyboard commands would work.
Moving to computer with more RAM gave enough up time to copy files off of the zpool and rm them (which i hoped would free up more RAM, less data smaller dedupe table). The process could not be parallelized or the OS would hang, so setting things and wating for them to complete was necessary.

The more RAM the better. Lack of RAM and deduplication is what got me into that mess.

To make things stable enough to get most of the data off the pool was as simple as having more RAM and using the cli. I was surprised how well FreeNAS handled being moved to a new system.

Commands that were useful for getting things going were

Code:

top
tmux
gstat
mount -uw /
zpool import
zpool import -R /mnt ${pool id}

Deduplication is pretty much irresponible for most users, and it should probably be an advanced feature that needs to be turned on through the cli. Though watching all of the duplicated data go back and forth i still wish it would be a tool i could use.

The system has been reconstruted, the data has been put back in place, and it is running with more free RAM. After this rather horrible experience, almost lost a decades worht of my wife's pictures, I'm still going to stick with FreeNAS for the time being. My data was still on the pool, just difficult to get to.

 

[JaimieV]

Good news - thanks for following up with the end of the story, always appreciated.

Your next task is to setup a backup routine!

 

[brentaar]

For the FreeNAS box or other computers?
The FreeNAS box is supposed to be the backup.

 

[gpsguy]

At a minimum, I'd back up "almost lost a decades worth of my wife's pictures" :)

For the FreeNAS box or other computers?
The FreeNAS box is supposed to be the backup.

 

[JaimieV]

If there's anything on the FreeNAS box that is unique to it, it needs backing up.

If the FreeNAS box is only backups, then you didn't 'almost lose' anything...!