ZFS Has Failed you

[jpaetzel]

This post is howto for what to do when ZFS has failed you. Generally you'll be interested in this when FreeNAS is panicing at boot. You've discovered that the reason the system is panicing is during the ZFS pool import.

Step 1. Don't panic.

Step 2.

Reboot FreeNAS, and select the Escape to Loader option in the boot menu.



In the attached screenshot it's option 6.

enter the following commands:

set vfs.zfs.recover=1
set vfs.zfs.debug=1
boot -s

This will get you to the following screen, go ahead and press enter to start /bin/sh



# sh /etc/rc.initdiskless

Now you are ready to start trying to import your pools.

First pass is just trying a straight import:

# zpool import -R /mnt poolname

If this works, great. Backup your data and life goes on.

If you get a panic then repeat the above steps, but try:

# zpool import -o readonly=on -R /mnt poolname

If this works your pool imported read only. Backup your data and life goes on.

As a final desperation move, or after you've managed to import the pool readonly and copy everything off you can try:

# zpool import -R /mnt -FX poolname

The above command has a good chance of grenading your pool, but it also has a chance of importing it.


ZFS is very robust, but non ECC RAM and desktop drives can cause problems that it can't fix, especially in larger pool configurations.

 

[Sir.Robin]

Always good to have a plan-B the day Mr. Panic knocks on your door. Thanx :)

 

[ThomasH]

Hi there,

Thanks for the post, I have experienced exactly this problem! I followed your excellent guide and managed to mount the zpool in read-only mount mode. But... I don't know how to back it up from there. My Freenas machine is exactly that: a NAS. I only ever accessed the data through CIFS. Which I configured from the GUI. So, after mounting my zpool in read-only mode, I typed 'exit' and the NAS booted to Freenas. But once I was there, I couldn't see the zpool I just mounted, so I cannot share it to the network.... Help? :) Thanks in advance!

-update-
When importing the zpool in read-only mode from the shell thingy in the Freenas GUI, something weird happens: I can see the volume in the 'reporting' graphs, but I cannot see the zpool listed under my 'volumes' in the Storage area:

If I get this fixed, I'll be able to back the files up :)

-update 2-
Using SSH file transfer (SCP) to copy the files to my Windows machine(s). Slow but at least I'm securing my data! :D

 

[thinker7]

Hi,
Where is Escape to Loader in 9.3?
Thanks

 

[scootmandoo]

+1

I have the same problem. in v9.3 I no longer see Escape to Loader. I did get to a command line but without the kernel loaded I couldn't do much. Is there an updated list of these steps for v9+?

Thanks!

 

[technopop]

V9.3 procedure that just worked for me.

When GRUB loads up, hit the letter "e" on the keyboard to edit.


Go down to the SINGLE USER MODE section and insert two lines. Type in the following into the two lines.


set kFreeBSD.vfs.zfs.recover=1
set kFreeBSD.vfs.zfs.debug=1



Look on the bottom of the screen for instructions. You want to boot by hitting CTRL-X or F10.

You'll be into the grub sub menu to see Normal boot up, single user mode and Verbose boot up. Arrow down to Pick single user.


Continue with the instructions from the original post at

"This will get you to the following screen, go ahead and press enter to start /bin/sh"

etc...

 

[brandon3055]

This post just saved my pool! Thankyou

I ran into an issue where the system would freeze on boot at importing pools. Before i found this post i tried reinstalling and updating to FreeNAS 11 but when i tried to import the pool the system would just freeze. In the end i was able to use the readonly command from this thread with the -f flag to mount the pool and i am currently in the process of transferring my files via FTP.
This did not require single user mode due to the fact that i had already reinstalled the os and was able to access the console.