Panic: Solaris(panic): zfs: allocating allocated segment

[Doug183]

Ok, I am trying to breath deeply. Please also understand that I know a lot, but not an expert by any means. So as many steps as you can describe without skipping much will be very appreciated.

I have two Raidz2 pools of 10 x 3 gig drives . So I have 22 tb on each array or so. One is near full like 18TB and the other is near empty. They have been working fine until up to yesterday. Also, I have no snapshots or duplication stuff going on. I am connecting a OSX 10.7.4 and Windows 7 machine via NFS.

I do not know what is related here so I am going to give you the list of events.

1) Went to copy a file from the Mac to zfs pool "Storage2" on the server.
2) The copy paused for awhile and then the copy on the Mac timed out. And the "Storage2" went offline from my local computer.
3) Without rebooting, I copied the file to zfs pool "Storage1" This copy also timed out.
4) Rebooted the computer, server and gigabit ethernet switch.
5) Server came back up, but I could only mount "Storage2" "Storage1" could not be mounted in the finder. (I didn't check web interface)
6) A few reboots and I saw this error in the startup screen:

"Cannot import "Storage1' : I/O error
Recovery is possible, but will result in some data loss. Returning the pool to its state as of Tue March 19 19:57:33 should correct the problem. Approximately 51 seconds of data must be discarded, irreversibly. Recovery can be attempted by executing "zpool import -F Storage1" A scrub of the pool is strongly recommended after recovery."

7) From the web interface of Freenas, I went into the shell and typed 'zpool import -F Storage1' A minute or so later I got this message on the monitor of the server (not the web interface):

panic: Solaris(panic): zfs: freeing free segment (offset-27763017236480 size-4608)
cpuid=0
KDB: enter panic
(thread pid 193 tid 100474 ]
Stopped at kdb_enter+0x3b: move $0,0x74ce42(xrip)
db>

8) Web interface now not responsive.

9) At the db> prompt on the server itself (not web interface), I typed "reboot" and server rebooted.

10) Upon reboot, I do not get into FreeNas. Upon the code: ZFS filesystem verison 5 and ZFS storage pool version 28, there is a lot of grey text about FreeNASsa1, FreeNASs4, FreeNASs3 skipping checks, being clean and reporting small frgamatentaion (like 0.7%) Then I get this panic:


panic: Solaris(panic): zfs: allocating allocated (offset=27763017236480 size-4608)
cpuid=1
KDB: enter panic
(thread pid 0 tid 100171 ]
Stopped at kdb_enter+0x3b: move $0,0x74ce42(xrip)
db>



Please help. That 20 TB Storage1 is critical and need to save the data off it.

Hardware:
16 of the drives are connected to the server by an Adaptec Raid SAS card but using only JBOD, and 4 drives are connected through the motherboard's SATA ports. I am booting 8.2 off a USB drive plugged directly into the motherboard. Dual Xeonon Intel Server (5000SPL) 16 gigs of memory.


Thanks
Doug

 

[paleoN]

Ok, I am trying to breath deeply.

Take a breath.

10) Upon reboot, I do not get into FreeNas. Upon the code: ZFS filesystem verison 5 and ZFS storage pool version 28, there is a lot of grey text about FreeNASsa1, FreeNASs4, FreeNASs3 skipping checks, being clean and reporting small frgamatentaion (like 0.7%) Then I get this panic:

This is happening as FreeNAS is attempting to import Storage1 during boot as you may have surmised already.

I am booting 8.2 off a USB drive plugged directly into the motherboard.

Install 8.3 to a new USB and boot off that. The newer ZFS code contains a number of fixes. Which may or may not help. Setup SSH and as root output of the following:

Code:

zpool import

camcontrol devlist

glabel status

gpart show

What's the SMART info for the drives look like? E.G.

Code:

smartctl -q noserial -a /dev/adaX