Help with kernel panic

[RucanorX]

Hi all - I'm a little frustrated with an issue that seemingly came out of nowhere and hoping there's a better solution than losing all my data.

I have two WD Red 6 TB drives in a ZFS RAID 1 (mirror) config, only two jails running (Plex and Tautulli). What seems to be out of the blue, I started experiencing kernel panics and total shutdown/reboots after roughly 7-8 minutes of uptime. Everything I have troubleshooted has indicated that this problem seems to be occurring due to errors in the storage pool (and it also occurs whether the jails are running or not).

I've eliminated all hardware possibilities (completely forklifted the drives into a new chassis, different CPU, RAM, board, P/S, cables - the works) - did memory tests and HDD checks for bad sectors with no bad results. I've also individually disconnected each disk in the RAID seperately to see if i can isolate it to one of the disks, but it happens on either one. Also, when I moved to the new chassis/system, I booted it with a fresh install of FreeNas 11.2 STABLE on a USB disk. I've used different USB disks to eliminate that as an issue also.

This is the dump/trace that happens right before reboot. I'm kinda lost with this and understand that a corrupted pool will involve rebuilding it, but i have a considerable amount of data on there that I prefer not to lose.

Any ideas on how to approach this are appreciated!

 

[artlessknave]

#1. you clearly didn't read the forum post guild lines. people, particularly the ones with the most knowledge here, are FAR less likely to try to help if you don't do that basic step. mainly, we need to know your hardware. telling us you swapped it doesn't really help much.
#2. every time you say "zfs RAID", we all die a little inside.
#3. RAID/RAIDz/mirror is not a backup.

smartctl -x /dev/daX

I recomend stopping the jails, and making sure nothing else is writing to the pool.
you might have some success importing your pool readonly, and then copying data off of it

this is, unfortunately, exactly how zfs tends to behave, works great right up until it doesn't work; as an enterprise solution, it is expected to have backups, and so zfs doesn't really do anything more than tell you your data is boned.

 

[RucanorX]

I never followed up on this but it just occurred to me that I should close it out for those looking for answers.

1) If all hardware elements are completely changed except the disk, logic would tell you the problem exists as a physical or logical problem with said disks. I'm not trying to be a jerk, but in all honesty, the hardware details were really irrelevant in this case. But point taken, I will make sure to be more detail specific in the future.
2) Even Wikipedia calls it ZFS RAID, so being an elitist about terminology is a waste of words. It was a mirrored disk.
3) Did I say anything about backup? All I asked was a way to get data off. I had no problem blowing the whole thing away and starting over. As most people would want in this predicament, I wanted to save as much data as I can.

Be nice for a change - its shortens these threads when you're helpful :) I appreciate your help all the same, but ragging people about semantics.... Chill. The Read only advice helped stopped the panics but obviously wasn't usable in that configuration, but here's what i did:

disabled all jails and set to R/O
set grub parameters for recovery
used fuse and ntfs 3g to mount a WD external drive and painfully get as much data as I could. Not the fastest process in the whole but saved a solid 4 TiB of media.

I discussed with other people more proficient with ZFS and nobody can really explain what happened to my storage pool. I rebuilt from scratch on new everything, same data and no problems. Thinking maybe some bad blocks on one of those drives, but i couldnt keep it on long enough with the old hardware to do exhaustive tests. Oh well, gave me an excuse to get a Fractal Design case - these drives get loud!