Log SSD failed, can't import zpool with -m

[Markj]

I'm hoping someone knows of a solution. What I WANT to do is import the zpool in spite of the missing log device, RO would be fine, and copy off the work I did since my last tape backup. I believe that all the data I need is on the hard drives and good. I can then destroy and rebuild the pool with mirrored cache SSDs.

I've tied 'zpool import -m tank' and also with the -f, -F, -0 rdonly=yes options in various combinations without success, getting the same response:
"cannot import 'tank': one or more devices is currently unavailable"

The pool does show up in the gui and the alert button states that the volume status is UNKNOWN.

What happened is the cache device started throwing errors and then failed hard. FreeNAS would not get through POST to reboot until I unplugged the cache device. I was then able to reboot and scrub the pool with no errors. I then powered down, plugged in a new SSD, and could not reboot without an "Fatal Trap 12: Page fault while in Kernel mode" error. Unplugging the new SSD (and the cache device too) allowed me to boot. I can re-plug them after booting and they do show up from camcontrol devlist:
<Samsung SSD 840 EVO 120GB EXT0BB0Q> at scbus0 target 0 lun 0 (pass2,ada1)
<INTEL SSDSC2BW120A4 DC22> at scbus1 target 0 lun 0 (pass1,ada0)
<TOSHIBA TransMemory 1.00> at scbus6 target 0 lun 0 (pass0,da0)

Now output of gpartd shows (the first two 1TB drives scrolled off, but are the same):

=> 34 1952448445 mfid2 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid3 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid4 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid5 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid6 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid7 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid8 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 1952448445 mfid9 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1948254047 2 freebsd-zfs (929G)

=> 34 234441581 ada1 GPT (111G)
34 94 - free - (47k)
128 234441487 1 freebsd-zfs (111G)

=> 34 234441581 ada0 GPT (111G)
34 94 - free - (47k)
128 234441480 1 freebsd-zfs (111G)
234441608 7 - free - (3.5k)

Suggestions?

 

[Markj]

Oh, the configuration is:
Dell C2100, 16GB ECC ram, 10 WD Red 1TB in ZFS2, 1 SSD for cache, 1 for ZIL (will be mirrored once I get this un-fubared), FreeNAS 9.1

 

[Markj]

more info
[root@freenas /]# zpool import -o readonly=on -R /mnt ZFS2_7TB_V0
The devices below are missing, use '-m' to import the pool anyway:
15507427855844542268 [log]

cannot import 'ZFS2_7TB_V0': one or more devices is currently unavailable
[root@freenas /]# zpool import -m -o readonly=on -R /mnt ZFS2_7TB_V0
cannot import 'ZFS2_7TB_V0': one or more devices is currently unavailable
[root@freenas /]#

 

[cyberjock]

Money tells me you probably have a v15 pool that has that log bug that REQUIRES the ZIL to function for a pool to be mountable.

Grab one of your partitions that has ZFS on it and post the output of

zdb -l /dev/XXXp2 | grep version

If you get v15.. that's really bad afaik.

 

[Markj]

[root@freenas ~]# zdb -l /dev/mfid4p2 | grep version
version: 5000
version: 5000
Pool was created three weeks ago from a new/clean install of FreeNAS-9.1.1-RELEASE-x64 (a752d35)

I don't see how the pool version could be that far back level, but...

 

[cyberjock]

Well, that's slightly better than it could have been. v15 has a bug that fails the pool if your zil is lost.

But maybe you should go back to your first post and proof read your work. You said the cache drive failed. Well, if it were a cache drive you wouldn't be in this problem as you can even go so far as to unplug an l2arc without any consequence except reduced performance.

Post the output of zpool status from the command line and put it in CODE tags. Without the CODE tags you'll lose the text formatting and your output is useless.

 

[Markj]

Well, like it says in the title of the thread, the log device (ZIL) failed. I also took the cache offline as a shot-in-the-dark to get around the kernel panic, but it's back online. I did make a mistake in the text, Iwill create a new mirrored LOG (not cache) when I rebuild the pool. First I hope to recover the data. :)

Code:

[root@freenas ~]# zpool status -v                                                                                                 
no pools available                                                                                                                 
[root@freenas ~]# zpool status -x                                                                                                 
no pools available      

Code:

[root@freenas ~]# zpool import -o readonly=on -R /mnt ZFS2_7TB_V0                                                                 
The devices below are missing, use '-m' to import the pool anyway:                                                                 
            15507427855844542268 [log]                                                                                             
                                                                                                                                   
cannot import 'ZFS2_7TB_V0': one or more devices is currently unavailable                                                         
[root@freenas ~]#    

 

[cyberjock]

Doh.. I screwed that up. I didn't mean zpool status. I meant zpool import.. :(

 

[Markj]

Code:

pool: ZFS2_7TB_V0                                                           
    id: 3012034759271673160                                                   
  state: UNAVAIL                                                               
status: One or more devices are missing from the system.                     
action: The pool cannot be imported. Attach the missing                       
        devices and try again.                                                 
  see: http://illumos.org/msg/ZFS-8000-6X                                     
config:                                                                       
                                                                               
        ZFS2_7TB_V0  UNAVAIL  missing device                                   
          raidz2-0  ONLINE                                                   
            mfid9p2  ONLINE                                                   
            mfid0p2  ONLINE                                                   
            mfid1p2  ONLINE                                                   
            mfid2p2  ONLINE                                                   
            mfid3p2  ONLINE                                                   
            mfid4p2  ONLINE                                                   
            mfid5p2  ONLINE                                                   
            mfid6p2  ONLINE                                                   
            mfid7p2  ONLINE                                                   
            mfid8p2  ONLINE                                                   
                                                                               
        Additional devices are known to be part of this pool, though their     
        exact configuration cannot be determined.                             
[root@freenas ~]# zpool import  

Thanks VERY much for your help! I shoulda mirrored the ZIL device from the git-go, but...

 

[cyberjock]

I don't have any further advice specifically. But I just commented in your other thread...