SOLVED CRITICAL: Jan. 19, 2019, 7:34 a.m. - The volume Vol1 state is UNKNOWN - after power failure / Lightning

[Hoggy]

Hi all,
Last week we had a brief power failure and my beloved FreeNAS server is not happy. (We did have a Lightning strike in the vicinity).
Not sure if it's fixable and what steps to try to fix it.
I boot off 2 x USB's that are mirrored.
also, have 1 x Volume Vol1 that won't mount.
I've run the following commands with results.
The gpart command makes me think one disk ADA1 has died, but I really don't know.
Is it possible to break the mirror, repair ADA0 and then rebuild the mirror?
PC specs are in my signature. No hardware raid, cause that will defeat the object of ZFS.

Code:

[root@freenas ~]# zpool status -v                                                                                                 
  pool: freenas-boot                                                                                                             
state: ONLINE                                                                                                                   
  scan: scrub repaired 0 in 0 days 00:05:22 with 0 errors on Tue Jan 22 03:50:23 2019                                             
config:                                                                                                                           
                                                                                                                                  
        NAME        STATE     READ WRITE CKSUM                                                                                   
        freenas-boot  ONLINE       0     0     0                                                                                 
          mirror-0  ONLINE       0     0     0                                                                                   
            da0p2   ONLINE       0     0     0                                                                                   
            da1p2   ONLINE       0     0     0                                                                                   
                                                                                                                                  
errors: No known data errors

[root@freenas ~]# zpool status -x                                                                                                 
all pools are healthy
          
[root@freenas ~]# zpool import                                                                                                   
   pool: Vol1                                                                                                                     
     id: 17537304969476280379                                                                                                     
  state: UNAVAIL                                                                                                                 
status: One or more devices are missing from the system.                                                                         
action: The pool cannot be imported. Attach the missing                                                                         
        devices and try again.                                                                                                   
   see: http://illumos.org/msg/ZFS-8000-3C                                                                                       
config:                                                                                                                         
                                                                                                                                  
        Vol1                      UNAVAIL  insufficient replicas                                                                 
          mirror-0                UNAVAIL  insufficient replicas                                                                 
            13515042998217772309  UNAVAIL  corrupted data                                                                         
            11162558757959515843  UNAVAIL  cannot open  
[root@freenas ~]# diskinfo -v ada0                                                                                               
ada0                                                                                                                             
        512             # sectorsize                                                                                             
        3000615492608   # mediasize in bytes (2.7T)                                                                               
        5860577134      # mediasize in sectors                                                                                   
        4096            # stripesize                                                                                             
        0               # stripeoffset                                                                                           
        5814064         # Cylinders according to firmware.                                                                       
        16              # Heads according to firmware.                                                                           
        63              # Sectors according to firmware.                                                                         
        ST4000DM004-2CV104      # Disk descr.                                                                                     
        ZFN08X2K        # Disk ident.                                                                                             
        Not_Zoned       # Zone Mode                                                                                               
                                                                                                                                  
[root@freenas ~]# diskinfo -v ada1                                                                                               
ada1                                                                                                                             
        512             # sectorsize                                                                                             
        3000615492608   # mediasize in bytes (2.7T)                                                                               
        5860577134      # mediasize in sectors                                                                                   
        4096            # stripesize                                                                                             
        0               # stripeoffset                                                                                           
        5814064         # Cylinders according to firmware.                                                                       
        16              # Heads according to firmware.                                                                           
        63              # Sectors according to firmware.                                                                         
        ST4000DM004-2CV104      # Disk descr.                                                                                     
        ZFN1535S        # Disk ident.                                                                                             
        Not_Zoned       # Zone Mode                                                                                               
                                                                                                                                  
[root@freenas ~]#
[root@freenas ~]# gpart show ada0                                                                                                 
=>        40  5860577056  ada0  GPT  (2.7T) [CORRUPT]                                                                             
          40          88        - free -  (44K)                                                                                   
         128     4194304     1  freebsd-swap  (2.0G)                                                                             
     4194432  5856382656     2  freebsd-zfs  (2.7T)                                                                               
  5860577088           8        - free -  (4.0K)                                                                                 
                                                                                                                                  
[root@freenas ~]# gpart show ada1                                                                                                 
gpart: No such geom: ada1.   ????????????

 

[Hoggy]

Update:
Tried to mount the Vol1 read-only and that failed, Thinking I really need to get 2 x servers, and a UPS (for home use).
Then
located & unplugged the disks and tried to boot from the corrupted one.
It worked. so now I'm doing a scrub and will try a replacement disk when the scrub finishes.
Holding thumbs after a lot of googling and playing in Virtual box.

CRITICAL: Jan. 25, 2019, 2:30 p.m. - The volume Vol1 state is DEGRADED: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state.

Code:

root@freenas ~]# zpool status                                                                                                   
  pool: Vol1                                                                                                                     
state: DEGRADED                                                                                                                 
status: One or more devices could not be opened.  Sufficient replicas exist for                                                   
        the pool to continue functioning in a degraded state.                                                                     
action: Attach the missing device and online it using 'zpool online'.                                                             
   see: http://illumos.org/msg/ZFS-8000-2Q                                                                                       
  scan: scrub in progress since Fri Jan 25 14:39:10 2019                                                                         
        394G scanned at 1.17G/s, 37.1G issued at 113M/s, 1.63T total                                                             
        0 repaired, 2.23% done, 0 days 04:06:26 to go                                                                             
config:                                                                                                                           
                                                                                                                                  
        NAME                                            STATE     READ WRITE CKSUM                                               
        Vol1                                            DEGRADED     0     0     0                                               
          mirror-0                                      DEGRADED     0     0     0                                               
            13515042998217772309                        UNAVAIL      0     0     0  was /dev/ada0p2                               
            gptid/a8f885ec-2103-11e8-a49e-0013f7cb90c4  ONLINE       0     0     0                                               
                                                                                                                                  
errors: No known data errors                                                                                                     
                                                                                                                                  
  pool: freenas-boot                                                                                                             
state: ONLINE                                                                                                                   
  scan: scrub repaired 0 in 0 days 00:05:22 with 0 errors on Tue Jan 22 03:50:23 2019                                             
config:

 

[dlavigne]

Did it work out for you?

 

[Hoggy]

Did it work out for you?

Yes!! Wow, I didn't lose any data!
I physically disconnected the 2nd drive and booted up trying to only connect to the corrupted half of the mirror. It worked! Did the scrub.
I then reattached the 2nd disk to the mirror and did a resilver.

root@freenas:/mnt/pool/scripts # zpool status -v Vol1
pool: Vol1
state: ONLINE
status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: resilvered 1.63T in 0 days 10:06:12 with 6 errors on Sat Jan 26 04:49:49 2019
config:

NAME STATE READ WRITE CKSUM
Vol1 ONLINE 0 0 6
mirror-0 ONLINE 0 0 12
ada0p2 ONLINE 0 0 15
gptid/a8f885ec-2103-11e8-a49e-0013f7cb90c4 ONLINE 0 0 12
errors: Permanent errors have been detected in the following files:

/mnt/Vol1/Hogg/Old NAS Server/Public/wifesData/iTunes/iTunes Music/Compilations/The Complete Classical Collection [Disc/2-01 Mozart_ Piano Concerto #21 In C.m4a
root@freenas:/mnt/pool/scripts #

Because I was able to read the Data, I simply copied the data out the folder and back in. Problem solved!

I've set up more scrubs and smart test to see if I get any errors like the checksum errors. Haven't seen any so far.

The last thing I'm trying to figure out (Googling) is why the output is different for the 2 x disks.

Code:

 NAME                                                   STATE     READ WRITE CKSUM
        Vol1                                            ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            ada0p2                                      ONLINE       0     0     0
            gptid/a8f885ec-2103-11e8-a49e-0013f7cb90c4  ONLINE       0     0     0

UPDATE: Looks like this isn't a problem, all sorted!

I don't think I'm a Guru... I'm a handyman (handymanhogg.com.au) and I fixed it!
I reckon ZFS rocks!

Reference:

https://docs.joyent.com/private-cloud/troubleshooting/disk-replacement#unavail/
https://forums.freebsd.org/threads/checking-memory-ram-and-blacklisting-addresses.63369/
https://superuser.com/questions/114...d-i-be-about-checksum-errors-in-a-zfs-mirror/
http://bytesandbolts.com/fixing-freenas-error-currently-unreadable-pending-sectors/

Didn't do everything in the links above, but they gave me some insight into what is causing the issue, and how I can try and resolve it.
I didn't clear checksum error's cause I read that the checksum will be removed after an update reboot.