offline pool after upgrade from 9.10 to 11.1

Brad T

Dabbler
Joined
Nov 23, 2016
Messages
15
After upgrading from FreeNAS-9.10.2-U6 (561f0d7a1) to FreeNAS-11.1-U7, I've lost a pool, with 4 disks being marked as UNAVAIL (or more precisely 4 out of 6 disks in a Z1 vdev).

In the GUI, I get this message:

1566159055650.png


Before (after expanding the pool in January 2019):
Code:
[root@freenas02] /mnt/vol01# zpool list -v
NAME                                     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot                            7.44G  5.70G  1.74G         -      -    76%  1.00x  ONLINE  -
  mirror                                7.44G  5.70G  1.74G         -      -    76%
    gptid/bd74e4fb-4470-11e5-bb9f-0014fd1702be      -      -      -         -      -      -
    gptid/1d862695-c6e5-11e4-b757-0014fd1702be      -      -      -         -      -      -
vol01                                   32.5T  15.2T  17.3T         -    19%    46%  1.00x  ONLINE  /mnt
  raidz1                                16.2T  15.2T  1.04T         -    39%    93%
    gptid/bd56e573-8b83-11e6-9a21-d05099c10963      -      -      -         -      -      -
    gptid/d65641e2-b536-11e6-9d70-d05099c10963      -      -      -         -      -      -
    gptid/bee5773e-8b83-11e6-9a21-d05099c10963      -      -      -         -      -      -
    gptid/bf9a5e0b-8b83-11e6-9a21-d05099c10963      -      -      -         -      -      -
    gptid/c0540768-8b83-11e6-9a21-d05099c10963      -      -      -         -      -      -
    gptid/c110d0a2-8b83-11e6-9a21-d05099c10963      -      -      -         -      -      -
  raidz1                                16.2T  17.0M  16.2T         -     0%     0%
    da0                                     -      -      -         -      -      -
    da1                                     -      -      -         -      -      -
    da2                                     -      -      -         -      -      -
    da3                                     -      -      -         -      -      -
    da4                                     -      -      -         -      -      -
    da5                                     -      -      -         -      -      -


After
Code:
root@freenas02:~ # zpool import -V
   pool: vol01
     id: 10112117251247527724
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

vol01                                           UNAVAIL  insufficient replicas
  raidz1-0                                      ONLINE
    gptid/bd56e573-8b83-11e6-9a21-d05099c10963  ONLINE
    gptid/d65641e2-b536-11e6-9d70-d05099c10963  ONLINE
    gptid/bee5773e-8b83-11e6-9a21-d05099c10963  ONLINE
    gptid/bf9a5e0b-8b83-11e6-9a21-d05099c10963  ONLINE
    gptid/c0540768-8b83-11e6-9a21-d05099c10963  ONLINE
    gptid/c110d0a2-8b83-11e6-9a21-d05099c10963  ONLINE
  raidz1-1                                      UNAVAIL  insufficient replicas
    da0                                         ONLINE
    da1                                         ONLINE
    16339273880762305465                        UNAVAIL  cannot open
    10436155621961562286                        UNAVAIL  cannot open
    5238054744404005019                         UNAVAIL  cannot open
    3466877252226870208                         UNAVAIL  cannot open


I would be surprised if the HBA (or channel A of the HBA) or 4 disks failed simultaneously. (Although I know it's a possibility.)

Here's what the disks look like now individually:
Code:
root@freenas02:~ # camcontrol devlist -v
scbus0 on mps0 bus 0:
<ATA WDC WD30EFRX-68E 0A80>        at scbus0 target 4 lun 0 (pass0,da0)
<ATA WDC WD30EFRX-68E 0A82>        at scbus0 target 5 lun 0 (pass1,da1)
<ATA WDC WD30EFRX-68E 0A80>        at scbus0 target 6 lun 0 (pass2,da2)
<ATA WDC WD30EFRX-68E 0A80>        at scbus0 target 8 lun 0 (pass3,da3)
<ATA WDC WD30EFRX-68E 0A80>        at scbus0 target 9 lun 0 (pass4,da4)
<ATA WDC WD30EFRX-68E 0A80>        at scbus0 target 10 lun 0 (pass5,da5)
<>                                 at scbus0 target -1 lun ffffffff ()
scbus1 on ahcich0 bus 0:
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus1 target 0 lun 0 (pass6,ada0)
<>                                 at scbus1 target -1 lun ffffffff ()
scbus2 on ahcich1 bus 0:
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus2 target 0 lun 0 (pass7,ada1)
<>                                 at scbus2 target -1 lun ffffffff ()
scbus3 on ahcich2 bus 0:
<WDC WD30EFRX-68AX9N0 80.00A80>    at scbus3 target 0 lun 0 (pass8,ada2)
<>                                 at scbus3 target -1 lun ffffffff ()
scbus4 on ahcich3 bus 0:
<WDC WD30EFRX-68EUZN0 82.00A82>    at scbus4 target 0 lun 0 (pass9,ada3)
<>                                 at scbus4 target -1 lun ffffffff ()
scbus5 on ahcich4 bus 0:
<WDC WD30EFRX-68EUZN0 82.00A82>    at scbus5 target 0 lun 0 (pass10,ada4)
<>                                 at scbus5 target -1 lun ffffffff ()
scbus6 on ahcich5 bus 0:
<WDC WD30EFRX-68EUZN0 82.00A82>    at scbus6 target 0 lun 0 (pass11,ada5)
<>                                 at scbus6 target -1 lun ffffffff ()
scbus7 on camsim0 bus 0:
<>                                 at scbus7 target -1 lun ffffffff ()
scbus8 on umass-sim0 bus 0:
<SanDisk Cruzer Fit 1.26>          at scbus8 target 0 lun 0 (da6,pass12)
scbus-1 on xpt0 bus 0:
<>                                 at scbus-1 target -1 lun ffffffff (xpt0)

root@freenas02:~ # gpart show
=>        34  5860533101  da2  GPT  (2.7T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128           7       - free -  (3.5K)

=>        34  5860533101  da3  GPT  (2.7T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128           7       - free -  (3.5K)

=>        34  5860533101  da4  GPT  (2.7T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128           7       - free -  (3.5K)

=>        34  5860533101  da5  GPT  (2.7T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128           7       - free -  (3.5K)

=>      34  15633341  da6  GPT  (7.5G)
        34      1024    1  freebsd-boot  (512K)
      1058         6       - free -  (3.0K)
      1064  15632304    2  freebsd-zfs  (7.5G)
  15633368         7       - free -  (3.5K)

=>      34  15633341  da7  GPT  (7.5G)
        34      1024    1  freebsd-boot  (512K)
      1058         6       - free -  (3.0K)
      1064  15632304    2  freebsd-zfs  (7.5G)
  15633368         7       - free -  (3.5K)

=>        34  5860533101  ada0  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada1  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada3  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada4  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada5  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  da0  GPT  (2.7T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128           7       - free -  (3.5K)

=>        34  5860533101  da1  GPT  (2.7T)
          34          94       - free -  (47K)
         128     4194304    1  freebsd-swap  (2.0G)
     4194432  5856338696    2  freebsd-zfs  (2.7T)
  5860533128           7       - free -  (3.5K)

root@freenas02:~ # glabel status
                                      Name  Status  Components
gptid/b028e667-f483-11e4-a2e2-ac220b4f4343     N/A  da2p2
gptid/aec2bf48-f483-11e4-a2e2-ac220b4f4343     N/A  da3p2
gptid/afc5b4b8-f483-11e4-a2e2-ac220b4f4343     N/A  da4p2
gptid/be17ff68-8b83-11e6-9a21-d05099c10963     N/A  da5p2
gptid/bd45140b-4470-11e5-bb9f-0014fd1702be     N/A  da6p1
gptid/bd74e4fb-4470-11e5-bb9f-0014fd1702be     N/A  da6p2
gptid/1d7804a0-c6e5-11e4-b757-0014fd1702be     N/A  da7p1
gptid/1d862695-c6e5-11e4-b757-0014fd1702be     N/A  da7p2
gptid/bf9a5e0b-8b83-11e6-9a21-d05099c10963     N/A  ada0p2
gptid/c0540768-8b83-11e6-9a21-d05099c10963     N/A  ada1p2
gptid/bee5773e-8b83-11e6-9a21-d05099c10963     N/A  ada2p2
gptid/c110d0a2-8b83-11e6-9a21-d05099c10963     N/A  ada3p2
gptid/bd56e573-8b83-11e6-9a21-d05099c10963     N/A  ada4p2
gptid/d65641e2-b536-11e6-9d70-d05099c10963     N/A  ada5p2
gptid/b07c7234-f483-11e4-a2e2-ac220b4f4343     N/A  da0p1
gptid/b088f945-f483-11e4-a2e2-ac220b4f4343     N/A  da0p2
gptid/b121c1c9-f483-11e4-a2e2-ac220b4f4343     N/A  da1p1
gptid/b12d6a01-f483-11e4-a2e2-ac220b4f4343     N/A  da1p2


Here's the part of the boot log related to HDD:

Code:
…
Aug 17 13:46:54 freenas02 ahci0: <Intel Lynx Point AHCI SATA controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xf7c15000-0xf7c157ff irq 19 at device 31.2 on pci0
Aug 17 13:46:54 freenas02 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
Aug 17 13:46:54 freenas02 ahcich0: <AHCI channel> at channel 0 on ahci0
Aug 17 13:46:54 freenas02 ahcich1: <AHCI channel> at channel 1 on ahci0
Aug 17 13:46:54 freenas02 ahcich2: <AHCI channel> at channel 2 on ahci0
Aug 17 13:46:54 freenas02 ahcich3: <AHCI channel> at channel 3 on ahci0
Aug 17 13:46:54 freenas02 ahcich4: <AHCI channel> at channel 4 on ahci0
Aug 17 13:46:54 freenas02 ahcich5: <AHCI channel> at channel 5 on ahci0
…
Aug 17 13:46:54 freenas02 mps0: SAS Address for SATA device = d2624739dcc8c074
Aug 17 13:46:54 freenas02 mps0: SAS Address for SATA device = d267514bf8ccb972
Aug 17 13:46:54 freenas02 mps0: SAS Address for SATA device = d2624738ddc8c074
Aug 17 13:46:54 freenas02 mps0: SAS Address for SATA device = d2624738d9c3bd76
Aug 17 13:46:54 freenas02 mps0: SAS Address for SATA device = d2624734dfcabe71
Aug 17 13:46:54 freenas02 mps0: SAS Address for SATA device = d2624737dbc8b86d
Aug 17 13:46:54 freenas02 mps0: SAS Address from SATA device = d2624739dcc8c074
Aug 17 13:46:54 freenas02 mps0: SAS Address from SATA device = d267514bf8ccb972
Aug 17 13:46:54 freenas02 mps0: SAS Address from SATA device = d2624738ddc8c074
Aug 17 13:46:54 freenas02 mps0: SAS Address from SATA device = d2624738d9c3bd76
Aug 17 13:46:54 freenas02 mps0: SAS Address from SATA device = d2624734dfcabe71
Aug 17 13:46:54 freenas02 mps0: SAS Address from SATA device = d2624737dbc8b86d
…

Aug 17 13:46:54 freenas02 ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
Aug 17 13:46:54 freenas02 da0 at mps0 bus 0 scbus0 target 4 lun 0
Aug 17 13:46:54 freenas02 ada0: <WDC WD30EFRX-68AX9N0 80.00A80> ACS-2 ATA SATA 3.x device
Aug 17 13:46:54 freenas02 ada0: Serial Number WD-WCC1T0454382
Aug 17 13:46:54 freenas02 da0: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device
Aug 17 13:46:54 freenas02 ada0: 600.000MB/s transfersda0: Serial Number WD-WMC4N0755697
Aug 17 13:46:54 freenas02 da0: 600.000MB/s transfers
Aug 17 13:46:54 freenas02 da0: Command Queueing enabled
Aug 17 13:46:54 freenas02 da0: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da0: quirks=0x8<4K>
Aug 17 13:46:54 freenas02 da1 at mps0 bus 0 scbus0 target 5 lun 0
Aug 17 13:46:54 freenas02 da1: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device
Aug 17 13:46:54 freenas02 da1: Serial Number WD-WCC4N5AHPD25
Aug 17 13:46:54 freenas02 da1: 600.000MB/s transfers
Aug 17 13:46:54 freenas02 da1: Command Queueing enabled
Aug 17 13:46:54 freenas02 da1: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da1: quirks=0x8<4K>
Aug 17 13:46:54 freenas02 da2 at mps0 bus 0 scbus0 target 6 lun 0
Aug 17 13:46:54 freenas02 (SATA 3.x, UDMA6, PIO 8192bytes)
Aug 17 13:46:54 freenas02 ada0: Command Queueing enabled
Aug 17 13:46:54 freenas02 da2: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device
Aug 17 13:46:54 freenas02 ada0: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 ada0: quirks=0x1<4K>
Aug 17 13:46:54 freenas02 ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
Aug 17 13:46:54 freenas02 da2: Serial Number WD-WMC4N0764697
Aug 17 13:46:54 freenas02 da2: 600.000MB/s transfersada1: <WDC WD30EFRX-68AX9N0 80.00A80> ACS-2 ATA SATA 3.x device
Aug 17 13:46:54 freenas02 ada1: Serial Number WD-WCC1T0418229

Aug 17 13:46:54 freenas02 ada1: 600.000MB/s transfers
Aug 17 13:46:54 freenas02 da2: Command Queueing enabled
Aug 17 13:46:54 freenas02 da2: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 (SATA 3.x, da2: quirks=0x8<4K>
Aug 17 13:46:54 freenas02 da3 at mps0 bus 0 scbus0 target 8 lun 0
Aug 17 13:46:54 freenas02 da3: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device
Aug 17 13:46:54 freenas02 da3: Serial Number WD-WMC4N0717874
Aug 17 13:46:54 freenas02 da3: 600.000MB/s transfers
Aug 17 13:46:54 freenas02 da3: Command Queueing enabled
Aug 17 13:46:54 freenas02 UDMA6, PIO 8192bytes)
Aug 17 13:46:54 freenas02 ada1: Command Queueing enabled
Aug 17 13:46:54 freenas02 ada1: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da3: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da3: quirks=0x8<4K>
Aug 17 13:46:54 freenas02 da4 at mps0 bus 0 scbus0 target 9 lun 0
Aug 17 13:46:54 freenas02 da4: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device
Aug 17 13:46:54 freenas02 da4: Serial Number WD-WMC4N0751169
Aug 17 13:46:54 freenas02 da4: 600.000MB/s transfers
Aug 17 13:46:54 freenas02 da4: Command Queueing enabled
Aug 17 13:46:54 freenas02 da4: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da4: quirks=0x8<4K>
Aug 17 13:46:54 freenas02 da5 at mps0 bus 0 scbus0 target 10 lun 0
Aug 17 13:46:54 freenas02 da5: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device
Aug 17 13:46:54 freenas02 ada1: quirks=0x1<4K>
Aug 17 13:46:54 freenas02 ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
Aug 17 13:46:54 freenas02 da5: Serial Number WD-WMC4N0743610
Aug 17 13:46:54 freenas02 ada2: <WDC WD30EFRX-68AX9N0 80.00A80> ACS-2 ATA SATA 3.x device
Aug 17 13:46:54 freenas02 da5: 600.000MB/s transfers
Aug 17 13:46:54 freenas02 da5: Command Queueing enabled
Aug 17 13:46:54 freenas02 ada2: Serial Number WD-WCC1T0298346
Aug 17 13:46:54 freenas02 ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Aug 17 13:46:54 freenas02 ada2: Command Queueing enabled
Aug 17 13:46:54 freenas02 ada2: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da5: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 da5: quirks=0x8<4K>
Aug 17 13:46:54 freenas02 ada2: quirks=0x1<4K>
Aug 17 13:46:54 freenas02 ada3 at ahcich3 bus 0 scbus4 target 0 lun 0
Aug 17 13:46:54 freenas02 ada3: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
Aug 17 13:46:54 freenas02 ada3: Serial Number WD-WCC4N2TYF2UN
Aug 17 13:46:54 freenas02 ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Aug 17 13:46:54 freenas02 ada3: Command Queueing enabled
Aug 17 13:46:54 freenas02 ada3: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 ada3: quirks=0x1<4K>
Aug 17 13:46:54 freenas02 ada4 at ahcich4 bus 0 scbus5 target 0 lun 0
Aug 17 13:46:54 freenas02 ada4: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
Aug 17 13:46:54 freenas02 ada4: Serial Number WD-WCC4N7FATRR1
Aug 17 13:46:54 freenas02 ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Aug 17 13:46:54 freenas02 ada4: Command Queueing enabled
Aug 17 13:46:54 freenas02 ada4: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 ada4: quirks=0x1<4K>
Aug 17 13:46:54 freenas02 ada5 at ahcich5 bus 0 scbus6 target 0 lun 0
Aug 17 13:46:54 freenas02 ada5: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
Aug 17 13:46:54 freenas02 ada5: Serial Number WD-WCC4N4FCUHHY
Aug 17 13:46:54 freenas02 ada5: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Aug 17 13:46:54 freenas02 ada5: Command Queueing enabled
Aug 17 13:46:54 freenas02 ada5: 2861588MB (5860533168 512 byte sectors)
Aug 17 13:46:54 freenas02 ada5: quirks=0x1<4K>
Aug 17 13:46:54 freenas02 Trying to mount root from zfs:freenas-boot/ROOT/11.1-U7 []...
…
Aug 17 14:49:46 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=3671607037181642970
Aug 17 14:49:46 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=9652509477325859936
Aug 17 14:49:47 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=14436258189133544651
Aug 17 14:49:47 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=13407339990873068916
Aug 17 14:49:47 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=11357776941345595226
Aug 17 14:49:47 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=17955406522460418432
Aug 17 14:49:47 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=3535278873160926383
Aug 17 14:49:47 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=15861408793602997199
Aug 17 14:49:48 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=16339273880762305465
Aug 17 14:49:48 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=10436155621961562286
Aug 17 14:49:48 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=5238054744404005019
Aug 17 14:49:48 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=3466877252226870208
…
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=3671607037181642970
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=9652509477325859936
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=14436258189133544651
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=13407339990873068916
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=11357776941345595226
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=17955406522460418432
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=3535278873160926383
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=15861408793602997199
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=16339273880762305465
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=10436155621961562286
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=5238054744404005019
Aug 17 14:51:18 freenas02 ZFS: vdev state changed, pool_guid=10112117251247527724 vdev_guid=3466877252226870208
…
Aug 17 14:53:45 freenas02 ZFS: vdev state changed, pool_guid=3018436033628833179 vdev_guid=13331606250362451550
Aug 17 14:53:46 freenas02 ZFS: vdev state changed, pool_guid=3018436033628833179 vdev_guid=8155704046927289062
…



I haven't issued any zfs commands that write or update config and have only rebooted the system a few times to have a look at the EFI settings and watch the boot.

I am ok in terms of backup -- I snapshot/replicate this server's data to a twin, but I am concerned that an upgrade seemingly tanked the pool.

Questions:
  1. Can I recover these 4 disks and import the pool? (tweaking a few settings seems faster than destroying the remnants of the pool and then zfs sending 15 TB+ over the wire from the backup server) If so, what are the next steps?
  2. Are there any known issues with the 11.x upgrades and a pool that has been extended this way?
  3. Should I expect this behaviour when I upgrade the backup server as well?
thanks gurus. cheers... brad
 

Brad T

Dabbler
Joined
Nov 23, 2016
Messages
15
Were you able to resolve this? If so, how?

Unfortunately no. I had to recover the data from the server's twin. I did lose one jail that was not replicated.

I'm a bit gun shy to upgrade the backup server and my media server now, too. :(

Thanks for checking @dlavigne
 
Top