Interpreting Pool Status Output

[magnetar]

Hi,
I've had freenas running peacefully for many years with no trouble. Consequently, it's a long time since I've messed with it and I'm struggling a bit with how to deal with the 17% degredation of my pool. FreeNAS tells me there's a problems but doesn't seem to be explicit regarding what the problem actually is. Consequently, I'm unsure how to proceed. Though obviously I expect I'm going to need to replace one or more disks.

Here are some details about my system:
Version: 11.3-U3.2
RAM: 16GB
Disks: 6 x Seagate ST4000VN000-1H4168

Here is a screenshot of my pool status:


It looks like 2 disks have failed. Is that correct? I don't have any information on what the issue is. And I don't know which physical disks they are. The chassis is a Fractal Designs "Node" and so doesn't have hot swappable bays. I'm worried that if I power down and remove the wrong disk I'll cause myself bigger problems.


Regards
Dave

 

[Patrick M. Hausen]

Please use ssh to get a command line on your system, use these commands, and copy the output here wrapped in a CODE element:

Code:

zpool status
camcontrol devlist
gpart list

 

[magnetar]

Code:

root@freenas:~ # zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:02:18 with 0 errors on Mon Jul 13 03:47:18 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors

  pool: magus
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.

action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 0 days 01:21:33 with 0 errors on Thu Jul 16 17:36:01 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        magus                                           DEGRADED     0     0 0
          raidz1-0                                      DEGRADED     0     0 0
            gptid/03e42fa2-c039-11e4-88bf-d050990a6498  ONLINE       0     0 0
            16623217734093965384                        UNAVAIL      0     0 0  was /dev/gptid/045d08b2-c039-11e4-88bf-d050990a6498
            gptid/04bb3813-c039-11e4-88bf-d050990a6498  ONLINE       0     0
           16623217734093965384                        UNAVAIL      0     0 0  was /dev/gptid/045d08b2-c039-11e4-88bf-d050990a6498
            gptid/04bb3813-c039-11e4-88bf-d050990a6498  ONLINE       0     0 0
          raidz1-1                                      DEGRADED     0     0 0
            gptid/0515b5cd-c039-11e4-88bf-d050990a6498  ONLINE       0     0 0

errors: No known data errors

Code:

root@freenas:~ # camcontrol devlist
<ST4000VN000-1H4168 SC43>          at scbus0 target 0 lun 0 (pass0,ada0)
<ST4000VN000-1H4168 SC43>          at scbus2 target 0 lun 0 (pass1,ada1)
<ST4000VN000-1H4168 SC44>          at scbus3 target 0 lun 0 (pass2,ada2)
<ST4000VN000-1H4168 SC44>          at scbus5 target 0 lun 0 (pass3,ada3)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus6 target 0 lun 0 (ses0,pass4)
<SanDisk Extreme 0001>             at scbus8 target 0 lun 0 (pass5,da0)

Code:

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,03d332e2-c039-11e4-88bf-d050990a6498,0x80,0x400000)
   rawuuid: 03d332e2-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 3998639460352 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,03e42fa2-c039-11e4-88bf-d050990a6498,0x400080,0x1d180be08)
   rawuuid: 03e42fa2-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 3998639460352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 7814037127
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,04a82cd3-c039-11e4-88bf-d050990a6498,0x80,0x400000)
   rawuuid: 04a82cd3-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 3998639460352 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,04bb3813-c039-11e4-88bf-d050990a6498,0x400080,0x1d180be08)
   rawuuid: 04bb3813-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 3998639460352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 7814037127
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,0504f3aa-c039-11e4-88bf-d050990a6498,0x80,0x400000)
   rawuuid: 0504f3aa-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 3998639460352 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,0515b5cd-c039-11e4-88bf-d050990a6498,0x400080,0x1d180be08)
   rawuuid: 0515b5cd-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 3998639460352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 7814037127
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1
   efimedia: HD(1,GPT,05b7c89c-c039-11e4-88bf-d050990a6498,0x80,0x400000)
   rawuuid: 05b7c89c-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 3998639460352 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   efimedia: HD(2,GPT,05d3ea80-c039-11e4-88bf-d050990a6498,0x400080,0x1d180be08)
   rawuuid: 05d3ea80-c039-11e4-88bf-d050990a6498
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 3998639460352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 7814037127
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 4000787030016 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 61282597
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da0p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   efimedia: HD(1,GPT,47379c44-c002-11e4-b8e0-d050990a6498,0x22,0x400)
   rawuuid: 47379c44-c002-11e4-b8e0-d050990a6498
   rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
   label: (null)
   length: 524288
   offset: 17408
   type: freebsd-boot
   index: 1
   end: 1057
   start: 34
2. Name: da0p2
   Mediasize: 31376142336 (29G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   efimedia: HD(2,GPT,473f428d-c002-11e4-b8e0-d050990a6498,0x428,0x3a714f8)
   rawuuid: 473f428d-c002-11e4-b8e0-d050990a6498
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 31376142336
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 61282591
   start: 1064
Consumers:
1. Name: da0
   Mediasize: 31376707072 (29G)
   Sectorsize: 512
   Mode: r1w1e2

Thanks for helping me out :)

 

[Patrick M. Hausen]

So two of your disk are gone for good. You can use smartctl to get the serial numbers of the 4 remaining ones to make sure to replace the correct drives. Of course before you replace them you could at least check the cabling.
Or power down the machine, unplug your FreeNAS boot drive, unplug power from all hard disks and then plug the drive power back in to see if they spin up. It's noticeable. So you wil know if the two failed drives really are dead.

Every power cycle will put additional stress on your remaining drives and if one more fails you will lose all your data. You might want to perform an additional complete backup first.

 

[magnetar]

I've got nothing to backup to so I'm gonna have to move quickly and hope.

Regarding new disks do I need to care about which types I match up with my current set? Or will any NAS spec drive do the job?

 

[Patrick M. Hausen]

Make sure not to get SMR drives. ST4000VN008 seem to be a good more current replacement to me.