SOLVED Yet another botched attempt at drive replacement!

hex

Cadet
Joined
Jan 27, 2021
Messages
9
While I've searched, and found plenty of the same sorts of questions, I am prepared for the flaming for my ignorance, or arrogance, in thinking that my problem deserves its own post. But per forum rules specs first......

FreeNAS-9.3-STABLE-201506292130
A Super-micro chassis capable of holding 30 drives, plus a few more that were JB welded into custom made racks!


PlatformIntel(R) Xeon(R) CPU E5-2609 v2 @ 2.50GHz
Memory131011MB


2 main pools
Athene(spelled wrong by previous network admin) 36.2TB based on a mirrored array of 4tb drives per 9 pools in combo with an SSD cache(zpool list and status to follow)
This is the array that I'm most worried about as it's degraded and if I lose the other drive in pool 7 I lose it all....
Furthermore, the web GUI shows no pools else I'd just replace the drive per this video https://www.youtube.com/watch?v=c8bvtj-LQ_A

Mercury a 1.81TB pool of SSD drives that the old network admin couldn't resist building into his custom NAS device...


The system emailed us about the degraded array per the following.....

Code:
Checking status of zfs pools:
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Athene        36.2T  16.0T  20.3T         -    54%    44%  1.00x  DEGRADED  /mnt
Mercury       1.81T   629G  1.20T         -    46%    33%  1.00x  ONLINE  /mnt
freenas-boot  15.5G  1.07G  14.4G         -      -     6%  1.00x  ONLINE  -

  pool: Athene
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: scrub repaired 0 in 24h36m with 0 errors on Mon Jan 18 00:56:38 2021
config:

    NAME                                            STATE     READ WRITE CKSUM
    Athene                                          DEGRADED     0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/e17f9a3b-bf8b-11e9-961c-0cc47a18b26c  ONLINE       0     0     0
        gptid/8ba5b933-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-1                                      ONLINE       0     0     0
        gptid/8c1189e4-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/8c81e63a-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-2                                      ONLINE       0     0     0
        gptid/8cf3ded2-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/8d60e0c1-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-3                                      ONLINE       0     0     0
        gptid/8dd67a9f-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/8e4a7a85-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-4                                      ONLINE       0     0     0
        gptid/8ec2063a-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/8f3a9a82-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-5                                      ONLINE       0     0     0
        gptid/8fac2a90-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/901e01d0-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-6                                      ONLINE       0     0     0
        gptid/908e55cc-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/90fd58a5-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-7                                      DEGRADED     0     0     0
        gptid/91725279-482c-11e5-99e5-0cc47a18b26c  FAULTED      5   448     0  too many errors
        gptid/91e8270b-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-8                                      ONLINE       0     0     0
        gptid/925ede81-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/92d5d7ac-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
      mirror-9                                      ONLINE       0     0     0
        gptid/9354b223-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/93c75f2d-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
    logs
      mirror-10                                     ONLINE       0     0     0
        gptid/95fa1e36-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
        gptid/9646a173-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0
    cache
      gptid/943385e6-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0
      gptid/94a5871c-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0
      gptid/9516c330-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0
      gptid/9589c284-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0

errors: No known data errors

-- End of daily output --




When the web interface was examined we discovered that we had no access to our pools, so the IT-Director had me replace the drive anyway, expecting the system to automatically re-silver the array. That did not work and what little command line magic I could muster reveled the resulting error....

Code:
Zpool status Athene
action: Attach the missing device and online it using 'zpool online'.                                                               
   see: http://illumos.org/msg/ZFS-8000-2Q                                                                                         
  scan: scrub repaired 0 in 24h36m with 0 errors on Mon Jan 18 00:56:38 2021                                                       
config:                                                                                                                             
                                                                                                                                    
        NAME                                            STATE     READ WRITE CKSUM                                                 
        Athene                                          DEGRADED     0     0     0                                                 
          mirror-0                                      ONLINE       0     0     0                                                 
            gptid/e17f9a3b-bf8b-11e9-961c-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/8ba5b933-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-1                                      ONLINE       0     0     0                                                 
            gptid/8c1189e4-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/8c81e63a-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-2                                      ONLINE       0     0     0                                                 
            gptid/8cf3ded2-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/8d60e0c1-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-3                                      ONLINE       0     0     0                                                 
            gptid/8dd67a9f-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/8e4a7a85-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-4                                      ONLINE       0     0     0                                                 
            gptid/8ec2063a-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/8f3a9a82-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-5                                      ONLINE       0     0     0                                                 
            gptid/8fac2a90-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/901e01d0-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-6                                      ONLINE       0     0     0                                                 
            gptid/908e55cc-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/90fd58a5-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-7                                      DEGRADED     0     0     0                                                 
            2463650246752463858                         UNAVAIL      0     0     0  was /dev/gptid/91725279-482c-11e5-99e5-0cc47a18b26c                                                                                                                                 
            gptid/91e8270b-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-8                                      ONLINE       0     0     0                                                 
            gptid/925ede81-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/92d5d7ac-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
          mirror-9                                      ONLINE       0     0     0                                                 
            gptid/9354b223-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/93c75f2d-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
        logs                                                                                                                       
          mirror-10                                     ONLINE       0     0     0                                                 
            gptid/95fa1e36-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
            gptid/9646a173-482c-11e5-99e5-0cc47a18b26c  ONLINE       0     0     0                                                 
        cache                                                                                                                       
          gptid/943385e6-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0                                                 
          gptid/94a5871c-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0                                                 
          gptid/9516c330-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0                                                 
          gptid/9589c284-482c-11e5-99e5-0cc47a18b26c    ONLINE       0     0     0                                                 
                                                                                                                                    
errors: No known data errors                                                 




So the faulty drive was not properly offlined, the web interface is not showing the existing pools, and the new drive has yet to be imported into the pool correctly... However the terminal commands seem to be telling the truth. Thus, since this Freenas device is over 5 years old, and houses data for 4 clinics and over 80 VM's, my boss has decided to whip up a new NAS. In the meantime, I've decided to try and fix this array before we lose everything.

Now then, I think the web interface is working well enough to import the new drive into the athene array via storage --> import disc: See pic....
import disc.png





Once that's done I just need the commands to offline the now missing drive, and the command to put da31 in it's place.... At this point I have little interest in trying to fix the web interface by re-importing the active yet missing pools, as I cannot afford to take the array offline for more than a few hours..... However if the experts deem it necessary then so be it...

I am also aware that some some glabel trickery may be necessary as a dev/gptid seems to be a zpool thing and not so much a BSD device name thing..
2463650246752463858 UNAVAIL 0 0 0 was /dev/gptid/91725279-482c-11e5-99e5-0cc47a18b26c

Thus, I throw myself upon the mercy of the BSD zpool CLI Gods... May they have mercy...

Regards
HEX
 

hex

Cadet
Joined
Jan 27, 2021
Messages
9
Here's a glabel status, as If there's going to be any help, this is going to be needed by the helpers... It also shows that da31 has yet to be added to any pool...

Code:
glabel status
                                      Name  Status  Components
gptid/f9ed3ea6-4832-11e5-99e5-0cc47a18b26c     N/A  da0p1
gptid/bab883f5-4832-11e5-99e5-0cc47a18b26c     N/A  da1p2
gptid/bbca9bbb-4832-11e5-99e5-0cc47a18b26c     N/A  da2p1
gptid/bb758622-4832-11e5-99e5-0cc47a18b26c     N/A  da3p2
gptid/8ba5b933-482c-11e5-99e5-0cc47a18b26c     N/A  da4p2
gptid/8c1189e4-482c-11e5-99e5-0cc47a18b26c     N/A  da5p2
gptid/8c81e63a-482c-11e5-99e5-0cc47a18b26c     N/A  da6p2
gptid/8cf3ded2-482c-11e5-99e5-0cc47a18b26c     N/A  da7p2
gptid/8d60e0c1-482c-11e5-99e5-0cc47a18b26c     N/A  da8p2
gptid/8dd67a9f-482c-11e5-99e5-0cc47a18b26c     N/A  da9p2
gptid/8e4a7a85-482c-11e5-99e5-0cc47a18b26c     N/A  da10p2
gptid/8ec2063a-482c-11e5-99e5-0cc47a18b26c     N/A  da11p2
gptid/8f3a9a82-482c-11e5-99e5-0cc47a18b26c     N/A  da12p2
gptid/943385e6-482c-11e5-99e5-0cc47a18b26c     N/A  da13p1
gptid/94a5871c-482c-11e5-99e5-0cc47a18b26c     N/A  da14p1
gptid/9516c330-482c-11e5-99e5-0cc47a18b26c     N/A  da15p1
gptid/9589c284-482c-11e5-99e5-0cc47a18b26c     N/A  da16p1
gptid/8fac2a90-482c-11e5-99e5-0cc47a18b26c     N/A  da17p2
gptid/901e01d0-482c-11e5-99e5-0cc47a18b26c     N/A  da18p2
gptid/908e55cc-482c-11e5-99e5-0cc47a18b26c     N/A  da19p2
gptid/90fd58a5-482c-11e5-99e5-0cc47a18b26c     N/A  da20p2
gptid/91e8270b-482c-11e5-99e5-0cc47a18b26c     N/A  da22p2
gptid/925ede81-482c-11e5-99e5-0cc47a18b26c     N/A  da23p2
gptid/92d5d7ac-482c-11e5-99e5-0cc47a18b26c     N/A  da24p2
gptid/9354b223-482c-11e5-99e5-0cc47a18b26c     N/A  da25p2
gptid/93c75f2d-482c-11e5-99e5-0cc47a18b26c     N/A  da26p2
gptid/e17f9a3b-bf8b-11e9-961c-0cc47a18b26c     N/A  da27p2
gptid/f6c67c4d-d7ff-11e9-9c99-0cc47a18b26c     N/A  da28p1
gptid/f6e29483-d7ff-11e9-9c99-0cc47a18b26c     N/A  da28p2
gptid/eae9fdf1-d802-11e9-9c99-0cc47a18b26c     N/A  da29p1
gptid/88a066e2-0ae4-11e5-8fca-0cc47a18b26c     N/A  da30p1
gptid/88e0a3ba-0ae4-11e5-8fca-0cc47a18b26c     N/A  da30p2
gptid/95fa1e36-482c-11e5-99e5-0cc47a18b26c     N/A  ada0p2
gptid/9646a173-482c-11e5-99e5-0cc47a18b26c     N/A  ada1p2


 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399

hex

Cadet
Joined
Jan 27, 2021
Messages
9
When the web interface was examined we discovered that we had no access to our pools!(should have said volumes)
The web GUI shows no pools(volumes) else I'd just replace the drive per this video https://www.youtube.com/watch?v=c8bvtj-LQ_A

Volumes do not populate in the main view, see pic. They show up to the left in the drop down menu, but I still can't figure out how to implement the instructions in the 9.3 guide.
no pools in main vies.png
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Try clicking on View Volumes on the left.
 

hex

Cadet
Joined
Jan 27, 2021
Messages
9
Clicking view volumes on the left does nothing. My volumes still do not populate.

volumes.png
 

hex

Cadet
Joined
Jan 27, 2021
Messages
9
Relevant or not, clicking on view disks in either view also does not populate the disks.
 

hex

Cadet
Joined
Jan 27, 2021
Messages
9
Does anyone know if this command is going to save me or not?
Code:
zpool replace Athene /dev/gptid/91725279-482c-11e5-99e5-0cc47a18b26c /dev/da31
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
No, because da31 doesn't appear to have the correct partitioning yet. First, run gpart create -s gpt da31 to create a GPT partition table. Then gpart add -t freebsd-swap -b 128 -s 2G -i 1 da31 and gpart add -t freebsd-zfs -b 4194432 -i 2 da31 to create the swap and ZFS partitions. Finally, gpart list da31 to get the rawuuid of da31p2. Then you can run zpool replace Athene 2463650246752463858 gptid/<rawuuid of da31p2>.
 

hex

Cadet
Joined
Jan 27, 2021
Messages
9
Code:
 gpart list da31
Geom name: da31
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7814037134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: da31p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 65536
   Mode: r0w0e0
   rawuuid: b994314b-6256-11eb-adad-0cc47a18b26c
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: da31p2
   Mediasize: 3998639463936 (3.7T)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 2147549184
   Mode: r0w0e0
   rawuuid: c9ed81fe-6256-11eb-adad-0cc47a18b26c
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: 1
   length: 3998639463936
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 7814037134
   start: 4194432
Consumers:
1. Name: da31
   Mediasize: 4000787030016 (3.7T)
   Sectorsize: 512
   Mode: r0w0e0



Ran all commands successfully except the last, and since it's the most critical, and I don't want to underestimate my own stupidity, I'm making sure it should be:

zpool replace Athene 2463650246752463858 gptid/c9ed81fe-6256-11eb-adad-0cc47a18b26c
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Yes, that's correct. You're telling zpool to replace the vdev in pool Athene with hash 2463650246752463858 (which corresponds to the missing drive) with the vdev having the named rawuuid GPT partition.
 

hex

Cadet
Joined
Jan 27, 2021
Messages
9
Excellent, it re-silvered and seems to be working. I suppose the next should be an entirely different thread, but why is the web interface so screwed up?

It's an old Freenas version, so is grabbing the config, installing a newer version, and feeding it the config the best way to try to fix it?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,399
Glad your pool is back to normal. Yes, 9.3 is very old. You can carry your current 9.3 config forward to 11.3, but I don't recommend jumping directly to 11.3 if you have a lot of plugins/jails. You should first upgrade to 11.1, to verify your config is stable under 11.x. 11.1 is the first release with the new GUI as an optional login. Then upgrade to 11.2, which lays a lot of plumbing necessary for the deprecation of warden jails and to make the new GUI primary. I would prepare to rebuild all the jails in 11.3, and delete any jails before upgrading. Finally, upgrade to 11.3, which makes the new GUI mandatory, and rebuild all your jails in the new iocage jail manager.
 
Top