Disk failed, spare jumped in, replaced failed disk, but now cannot detach the spare

M

monte1299

Dabbler
Joined
Jun 4, 2017
Messages
23
Hello!

Before I get started, here's my system:
45 Drives Storinator
TrueNAS: TrueNAS-13.0-U5.3
MB: Supermicro X11DPL-i
HBA: LSI 9305 x 3
RAM: 128GB ECC
HD's: Mostly 8TB Seagate EXOS, some Seagate Ironwolf - all 8TB
Zpool is 4 vdevs of 10 drives as raidz3 with 5 drives as spares (I know, lol) - when I built this system, I was having multiple drive failures with the old system and decided to be overly cautious

Here's the problem:

I recently had a drive fail (da16), and one of my hot spares jumped in (da40). I got the notification, so I went to the UI, put the failed drive offline, and then shut down to replace da16. I put the new drive in, and started back up and let it resilver. When it completed without problems, and both drives were online, the spare did not return automatically to the spare pool. Instead of detaching the spare, I put it offline (doh!). It did not return to the spare set, so I tried to detach it, and the UI gives this error:

1695170660056.png


Even though the operation was unsuccessful, I do now see da40 back in the list of spares and ONLINE. However, it is also still in the pool where it originally jumped in as a spare:

1695170821066.png


I did some searching on the forums and the recommendation was to detach the drive from the CLI. Here's the pool:

Code:
root@goliath[~]# zpool status GOLIATH -f
cannot open '-f': name must begin with a letter
  pool: GOLIATH
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Online the device using 'zpool online' or replace the device with
    'zpool replace'.
  scan: resilvered 4.69T in 13:04:00 with 0 errors on Sat Aug 26 00:34:41 2023
config:

    NAME                                                  STATE     READ WRITE CKSUM
    GOLIATH                                               DEGRADED     0     0     0
      raidz3-0                                            ONLINE       0     0     0
        gptid/955cf876-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/a23ef856-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/96793d87-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/9cf24cd8-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/986b2ff2-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/a5bbaa71-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/7e3f0319-4375-11ee-aa23-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/add0e87d-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/b23ffb56-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/b1a2259c-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
      raidz3-1                                            DEGRADED     0     0     0
        gptid/9f55db7e-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/a3cb392f-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/a098b204-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/9ebdebc0-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/a4904562-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/5b63b7a9-4d49-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        spare-6                                           DEGRADED     0     0     0
          gptid/f1d1d7af-d59c-11ed-a862-3cecef6e76ba.eli  ONLINE       0     0     0
          gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba.eli  OFFLINE      0     0     0
        gptid/d0f5e63f-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/d1f02e0f-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/dd660045-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
      raidz3-2                                            ONLINE       0     0     0
        gptid/d7875504-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/dc3d4213-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e092bb98-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e27dd2db-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e456ac7b-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/da30a26a-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/de1d2773-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e8b14a04-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e8165f17-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e084d613-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
      raidz3-3                                            ONLINE       0     0     0
        gptid/e68ae0f8-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/e7662f77-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/ed566e44-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/fa460ef2-4c56-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/05452e50-4c57-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/074b1868-4c57-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/0a02dac7-4c57-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/0c65792b-4c57-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/1349e22f-4c57-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
        gptid/1445f15a-4c57-11eb-b083-3cecef6e76ba.eli    ONLINE       0     0     0
    spares
      gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba.eli      AVAIL  
      gptid/0ffba24a-4c57-11eb-b083-3cecef6e76ba.eli      AVAIL  
      gptid/1430a462-4c57-11eb-b083-3cecef6e76ba.eli      AVAIL  
      gptid/162bb3a8-4c57-11eb-b083-3cecef6e76ba.eli      AVAIL  
      gptid/16a0022d-4c57-11eb-b083-3cecef6e76ba.eli      AVAIL  

errors: No known data errors
root@goliath[~]#




The next thing I tried to was to online the drive with 'zpool online':

Code:
root@goliath[~]# zpool online GOLIATH gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba
cannot online gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba: no such device in pool
root@goliath[~]#



Got the same result if I try and detach:

Code:
root@goliath[~]# zpool detach GOLIATH gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba
cannot detach gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba: no such device in pool
root@goliath[~]#


So I'm a little stuck. da40 (gptid/0d93ec08-4c57-11eb-b083-3cecef6e76ba) is no longer part of the pool and is back in the list of spares, but zpool status still shows it in the output.

Do you guys have any ideas?

Thanks!
 
Johnny Fartpants

Johnny Fartpants

Guru
Joined
Jul 3, 2015
Messages
926
Have you tried to online the disk from the UI?

Also have you tried a reboot since you offlined the drive?

Failing that I would remove the spare from within the UI and see what happens.
 
Last edited:
M

monte1299

Dabbler
Joined
Jun 4, 2017
Messages
23
Johnny Fartpants said:
Have you tried to online the disk from the UI?

Also have you tried a reboot since you offlined the drive?

Failing that I would remove the spare from within the UI and see what happens.
Click to expand...

I don't have the option to online the drive from the UI:

1695222028856.png


I have rebooted multiple times, but no change.

When I try and remove da40 from the spares list, I get this error:

1695222138983.png
 
M

monte1299

Dabbler
Joined
Jun 4, 2017
Messages
23
I rebooted and tried the steps again and the results were the same. Any other thoughts?
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Disk failed, spare jumped in, replaced failed disk, but now cannot detach the spare"

Similar threads

Bill Calkins
Hot Spare still in use after replaced disk (cannot free up the hot spare)
Replies
5
Views
5K
House Of Cards
House Of Cards
A
freenas raidz1 all disks DEGRADED too many errors, please help
Replies
2
Views
2K
andyvfx
A
Jimbob
Replaced Disk Duplicates / Not Detaching or Removing (have been researched a bunch of threads).
2
Replies
35
Views
7K
isopropyl
isopropyl
M
Extra drives when using glabel status
Replies
2
Views
2K
Johnny Fartpants
Johnny Fartpants
P
Resilver Completed with 2 errors -- how can I delete/restore only the affected file?
2
Replies
28
Views
7K
Ericloewe
Ericloewe
Top