Does order of drive replacement matter?

[David Simpson]

I recently purchased two new drives for my aging Freenas Mini (it is such a champ!). In the attached screenshot you will see 2 of the 4 drives are troubled (fault and degraded). I just got two new WD Reds, but I'm curious if the order matters in how I replace them. When I click replace, it is looking for a new drive but I don't have a spare slot so the dropdown is blank. Any recommendations?

 

[awasb]

What is your pool layout? Any smart errors? What does zpool status show?

 

[Alecmascot]

With a RaidZ1 and 2 drives down I think you should restore from your backup after recreating your pool with the the replacement drives.

 

[David Simpson]

Code:

ot@freenas:~ # zpool status
  pool: Drive
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: resilvered 31.0G in 00:11:35 with 0 errors on Sat Mar  4 13:12:14 2023
config:

        NAME                                            STATE     READ WRITE CKSUM
        Drive                                           DEGRADED     0     0 0
          raidz1-0                                      DEGRADED     0     0 0
            gptid/abb78c4a-88b8-11e5-beea-d05099c043da  ONLINE       0     0 0
            gptid/ac7c0f67-88b8-11e5-beea-d05099c043da  ONLINE       0     0 1
            gptid/ad42f84f-88b8-11e5-beea-d05099c043da  ONLINE       0     0 0
            gptid/ae110dae-88b8-11e5-beea-d05099c043da  DEGRADED     0     0 0  too many errors

@awasb this doesn't show a faulted drive now, but I'm guessing that drive is on the fritz because it pops up occasionally. This Freenas mini has been running since Dec 2015 and this is the first issues I've ever had, these drives are getting tired after 8 years

 

[Johnny Fartpants]

gptid/ae110dae-88b8-11e5-beea-d05099c043da DEGRADED 0 0 0 too many errors

Replace this one first as its no longer in the pool.

 

[Johnny Fartpants]

gptid/ac7c0f67-88b8-11e5-beea-d05099c043da ONLINE 0 0 1

If it survives after that (wait for re-silver to complete) then this one.

 

[Johnny Fartpants]

PS: your pool is potentially moments away from death and losing all your data so if you can you may want to backup anything important before you replace anything as replacing disks may just push it over the edge.

 

[David Simpson]

PS: your pool is potentially moments away from death and losing all your data so if you can you may want to backup anything important before you replace anything as replacing disks may just push it over the edge.

Can I hot swap these or should I shutdown and power up after the swap?

 

[Johnny Fartpants]

You tell me. Does your system support hot-swap?

 

[Johnny Fartpants]

Do you know which drive is which?

 

[Alecmascot]

Those replacement WD Reds are CMR are'nt they ?

 

[Johnny Fartpants]

Tbh anything would be better than what’s there atm

 

[Alecmascot]

Tbh anything would be better than what’s there atm

but the resilver could take days....

 

[Johnny Fartpants]

Agreed but if they are SMR it could take weeks to get CMR drives.

 

[Johnny Fartpants]

If this pool was a person it would be being told by a Dr to get its affairs in order.

 

[Alecmascot]

Agreed but if they are SMR it could take weeks to get CMR drives.

like you said.. there's a huge risk in all of this...

 

[awasb]

First and foremost: Don't hurry. And as already mentioned ...

[1] Make some backup(s).
[2] Don't scrub or resilver. (Do not put any stress on the drives. At least as long as you don't have a _valid_ and functioning backup. But I wouldn't do anything else apart from backups.)
[3] Get _good_ replacement drives, i.e. CMR (maybe even larger ones than before). Get four of them ... maybe five. (If one of Your drives shows symptoms of defects of old age, it's likely that others will follow.)
[4] Export/Disconnect the old pool. (Don't purge/delete it and keep the drives. Just for sure.)
[5] Do some stress tests / disk burn in. Build a new pool. Preferably raidz2 from four to five drives.
[6] Restore data from the backup(s).

 

[David Simpson]

First and foremost: Don't hurry. And as already mentioned ...

[1] Make some backup(s).
[2] Don't scrub or resilver. (Do not put any stress on the drives. At least as long as you don't have a _valid_ and functioning backup. But I wouldn't do anything else apart from backups.)
[3] Get _good_ replacement drives, i.e. CMR (maybe even larger ones than before). Get four of them ... maybe five. (If one of Your drives shows symptoms of defects of old age, it's likely that others will follow.)
[4] Export/Disconnect the old pool. (Don't purge/delete it and keep the drives. Just for sure.)
[5] Do some stress tests / disk burn in. Build a new pool. Preferably raidz2 from four to five drives.
[6] Restore data from the backup(s).

The new drives are CMR, and 6TB versus 4TB. I've turned it off for now, I have the degraded drive identified by SN.

If it craps the bed I will survive, most of the data I have replicated to a Mac and that mirrors to Dropbox. I run Plex and a bunch of other stuff I would be sad to lose but can rebuild over time

 

[WI_Hedgehog]

First and foremost: Don't hurry. And as already mentioned ...

[1] Make some backup(s).
[2] Don't scrub or resilver. (Do not put any stress on the drives. At least as long as you don't have a _valid_ and functioning backup. But I wouldn't do anything else apart from backups.)
[3] Get _good_ replacement drives, i.e. CMR (maybe even larger ones than before). Get four of them ... maybe five. (If one of Your drives shows symptoms of defects of old age, it's likely that others will follow.)
[4] Export/Disconnect the old pool. (Don't purge/delete it and keep the drives. Just for sure.)
[5] Do some stress tests / disk burn in. Build a new pool. Preferably raidz2 from four to five drives.
[6] Restore data from the backup(s).

Excellent advice.

 

[David Simpson]

Looks like it will be a long time, yikes