SOLVED Replace Failing Disk with Larger Disk

[selfsame]

I have a failing 1TB drive next to a 2TB and a 4TB drive in my data pool (Kurgan). I'm not sure what the exact configuration is but the results of

Code:

$zpool status

are:

Code:

  pool: Kurgan
 state: ONLINE
  scan: scrub in progress since Sun Feb 28 00:00:01 2016
        3.79T scanned out of 3.88T at 105M/s, 0h15m to go
        0 repaired, 97.54% done
config:

    NAME                                          STATE     READ WRITE CKSUM
    Kurgan                                        ONLINE       0     0     0
      gptid/90e48b8b-409c-11e5-a865-d0509964e4cf  ONLINE       0     0     0
      gptid/8d79ac89-58e9-11e5-a961-d0509964e4cf  ONLINE       0     0     0
      gptid/94b3f008-58e9-11e5-a961-d0509964e4cf  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h2m with 0 errors on Wed Feb 24 03:47:38 2016
config:

    NAME                                          STATE     READ WRITE CKSUM
    freenas-boot                                  ONLINE       0     0     0
      gptid/bc0ea89f-4098-11e5-90c7-d0509964e4cf  ONLINE       0     0     0

errors: No known data errors

It's not reporting errors, but in the FreeNAS GUI I see:

• CRITICAL: Device: /dev/ada1, 1 Currently unreadable (pending) sectors
• CRITICAL: Device: /dev/ada1, 1 Offline uncorrectable sectors

And /dev/ada1 maps to the 1TB drive. I've had to reboot twice now due to failing services (web services and ssh all become unresponsive) I think due to the scrub failing on the 1TB drive (scrub takes more than 12 hours to get to 97.54% done. I think it's hanging on bad sectors but not sure at all).

All this is to say - I want to replace the 1TB drive, but I'd rather replace it with a larger drive. According to the FreeBSD docs on ZFS, this should be possible, but I'm not clear on if I need to do anything fancy before replacing the drive (e.g. partition replacement 4TB drive as 1TB | 3TB) or which instructions I need to follow ('replace a functioning device 19.3.5' or 'dealing with failed devices 19.3.6').

Any help, advice, or pointers to relevant threads would be most welcome.

 

[depasseg]

Have you seen the FreeNAS manual? http://doc.freenas.org/9.3/freenas_storage.html#replacing-a-failed-drive

 

[Yatti420]

Does the speed slow down? Figure out the pool config if you have redundancy will be fine.. Follow guide for replacement..

 

[gpsguy]

I wouldn't follow the directions above. It looks like you have a striped pool. If you remove a drive - you will loose all your data.

I would back it up and start over with either mirrors or some form of RAIDzX.


Sent from my iPhone using Tapatalk

 

[Yatti420]

Needs to be way clearer if that is the case.. There should be warnings everywhere about removing drives (shouldn't be possible under a stripe)\ and failure warnings that are unresolvable(permanant) in the GUI if you run stripes.. My z2 does show an extra line.. It should show STRIPED POOL WARNING if you use stripes..

Code:

/mnt/z2pool# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: resilvered 533M in 2h3m with 0 errors on Sun Jan 17 17:43:12 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        freenas-boot                                    ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/52daeee6-b980-11e5-9e74-002590d50af7  ONLINE       0     0     0
            gptid/66aafbbd-bd5a-11e5-b6d6-002590d50af7  ONLINE       0     0     0

errors: No known data errors

  pool: z2pool
 state: ONLINE
  scan: scrub repaired 0 in 7h57m with 0 errors on Tue Feb 16 21:42:29 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        z2pool                                          ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/cfb437fa-b293-11e5-8655-002590d50af7  ONLINE       0     0     0
            gptid/2f809b37-7734-11e4-80af-002590d50af7  ONLINE       0     0     0
            gptid/ccc661da-5682-11e4-9d00-002590d50af7  ONLINE       0     0     0
            gptid/41240031-b75e-11e3-9569-002590d50af7  ONLINE       0     0     0
            gptid/41a0a0c4-b75e-11e3-9569-002590d50af7  ONLINE       0     0     0
            gptid/fc650f19-524a-11e4-8a2e-002590d50af7  ONLINE       0     0     0

errors: No known data errors

 

[depasseg]

I wouldn't follow the directions above. It looks like you have a striped pool. If you remove a drive - you will loose all your data.

That's why you shouldn't follow the FreeBSD directions. However, the FreeNAS directions should copy the data to the new drive prior to removing the failing disk from the stripe. But the underlying point is well advised - backup that dataset ASAP and put it on a protected configuration. You are flirting with major data loss.

 

[gpsguy]

Thanks depasseg. I panicked a bit too fast when I saw the stripe.


Sent from my iPhone using Tapatalk

 

[depasseg]

Thanks depasseg. I panicked a bit too fast when I saw the stripe.

That was a great catch! I didn't even notice.

 

[Yatti420]

I still think it's an easy improvement.. I hope FreeNAS atleast pops alerts or alarms to inform about the stripe.. and no.. I don't care if (the alerts etc) its ugly.. I want it to be if anything!..

 

[Bidule0hm]

Well, this is FreeNAS, not Bisounours-land... If you need an alert for each thing that can be dangerous then use Windows :)

 

[selfsame]

Thanks for all the advice. I'm not worried about data loss - I have an offsite backup for all my important stuff and a pretty recent snapshot of almost everything else.

After reading through the FreeNAS Storage page, at the recommendation of depasseg, section 8.1.11 indicates that I should be able to just add a larger drive and grow the pool with a replacement drive. Is there any risk in doing this with the failing drive? As I've only got them in a stripe configuration I can't offline the failing drive (trying to offline the drive in the GUI shows "Error: Disk offline failed: "cannot offline gptid/xxx: no valid replicas") so I'd like to make sure I'm taking a relatively safe path.

 

[depasseg]

Maybe it's because I'm on my phone, but I didn't see the instructions say "offline" anywhere. I saw "replace". I tried it quickly on a test system and it seemed to work. I think the failing drive adds an additional element of possible failure, but that is because it's a large stripe. And the only thing that can help that is your backup.

 

[selfsame]

Thanks again for the help here. I successfully replaced the hard drive with a larger drive and all went well. However, the system continues to hang after midnight, so I'll open a new thread about that.

Also, @depasseg, the first link you provided to 'replacing a failed drive' specifically mentions bringing the failed (failing) drive offline first. Replacing a non-failing disk does not require bringing the disk offline before replacing.