Replacing non-failed disk without resilver from parity

[Blokmeister]

Just a quick question here. I'm running an 8x3TB pool in raidz2. I have a drive that currently has unreadable sectors and gives errors at numbers 197 and 198 in the smartctl utility. I googled a bit and decided that it is time to replace it. However, since the drive is still operational and a resilver takes about 3 days non stop, I'd much rather just copy the data from the old drive to the new one without calculating the data from parity.

Is this possible, and if so, how do I do this?

 

[Bidule0hm]

Good question, never thought about that before but it should be possible to offline the pool and copy the drive's content byte per byte with dd. The biggest concern is the GPTID because you'll have drive with the same content but a different serial number and I have no idea if it'll have the same GPTID or not and if it can be a problem or not.

But 3 days of resilver with 8x 3 TB? there's a problem somewhere. How long is a scrub on your pool usually?

 

[diedrichg]

Clonezilla does bit:bit cloning.

 

[Blokmeister]

How long is a scrub on your pool usually?

How do I check this? I ran a scrub from the UI today, and after a few minutes, the menu disappeared. Does that mean it takes a few minutes?

 

[wblock]

Would a dd copy be any faster than a resilver? dd will have to copy the entire drive. A resilver probably will not.

And of course this assumes that the remaining drives will be used or modified during the time the dd copy is running, so the data will not change there.

 

[Bidule0hm]

Use the zpool status command. IIRC it's also displayed somewhere in the GUI but I don't remember where. No, it can't take minutes (unless the pool size is in the dozens of GB range).

Yep, a resilver only reconstruct the data so if the pool is half empty it'll only write half the drive ;)

 

[Blokmeister]

Use the zpool status command.

Wow, I just checked this and it's saying:

scrub in progress since Sun Aug 28 14:39:53 2016 (That is since that time GMT+1, or for 6 hours now.)
270G scanned out of 17.5T at 20.6M/s, 243h56m to go​

That is TEN DAYS! Could something be off?

 

[CraigD]

Not good

What hardware are you running?

 

[Blokmeister]

What hardware are you running?

Intel(R) Celeron(R) CPU G530 @ 2.40GHz
ASRock B75 Pro3-M
7x Western Digital Green WD30EZRX, 3TB
1x WD Red 4TB (as replacement of failed WD Geen)
2x Kingston KVR1333D3N9/8G

 

[Bidule0hm]

The speed usually go higher as the scrub is progressing but 20 MB/s is very low, even at the start.

Are the drives directly connected to the MB?

 

[Blokmeister]

Are the drives directly connected to the MB?

Yes they are.

I ran the zpool status command again, and it said this:

scrub in progress since Sun Aug 28 14:39:53 2016
1.70T scanned out of 17.5T at 81.8M/s, 56h16m to go
​

The CPU is hovering at around 10% usage and the powerd deamon is on. All disks are being read at about 30MB/s.

 

[joeschmuck]

That is an interesting thought about using "dd" or similar to just copy the data. Maybe someone has a drive they want to replace and give it a shot and report back the results. I'm a bit skeptical.

 

[Bidule0hm]

That's better. Well, we'll see how it goes I guess ;)

 

[Arwen]

If I understand ZFS correctly, (and I may not), you can use the Zpool replace command;

Code:

zpool  replace  pool  old_device  new_device

This requires a free slot for the replacement disk and seems to cause the new disk to be a
temporary mirror of the old device. Then whence the replacement is complete, the old
device is broken off. Then pool is as it was before the replacement, (but no bad blocks).

In theory, no parity calculations required, except for the few bad blocks.

Doing a replacement this way helps reduce the impact to the rest of the disks. Data is
only read from the old / failing disk, unless bad blocks need recovery.

I really like the idea since you still have close to 100% redundancy during replacement.
Ideally ZFS would recover the checksumed errored blocks first to the new disk, restoring
redundancy as quickly as possible. But, that's probably too much to ask for, (and too
difficult to program).

 

[SweetAndLow]

You are correct but don't use the cli with freenas. This can all be done via the GUI.

Sent from my Nexus 5X using Tapatalk

 

[Stux]

The speed usually go higher as the scrub is progressing but 20 MB/s is very low, even at the start.

Are the drives directly connected to the MB?

I've had a scrub slow down to 3MB/s with a a flaky disk.

I plan to replace the disk as soon as I work out which one it is.

 

[Blokmeister]

That's better. Well, we'll see how it goes I guess ;)

Well, I just checked it, and it still needs 25 hours. I think that's still a very long time.

I also have a system where the block size is not equal to the native block size (512B instead of 4096B). I don't know if this will affect performance and if yes, by how much.

 

[Bidule0hm]

Well, I just checked it, and it still needs 25 hours. I think that's still a very long time.

I also have a system where the block size is not equal to the native block size (512B instead of 4096B). I don't know if this will affect performance and if yes, by how much.

Yep, on my system with a 13 TB pool filled with about 3.5 TB of data a scrub takes a bit more than 2 hours.

Yes, it'll affect perfs if the pool is set to 512 B blocks and the drives have 4 kB sectors.

 

[danb35]

zpool replace pool old_device new_device

Isn't that what the GUI does behind the scenes for a disk replacement? I'd be very surprised if it were otherwise. And if this is what the GUI does, having recently replaced two disks in-place (i.e., without removing the old disks first), I can say from recent experience that no, it doesn't simply copy over the data from the disk being replaced--there was lots of activity on the other five disks in the vdev too.

 

[rs225]

I have also seen read activity on all disks(including the one being replaced) during an online replace.

Using dd instead should work just fine and has good odds of being faster. However, there is a chance you will have to export/reimport the pool to get it to accept the image. (It may refuse to online/replace with it)