New drive faulted during resliver

[Bill McCormick]

I'm working on expanding a zpool, replacing smaller drives with larger ones. I'm following the instructions from 8.1.11. Replacing Drives to Grow a ZFS Pool.

I connected a docking device from a USB3 port, did the replace from the GUI, and watched the process start reslivering. Somewhere in the process the drive started getting errors, and the manual doesn't cover what happens if.

What should be my next step(s)? Can I go back to the old drive? Should I have done some sort of testing on the drive before initiating the replace?

[bill@freenas] ~# zpool status z1
pool: z1
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Nov 10 19:45:37 2015
1.36T scanned out of 4.09T at 214M/s, 3h43m to go
69.8G resilvered, 33.22% done
config:

NAME STATE READ WRITE CKSUM
z1 DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
gptid/87749786-6b1b-11e5-83d2-d050994770a0 ONLINE 0 0 0
gptid/d51e3c72-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
gptid/d578b5c4-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
gptid/d5e49cd0-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
gptid/d644795b-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
gptid/d6bcd58e-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
gptid/d71b1b5c-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
replacing-7 DEGRADED 0 0 0
gptid/d7802973-697c-11e5-b4e6-d050994770a0 ONLINE 0 0 0
gptid/ebed07ca-8810-11e5-82b3-d050994770a0 FAULTED 0 884 0 too many errors (resilvering)

errors: No known data errors

 

[SweetAndLow]

You should probably offline the drive you just put in then put the old drive in and replace it.

Then don't use USB external drives again. Next figure out how to backup all your data so you can rebuild your pool as raidz2, you have way too many disks for a raidz1 pool.

 

[Bill McCormick]

The old drive is actually still physically connected. What are the implications of letting it finish?

Is there typically some prep (like wipe or something) that I should have done to test the drive before starting the replace?

It is raidz2 already. I need to put together a sig with my setup.

 

[Bill McCormick]

... also, is the Detach GUI function from Volume Status the correct method to offline a drive?

 

[Robert Trevellyan]

don't use USB external drives again

+1, especially USB3.

Is there typically some prep (like wipe or something) that I should have done to test the drive before starting the replace?

Yes, do a forum search for badblocks.

is the Detach GUI function from Volume Status the correct method to offline a drive?

NO!

I don't even see a Detach button on the Volume Status screen. There is a Detach button on the Storage screen, and it's not what you want.

Look for the Offline button, which appears when you select a drive on the Volume Status screen. See section 8.1.10 of the manual.

The old drive is actually still physically connected. What are the implications of letting it finish?

Good question. My guess is waiting for the resilver to finish might be the best thing right now. Anything else might just be digging a deeper hole.

 

[Bill McCormick]

I see that there there is no Offline button since thr drive is technically NOT Online. Option buttons are Detach and Replace.

Is it likely that the errors were caused by using USB3?

 

[Bill McCormick]

I ended up detaching before it said it was complete. The detach was successful and everything is healthy again.

 

[SweetAndLow]

Is it likely that the errors were caused by using USB3?

Absolutely! Don't use any kind of USB device for storage disks.

 

[Robert Trevellyan]

Option buttons are Detach and Replace.

Sorry, my mistake.

The detach was successful and everything is healthy again.

Great!

 

[Bill McCormick]

My DIY build does not have any eSATA port that I can connect a new drive to. This was an oversight on my part, but I think I'm stuck now since my only PCIe port is used by my SAS 9341-4i. So in order to replace a drive, what I'm currently faced with is opening op the system and connecting the drive to an unused SAT port on the MB (there are 2). Or, take a drive offline and put in the new one. Perhaps there are other options I'm not aware of that might allow me take advantage of 2 unused SATA ports on the MB?

A bit of history on this build: My 1st NAS build of any kind. I was going for very low profile and low power. I chose a SilverStone DS380 case and wanted to fill up all 8 bays with a RaidZ2 configuration. The MB only offered 6 SATA ports, so I needed more. I choose to go with the SAS 9341-4i because it offered the JBOD option for 4 drives, and this left 2 free MB SATA ports in the event I wanted to add SSD's for write cache.

 

[Robert Trevellyan]

in order to replace a drive, what I'm currently faced with is opening op the system and connecting the drive to an unused SAT port on the MB (there are 2)

This sounds like the way to go. You can physically connect 2 new drives, run the replacement process twice, then physically remove 2 old drives and add 2 more new drives. You will only have to go inside the case 5 times.

Make sure you track serial #s carefully to avoid removing the wrong drive at any stage.

If you want to be super cautious, do one physical removal at a time and test in-between. However, with RAIDZ2 your pool can survive the loss of 2 drives, so if you screw up you can recover even if you remove two drives.

Perhaps there are other options I'm not aware of that might allow me take advantage of 2 unused SATA ports on the MB?

You can get a bracket that converts 2 internal ports to 2 eSATA ports, then use a drive dock. If you go this way, choose a drive dock that doesn't have a port multiplier. You can tell this by how many eSATA ports it has, e.g. if there are 2 drive bays and 2 eSATA ports, theres no port multiplier.

 

[joeschmuck]

Also, you can remove the failed drive and just install the new drive and then go through the resilvering process. This is what you have to do if you do not have any available SATA ports and if the failed drive is bad, I say just pull it. But this is entirely up to you.

 

[TheDubiousDubber]

Also, you can remove the failed drive and just install the new drive and then go through the resilvering process. This is what you have to do if you do not have any available SATA ports and if the failed drive is bad, I say just pull it. But this is entirely up to you.

I was kind of curious why this wasn't done in the first place, or suggested earlier on. If the drive is bad, then what reason is there for leaving it in the system? Is there some reason behind resilvering using an eSATA dock prior to replacing?

 

[Bill McCormick]

Thanks Joe, I think that was one of the options I saw, but I don't actually have a failed drive at this point. I don't really want to unnecessarily degrade my pool, so I'm going to get a SATA to eSATA converter/bracket and take care of it like that. I have a single port docking station already with eSATA connection.

 

[Bill McCormick]

I was kind of curious why this wasn't done in the first place, or suggested earlier on. If the drive is bad, then what reason is there for leaving it in the system? Is there some reason behind resilvering using an eSATA dock prior to replacing?

The drive is not bad. Firstly, I want to make I know how to do this and have all the pieces in place. Second, I want to increase my pool space, replacing smaller drives with lager ones.

 

[TheDubiousDubber]

The drive is not bad. Firstly, I want to make I know how to do this and have all the pieces in place. Second, I want to increase my pool space, replacing smaller drives with lager ones.

I had the wrong impression of what was going on here and conveniently forgot the fact that you were replacing smaller drives not a failed drive. Anyhow, I have always been under the impression that to upgrade the size it is conventional to replace one drive at a time and resilver, then repeat until all the smaller drives are replaced. That is clearly only one way to do it. Either way, hope you get it all figured out.

 

[joeschmuck]

The drive is not bad.

Sorry, I lost that too even though I know I read that earlier. But still, you have a RAIDZ2 setup on that pool so removing a single drive and installing a larger drive is still not a terrible risk if you do not have the extra SATA port.