drive replacement gone awry

[mdiesel]

Hey all,

I've replaced drives in my array numerous times (including expanding array by replacing with larger drives), however something has gone wrong this time, and I've been left with an array that, while functional, still shows the old drive (renamed) as a member, and I'm unsure as to the status of its replacement. A zpool status yields the following:

The "1563669..." device is the disk being replaced, and the "896a2..." device is the one replacing it. By the looks of this output, it almost looks like there's a sub-array ("replacing-3", currently degraded) within my primary array that contains the "bad" disk and the "good disk replacing it. I've tried detaching the bad drive, both via GUI and CLI, but no dice (I'm told "no valid replicas").

I've completed a scrub of the array, as well, which did not seem to help anything. Ideas?

 

[nightshade00013]

Hardware and software information per forum rules for support are needed.

 

[mdiesel]

Ah, my apologies:

• HP Proliant Microserver N40L (more)
  Turion II Neo X2 Dual Core L625
  16GB ECC RAM
  4 x 4TB SATA drives (HGST)
• Single ZFS volume, as shown above
• FreeNAS 11.1-RELEASE

 

[rs225]

Can you reconnect the old disk? A resilver or replace can not complete if you have top level errors. You may be able to use zpool status -v dieselnas1 to view the damaged data, and perhaps delete the files. Otherwise, if the errors can not be resolved, the pool will have to be abandoned.

 

[mdiesel]

Sadly, no, the replaced drive has already been re-commissioned elsewhere. I did, however, run zpool status -v, and found that there are two files that are damaged. If I simply delete those files, should that be enough to allow the re-silver to complete and "release" the replaced drive in the array?

(one of the files is completely unimportant, the other is my primary iSCSI extent file, but I can rsync that off elsewhere for the time being [if it'll let me] and copy it back once things have settled down)

 

[rs225]

Well, hopefully yes. Unless rsync can be instructed to skip over errors, it won't be able to copy the extent file beyond the error. You will need to use dd if=extent of=/new/location/extent.copy conv=noerror,sync,sparse Then you could rsync that copy somewhere else.

Any snapshots that contained those two files would also have to be dropped. Then, it might complete resilver. If not, you're looking at rebuilding the pool from scratch.