Hi all,
Help! We currently have a spectacular zpool status which contains the following:
We attempted to replace the disk gptid 6295ae4e-2bbe (/dev/da8) using the GUI "Replace" button.
We think this has happened because:
- we have turned on autoreplace for hot spares with zpool set autoreplace=on tank
- we didn't offline the disk before selecting "replace" on the FreeNAS GUI.
replacing-0 is the result of a disk replace command in FreeNAS GUI of disk gptid 6295ae4e-2bbe (/dev/da8) with an available disk gptid 244f1f47-c9d8 (da22)
Around two hours later, we got this in an alert email: "Device: /dev/da8 [SAT], 65527 Currently unreadable (pending) sectors". So it seems ZFS has detected da8 as failed and pulled in the spare with gpt-id 81874caf-2bdf (da21) during a replacement resilver. da8 was marked as "too many errors" in zfs status which did not appear before this point.
The resilver has now finished, and da8 is now showing as online. We cannot detach or offline da8 in the GUI. Maybe it is because 'replacing' is still happening, despite the resilver showing as finished?
We are wondering what people might think is safest way to get this back to a nice 3 way mirror, maybe by removing all of these disks with checksum errors and doing a zpool attach of a freshly formatted disk. What are the risks of all of these checksum errors, and that it tried to resilver from a bad disk?
We are running a Supermicro server with some Hitachi 3TB and a WD Black 4TB in each mirror, FreeNAS-9.10.2-U3 (e1497f269) Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz Memory 98241MB of enterprise ECC RAM
Thanks!
Michael
Help! We currently have a spectacular zpool status which contains the following:
Code:
state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://illumos.org/msg/ZFS-8000-9P scan: resilvered 4.47T in 17h31m with 0 errors on Thu Nov 16 14:28:25 2017 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 ... mirror-3 DEGRADED 0 0 0 replacing-0 DEGRADED 0 0 111 spare-0 DEGRADED 0 0 95 gptid/6295ae4e-2bbe-11e7-a503-002590c8d294 DEGRADED 0 0 157 too many errors gptid/81874caf-2bdf-11e7-a503-002590c8d294 ONLINE 0 0 266 gptid/244f1f47-c9d8-11e7-ab2e-002590c8d294 ONLINE 0 0 311 gptid/0e5ae824-2bda-11e7-a503-002590c8d294 ONLINE 0 0 0 gptid/6402f3f5-2bbe-11e7-a503-002590c8d294 ONLINE 0 0 0 spares 5571117140374537679 INUSE was /dev/gptid/81874caf-2bdf-11e7-a503-002590c8d294 .. errors: No known data errors
We attempted to replace the disk gptid 6295ae4e-2bbe (/dev/da8) using the GUI "Replace" button.
We think this has happened because:
- we have turned on autoreplace for hot spares with zpool set autoreplace=on tank
- we didn't offline the disk before selecting "replace" on the FreeNAS GUI.
replacing-0 is the result of a disk replace command in FreeNAS GUI of disk gptid 6295ae4e-2bbe (/dev/da8) with an available disk gptid 244f1f47-c9d8 (da22)
Around two hours later, we got this in an alert email: "Device: /dev/da8 [SAT], 65527 Currently unreadable (pending) sectors". So it seems ZFS has detected da8 as failed and pulled in the spare with gpt-id 81874caf-2bdf (da21) during a replacement resilver. da8 was marked as "too many errors" in zfs status which did not appear before this point.
The resilver has now finished, and da8 is now showing as online. We cannot detach or offline da8 in the GUI. Maybe it is because 'replacing' is still happening, despite the resilver showing as finished?
We are wondering what people might think is safest way to get this back to a nice 3 way mirror, maybe by removing all of these disks with checksum errors and doing a zpool attach of a freshly formatted disk. What are the risks of all of these checksum errors, and that it tried to resilver from a bad disk?
We are running a Supermicro server with some Hitachi 3TB and a WD Black 4TB in each mirror, FreeNAS-9.10.2-U3 (e1497f269) Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz Memory 98241MB of enterprise ECC RAM
Thanks!
Michael