Hi. This pertains to Build FreeNAS-9.3-STABLE-201512121950. (Yes we should upgrade, but cannot right now.)
On our NAS-Supermicro Model 847-12 there is a degraded multipath disk in a mirror. This multipath disk multipath/disk4 is a member of mirror0
It's operating in degraded mode. Both disks in this mirror are online but one disk has a bad segment (56). SMART tests increment the error count past a threshold to cause the alert and thus degraded. I need to replace this disk. The multipath is comprised of /dev/da34 (active) and /dev/da77 (fail); I know which physical disk this is.
Usually when multipath disks failed before I could offline and replace them; and the Resilvering would run OK.
This disk multipath/disk4 is giving me issues. Previous Offline attempts all Failed and the disk would remain online. Now it is in degraded mode and the option to Offline this disk is not even presented in the FreeNAS GUI.
I'm thinking the only way to resolve this, is properly shutdown the NAS, replace the drive while off, then restart. The mirror0 should initiate the Resilvering? I am not certain.
I have not yet attempted to run: (Should this work? ....below)
# zpool offline poolname /dev/multipath/whatever
ie: # zpool offline vol0 /dev/multipath//disk4
or ie: # zpool offline vol0 /dev/multipath//disk4p2
?
then # reboot
then
# zpool replace poolname oldidhere /dev/multipath/whatever
Is oldidhere = eaa9c650-ffb0-11e5-b656-002590c509e6 ?
ie: # zpool replace vol0 eaa9c650-ffb0-11e5-b656-002590c509e6 /dev/multipath/disk4
info from command output:
# zpool status -v |more
[root@Stor02 ~]# gmultipath status -s |more
(all multipath pairs Active/Passive except for...)
multipath/disk4 DEGRADED da77 (FAIL)
multipath/disk4 DEGRADED da34 (ACTIVE)
and
# smartctl -a /dev/da34
(same for # smartctl -a /dev/da77)
Is a shutdown needed to replace this disk, or do you suggest otherwise?
Thanks for your help.
On our NAS-Supermicro Model 847-12 there is a degraded multipath disk in a mirror. This multipath disk multipath/disk4 is a member of mirror0
It's operating in degraded mode. Both disks in this mirror are online but one disk has a bad segment (56). SMART tests increment the error count past a threshold to cause the alert and thus degraded. I need to replace this disk. The multipath is comprised of /dev/da34 (active) and /dev/da77 (fail); I know which physical disk this is.
Usually when multipath disks failed before I could offline and replace them; and the Resilvering would run OK.
This disk multipath/disk4 is giving me issues. Previous Offline attempts all Failed and the disk would remain online. Now it is in degraded mode and the option to Offline this disk is not even presented in the FreeNAS GUI.
I'm thinking the only way to resolve this, is properly shutdown the NAS, replace the drive while off, then restart. The mirror0 should initiate the Resilvering? I am not certain.
I have not yet attempted to run: (Should this work? ....below)
# zpool offline poolname /dev/multipath/whatever
ie: # zpool offline vol0 /dev/multipath//disk4
or ie: # zpool offline vol0 /dev/multipath//disk4p2
?
then # reboot
then
# zpool replace poolname oldidhere /dev/multipath/whatever
Is oldidhere = eaa9c650-ffb0-11e5-b656-002590c509e6 ?
ie: # zpool replace vol0 eaa9c650-ffb0-11e5-b656-002590c509e6 /dev/multipath/disk4
info from command output:
# zpool status -v |more
Code:
[root@Stor02 ~]# zpool status -v |more
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Tue Dec 14 03:46:07 2021
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
da94 ONLINE 0 0 0
errors: No known data errors
pool: vol0
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 23h45m with 7 errors on Sun Jan 9 23:45:11 2022
config:
NAME STATE READ WRITE CKSUM
vol0 DEGRADED 51 0 0
mirror-0 DEGRADED 51 0 0
gptid/5d566609-27ce-11ec-84b5-002590c509e6 ONLINE 0 0 81
gptid/eaa9c650-ffb0-11e5-b656-002590c509e6 DEGRADED 51 0 0 too many errors
mirror-1 ONLINE 0 0 0
gptid/edfc912f-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/f145a8e2-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/f492aa93-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/f7ea91a6-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
gptid/fb3fe2b6-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/7c22dd1f-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
gptid/01ed7c10-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/0539f157-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
gptid/0888391d-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/a4389c73-704e-11e8-a0a8-002590c509e6 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
gptid/0f2af859-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/127bb4bb-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
gptid/15c9e607-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/191c892c-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
gptid/1c750efe-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/1fc769a9-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-9 ONLINE 0 0 0
gptid/231b1712-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/597dd6ec-2d5d-11ec-84b5-002590c509e6 ONLINE 0 0 0
mirror-10 ONLINE 0 0 0
gptid/29c3294e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/2d0e73e2-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-11 ONLINE 0 0 0
gptid/30618e1f-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/33b46d23-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-12 ONLINE 0 0 0
gptid/370d8805-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/3a5bfb7e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-13 ONLINE 0 0 0
gptid/3db28a63-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/41050ec0-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-14 ONLINE 0 0 0
gptid/445dcad5-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/47af47b0-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-15 ONLINE 0 0 0
gptid/4b08ff92-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/4e56a93e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-16 ONLINE 0 0 0
gptid/51b37458-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/55079950-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-17 ONLINE 0 0 0
gptid/5866ad7e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/5bb9c885-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-18 ONLINE 0 0 0
gptid/5f191ef4-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/626a2a6c-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-19 ONLINE 0 0 0
gptid/65cdc22c-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/693b38b6-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-20 ONLINE 0 0 0
gptid/85f3101e-dec5-11eb-b60e-002590c509e6 ONLINE 0 0 0
gptid/706a2f6b-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
mirror-21 ONLINE 0 0 0
gptid/740a2f89-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/779af1d9-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
logs
mirror-22 ONLINE 0 0 0
gptid/78209949-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
gptid/7888e2b8-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0
cache
gptid/1e865cd9-097c-11e6-80b7-002590c509e6 ONLINE 0 0 0
gptid/1ee3d919-097c-11e6-80b7-002590c509e6 ONLINE 0 0 0
errors: Permanent errors have been detected in the following files:
/mnt/vol0/stor02/stor02_ext0
/mnt/vol0/stor02/stor02_ext1
<0x2cf9>:<0xb>
[root@Stor02 ~]#
[root@Stor02 ~]# gmultipath status -s |more
(all multipath pairs Active/Passive except for...)
multipath/disk4 DEGRADED da77 (FAIL)
multipath/disk4 DEGRADED da34 (ACTIVE)
and
# smartctl -a /dev/da34
(same for # smartctl -a /dev/da77)
Code:
[root@Stor02 ~]# smartctl -a /dev/da34
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: WD
Product: WD4001FYYG
Revision: D1R5
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Logical Unit id: 0x50000c0f01376374
Serial number: WMC1F0D76TAH
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Tue Jan 11 19:38:36 2022 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Disabled or Not Supported
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 31 C
Drive Trip Temperature: 64 C
Manufactured in week 03 of year 2015
Specified cycle count over device lifetime: 1048576
Accumulated start-stop cycles: 15
Specified load-unload count over device lifetime: 1114112
Accumulated load-unload cycles: 31128
Elements in grown defect list: 1
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 164959416 19 792978 164959435 2871 76450.387 2852
write: 1902521753 1 110219 1902521754 1 150089.871 0
Non-medium error count: 179
SMART Self-test log
Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ]
Description number (hours)
# 1 Background short Completed 48 47338 - [- - -]
# 2 Background long Failed in segment --> 56 46665 968064559 [0x3 0x16 0x0]
# 3 Background long Failed in segment --> 56 46659 968064559 [0x3 0x16 0x0]
# 4 Background long Failed in segment --> 56 46598 968067335 [0x3 0x16 0x0]
# 5 Background short Completed 48 46595 - [- - -]
Long (extended) Self Test duration: 31120 seconds [518.7 minutes]
[root@Stor02 ~]#
Is a shutdown needed to replace this disk, or do you suggest otherwise?
Thanks for your help.
