Hi. This pertains to Build FreeNAS-9.3-STABLE-201512121950. (Yes we should upgrade, but cannot right now.)
On our NAS-Supermicro Model 847-12 there is a degraded multipath disk in a mirror. This multipath disk multipath/disk4 is a member of mirror0
It's operating in degraded mode. Both disks in this mirror are online but one disk has a bad segment (56). SMART tests increment the error count past a threshold to cause the alert and thus degraded. I need to replace this disk. The multipath is comprised of /dev/da34 (active) and /dev/da77 (fail); I know which physical disk this is.
Usually when multipath disks failed before I could offline and replace them; and the Resilvering would run OK.
This disk multipath/disk4 is giving me issues. Previous Offline attempts all Failed and the disk would remain online. Now it is in degraded mode and the option to Offline this disk is not even presented in the FreeNAS GUI.
I'm thinking the only way to resolve this, is properly shutdown the NAS, replace the drive while off, then restart. The mirror0 should initiate the Resilvering? I am not certain.
I have not yet attempted to run: (Should this work? ....below)
# zpool offline poolname /dev/multipath/whatever
ie: # zpool offline vol0 /dev/multipath//disk4
or ie: # zpool offline vol0 /dev/multipath//disk4p2
?
then # reboot
then
# zpool replace poolname oldidhere /dev/multipath/whatever
Is oldidhere = eaa9c650-ffb0-11e5-b656-002590c509e6 ?
ie: # zpool replace vol0 eaa9c650-ffb0-11e5-b656-002590c509e6 /dev/multipath/disk4
info from command output:
# zpool status -v |more
[root@Stor02 ~]# gmultipath status -s |more
(all multipath pairs Active/Passive except for...)
multipath/disk4 DEGRADED da77 (FAIL)
multipath/disk4 DEGRADED da34 (ACTIVE)
and
# smartctl -a /dev/da34
(same for # smartctl -a /dev/da77)
Is a shutdown needed to replace this disk, or do you suggest otherwise?
Thanks for your help.
On our NAS-Supermicro Model 847-12 there is a degraded multipath disk in a mirror. This multipath disk multipath/disk4 is a member of mirror0
It's operating in degraded mode. Both disks in this mirror are online but one disk has a bad segment (56). SMART tests increment the error count past a threshold to cause the alert and thus degraded. I need to replace this disk. The multipath is comprised of /dev/da34 (active) and /dev/da77 (fail); I know which physical disk this is.
Usually when multipath disks failed before I could offline and replace them; and the Resilvering would run OK.
This disk multipath/disk4 is giving me issues. Previous Offline attempts all Failed and the disk would remain online. Now it is in degraded mode and the option to Offline this disk is not even presented in the FreeNAS GUI.
I'm thinking the only way to resolve this, is properly shutdown the NAS, replace the drive while off, then restart. The mirror0 should initiate the Resilvering? I am not certain.
I have not yet attempted to run: (Should this work? ....below)
# zpool offline poolname /dev/multipath/whatever
ie: # zpool offline vol0 /dev/multipath//disk4
or ie: # zpool offline vol0 /dev/multipath//disk4p2
?
then # reboot
then
# zpool replace poolname oldidhere /dev/multipath/whatever
Is oldidhere = eaa9c650-ffb0-11e5-b656-002590c509e6 ?
ie: # zpool replace vol0 eaa9c650-ffb0-11e5-b656-002590c509e6 /dev/multipath/disk4
info from command output:
# zpool status -v |more
Code:
[root@Stor02 ~]# zpool status -v |more pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0h1m with 0 errors on Tue Dec 14 03:46:07 2021 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 ada0p2 ONLINE 0 0 0 da94 ONLINE 0 0 0 errors: No known data errors pool: vol0 state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 0 in 23h45m with 7 errors on Sun Jan 9 23:45:11 2022 config: NAME STATE READ WRITE CKSUM vol0 DEGRADED 51 0 0 mirror-0 DEGRADED 51 0 0 gptid/5d566609-27ce-11ec-84b5-002590c509e6 ONLINE 0 0 81 gptid/eaa9c650-ffb0-11e5-b656-002590c509e6 DEGRADED 51 0 0 too many errors mirror-1 ONLINE 0 0 0 gptid/edfc912f-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/f145a8e2-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 gptid/f492aa93-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/f7ea91a6-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 gptid/fb3fe2b6-ffb0-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/7c22dd1f-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 gptid/01ed7c10-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/0539f157-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 gptid/0888391d-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/a4389c73-704e-11e8-a0a8-002590c509e6 ONLINE 0 0 0 mirror-6 ONLINE 0 0 0 gptid/0f2af859-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/127bb4bb-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-7 ONLINE 0 0 0 gptid/15c9e607-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/191c892c-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-8 ONLINE 0 0 0 gptid/1c750efe-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/1fc769a9-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-9 ONLINE 0 0 0 gptid/231b1712-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/597dd6ec-2d5d-11ec-84b5-002590c509e6 ONLINE 0 0 0 mirror-10 ONLINE 0 0 0 gptid/29c3294e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/2d0e73e2-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-11 ONLINE 0 0 0 gptid/30618e1f-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/33b46d23-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-12 ONLINE 0 0 0 gptid/370d8805-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/3a5bfb7e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-13 ONLINE 0 0 0 gptid/3db28a63-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/41050ec0-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-14 ONLINE 0 0 0 gptid/445dcad5-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/47af47b0-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-15 ONLINE 0 0 0 gptid/4b08ff92-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/4e56a93e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-16 ONLINE 0 0 0 gptid/51b37458-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/55079950-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-17 ONLINE 0 0 0 gptid/5866ad7e-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/5bb9c885-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-18 ONLINE 0 0 0 gptid/5f191ef4-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/626a2a6c-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-19 ONLINE 0 0 0 gptid/65cdc22c-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/693b38b6-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-20 ONLINE 0 0 0 gptid/85f3101e-dec5-11eb-b60e-002590c509e6 ONLINE 0 0 0 gptid/706a2f6b-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 mirror-21 ONLINE 0 0 0 gptid/740a2f89-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/779af1d9-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 logs mirror-22 ONLINE 0 0 0 gptid/78209949-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 gptid/7888e2b8-ffb1-11e5-b656-002590c509e6 ONLINE 0 0 0 cache gptid/1e865cd9-097c-11e6-80b7-002590c509e6 ONLINE 0 0 0 gptid/1ee3d919-097c-11e6-80b7-002590c509e6 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /mnt/vol0/stor02/stor02_ext0 /mnt/vol0/stor02/stor02_ext1 <0x2cf9>:<0xb> [root@Stor02 ~]#
[root@Stor02 ~]# gmultipath status -s |more
(all multipath pairs Active/Passive except for...)
multipath/disk4 DEGRADED da77 (FAIL)
multipath/disk4 DEGRADED da34 (ACTIVE)
and
# smartctl -a /dev/da34
(same for # smartctl -a /dev/da77)
Code:
[root@Stor02 ~]# smartctl -a /dev/da34 smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build) Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Vendor: WD Product: WD4001FYYG Revision: D1R5 Compliance: SPC-4 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Logical block size: 512 bytes Rotation Rate: 7200 rpm Form Factor: 3.5 inches Logical Unit id: 0x50000c0f01376374 Serial number: WMC1F0D76TAH Device type: disk Transport protocol: SAS (SPL-3) Local Time is: Tue Jan 11 19:38:36 2022 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled Temperature Warning: Disabled or Not Supported === START OF READ SMART DATA SECTION === SMART Health Status: OK Current Drive Temperature: 31 C Drive Trip Temperature: 64 C Manufactured in week 03 of year 2015 Specified cycle count over device lifetime: 1048576 Accumulated start-stop cycles: 15 Specified load-unload count over device lifetime: 1114112 Accumulated load-unload cycles: 31128 Elements in grown defect list: 1 Error counter log: Errors Corrected by Total Correction Gigabytes Total ECC rereads/ errors algorithm processed uncorrected fast | delayed rewrites corrected invocations [10^9 bytes] errors read: 164959416 19 792978 164959435 2871 76450.387 2852 write: 1902521753 1 110219 1902521754 1 150089.871 0 Non-medium error count: 179 SMART Self-test log Num Test Status segment LifeTime LBA_first_err [SK ASC ASQ] Description number (hours) # 1 Background short Completed 48 47338 - [- - -] # 2 Background long Failed in segment --> 56 46665 968064559 [0x3 0x16 0x0] # 3 Background long Failed in segment --> 56 46659 968064559 [0x3 0x16 0x0] # 4 Background long Failed in segment --> 56 46598 968067335 [0x3 0x16 0x0] # 5 Background short Completed 48 46595 - [- - -] Long (extended) Self Test duration: 31120 seconds [518.7 minutes] [root@Stor02 ~]#
Is a shutdown needed to replace this disk, or do you suggest otherwise?
Thanks for your help.