Help with DEGRADED multipath

Status
Not open for further replies.

Tomer

Dabbler
Joined
Jun 26, 2017
Messages
17
hi,
i need some help replacing a bad disk.

in GUI i can see multipath/disk11 in degraded status..
degraded.jpg


the command:
gmultipath status:
gmultipath status
Name Status Components
multipath/disk24 OPTIMAL da48 (ACTIVE)
da47 (PASSIVE)
multipath/disk23 OPTIMAL da46 (ACTIVE)
da45 (PASSIVE)
multipath/disk22 OPTIMAL da44 (ACTIVE)
da43 (PASSIVE)
multipath/disk21 OPTIMAL da42 (ACTIVE)
da21 (PASSIVE)
multipath/disk19 OPTIMAL da41 (ACTIVE)
da20 (PASSIVE)
multipath/disk19-65239471 OPTIMAL da40 (ACTIVE)
da19 (PASSIVE)
multipath/disk18 OPTIMAL da39 (ACTIVE)
da18 (PASSIVE)
multipath/disk17 OPTIMAL da38 (ACTIVE)
da17 (PASSIVE)
multipath/disk16 OPTIMAL da37 (ACTIVE)
da16 (PASSIVE)
multipath/disk15 OPTIMAL da36 (ACTIVE)
da15 (PASSIVE)
multipath/disk14 OPTIMAL da35 (ACTIVE)
da14 (PASSIVE)
multipath/disk13 OPTIMAL da34 (ACTIVE)
da13 (PASSIVE)
multipath/disk12 OPTIMAL da33 (ACTIVE)
da12 (PASSIVE)
multipath/disk11 DEGRADED da32 (ACTIVE)
da11 (FAIL)
multipath/disk10 OPTIMAL da31 (ACTIVE)
da10 (PASSIVE)
multipath/disk9 OPTIMAL da30 (ACTIVE)
da9 (PASSIVE)
multipath/disk8 OPTIMAL da29 (ACTIVE)
da8 (PASSIVE)
multipath/disk7 OPTIMAL da28 (ACTIVE)
da7 (PASSIVE)
multipath/disk6 OPTIMAL da27 (ACTIVE)
da6 (PASSIVE)
multipath/disk5 OPTIMAL da26 (ACTIVE)
da5 (PASSIVE)
multipath/disk4 OPTIMAL da25 (ACTIVE)
da4 (PASSIVE)
multipath/disk3 OPTIMAL da24 (ACTIVE)
da3 (PASSIVE)
multipath/disk2 OPTIMAL da23 (ACTIVE)
da2 (PASSIVE)
multipath/disk1 OPTIMAL da22 (ACTIVE)
da1 (PASSIVE)

what are the steps to correctly replace that disk?

please help.
 

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457
It may not be the disk. As one path the that disk is up and working, Its would seem as likely to be a bad port, cable, HBA, or expander. We need to identify the disk and cable path in question. Do you have your drive bays labeled by disk serial number?
 

Tomer

Dabbler
Joined
Jun 26, 2017
Messages
17
It may not be the disk. As one path the that disk is up and working, Its would seem as likely to be a bad port, cable, HBA, or expander. We need to identify the disk and cable path in question. Do you have your drive bays labeled by disk serial number?


Hi.
I have 20+ disk in my bay, so labaling is preety hard..
Anyway i can identify bay11 via led on or off..
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Hi.
I have 20+ disk in my bay, so labaling is preety hard..
Anyway i can identify bay11 via led on or off..
If that works for you. Some people have had difficulty with getting that function to work. Would you be so kind as to share your hardware configuration with us? It is generally best if we have some idea what we are working with when we offer suggestions.

There is also this script that may help you identify drives if there is any question:

Utility: disklist.pl, for listing partition, gptid, slot, devices, disktype, serial num, & multipath
https://forums.freenas.org/index.ph...ktype-serial-num-multipath.59319/#post-421424
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
hi,
i need some help replacing a bad disk.

in GUI i can see multipath/disk11 in degraded status..
If you are sure that you want to replace the disk, this resource might help with the steps involved.

Replacing a failed/failing disk
https://forums.freenas.org/index.php?resources/replacing-a-failed-failing-disk.75/

Once you are certain you have identified the correct physical disk, you would offline it through the GUI. It is important that you are sure that you are pulling out the correct physical disk as we had a user this past week that removed the wrong disk and ended up with two disks out of the pool when their pool was only designed to support one failure. They were using mirrors.

It is generally best if you tell us (in addition to the hardware) what your pool configuration is. It can shape our guidance to know details like that.
 

Tomer

Dabbler
Joined
Jun 26, 2017
Messages
17
Hi all. thanks for supporting:)
my hardware is -hp-dl360G7 with 2X HBA adapter's and a old netapp draw.. containing 24disks(model NAJ-0801) FreeNAS-11.0-RELEASE (a2dc21583)

thanks for the script: the output..
root@freenas:/script # ./disklist.pl
partition zpool device disk size serial rpm
---------------------------------------------------------------------------------------------------------------
multipath/disk1p2 ESXI-Storage da22,da1 WDC WD2003FYYS-05TSM 2000 WD-WMAY02597305 7200
multipath/disk2p2 ESXI-Storage da2,da23 WDC WD2003FYYS-05TSM 2000 WD-WMAY02598408 7200
multipath/disk18p2 ESXI-Storage da18,da39 WDC WD2003FYYS-05TSM 2000 WD-WMAY02578689 7200
multipath/disk22p2 ESXI-Storage da44,da43 HGST HUS726020ALE61SM 2000 N4GBJS1S 7200
multipath/disk23p2 ESXI-Storage da45,da46 HGST HUS726020ALE61SM 2000 N4G9241Y 7200
multipath/disk24p2 ESXI-Storage da48,da47 HGST HUS726020ALE61SM 2000 N4G929JY 7200
multipath/disk19-65239471p2 ESXI-Storage da19,da40 WDC WD2003FYYS-05TSM 2000 WD-WMAY02597072 7200
multipath/disk4p2 General-Storage da25,da4 WDC WD2003FYYS-05TSM 2000 WD-WMAY02702451 7200
multipath/disk5p2 General-Storage da26,da5 HITACHI HUA722020ALA33SM 2000 BFGZABTF 7200
multipath/disk6p2 General-Storage da27,da6 HITACHI HUA722020ALA33SM 2000 BFH35EJF 7200
multipath/disk7p2 General-Storage da7,da28 HITACHI HUA722020ALA33SM 2000 BFH35E0F 7200
multipath/disk8p2 General-Storage da8,da29 WDC WD2003FYYS-05TSM 2000 WD-WMAY02576791 7200
multipath/disk9p2 General-Storage da30,da9 WDC WD2003FYYS-05TSM 2000 WD-WMAY02594984 7200
multipath/disk10p2 General-Storage da10,da31 WDC WD2003FYYS-05TSM 2000 WD-WMAY02493363 7200
multipath/disk11p2 Hyper-V da11,da32 WDC WD2000FYYZ-05USA 2000 WD-WCC1P1047027 7200
multipath/disk12p2 Hyper-V da33,da12 WDC WD2003FYYS-05TSM 2000 WD-WMAY02596374 7200
multipath/disk13p2 Hyper-V da34,da13 WDC WD2003FYYS-05TSM 2000 WD-WMAY02600165 7200
multipath/disk14p2 Hyper-V da14,da35 WDC WD2003FYYS-05TSM 2000 WD-WMAY02596080 7200
multipath/disk15p2 Hyper-V da15,da36 WDC WD2003FYYS-05TSM 2000 WD-WMAY02594361 7200
multipath/disk16p2 Hyper-V da16,da37 WDC WD2003FYYS-05TSM 2000 WD-WMAY02594001 7200
multipath/disk17p2 Hyper-V da17,da38 WDC WD2003FYYS-05TSM 2000 WD-WMAY02702230 7200
cd0 hp DVD D DS8D3SH 0 (null) ???
da0 HP RAID 0 250 5001438020EB4F80 ???
da24 HITACHI HUA722020ALA33SM 2000 BFH359UF 7200
da3 HITACHI HUA722020ALA33SM 2000 BFH359UF 7200
da41 HGST HUS726020ALE61SM 2000 N4G92KJY 7200
da20 HGST HUS726020ALE61SM 2000 N4G92KJY 7200
da42 WDC WD2003FYYS-05TSM 2000 WD-WMAY02597920 7200
da21 WDC WD2003FYYS-05TSM 2000 WD-WMAY02597920 7200


also gmultipath status shows:


Name Status Components
multipath/disk24 OPTIMAL da48 (ACTIVE)
da47 (PASSIVE)
multipath/disk23 OPTIMAL da46 (ACTIVE)
da45 (PASSIVE)
multipath/disk22 OPTIMAL da44 (ACTIVE)
da43 (PASSIVE)
multipath/disk21 OPTIMAL da42 (ACTIVE)
da21 (PASSIVE)
multipath/disk19 OPTIMAL da41 (ACTIVE)
da20 (PASSIVE)
multipath/disk19-65239471 OPTIMAL da40 (ACTIVE)
da19 (PASSIVE)
multipath/disk18 OPTIMAL da39 (ACTIVE)
da18 (PASSIVE)
multipath/disk17 OPTIMAL da38 (ACTIVE)
da17 (PASSIVE)
multipath/disk16 OPTIMAL da37 (ACTIVE)
da16 (PASSIVE)
multipath/disk15 OPTIMAL da36 (ACTIVE)
da15 (PASSIVE)
multipath/disk14 OPTIMAL da35 (ACTIVE)
da14 (PASSIVE)
multipath/disk13 OPTIMAL da34 (ACTIVE)
da13 (PASSIVE)
multipath/disk12 OPTIMAL da33 (ACTIVE)
da12 (PASSIVE)
multipath/disk11 DEGRADED da32 (ACTIVE)
da11 (FAIL)

multipath/disk10 OPTIMAL da31 (ACTIVE)
da10 (PASSIVE)
multipath/disk9 OPTIMAL da30 (ACTIVE)
da9 (PASSIVE)
multipath/disk8 OPTIMAL da29 (ACTIVE)
da8 (PASSIVE)
multipath/disk7 OPTIMAL da28 (ACTIVE)
da7 (PASSIVE)
multipath/disk6 OPTIMAL da27 (ACTIVE)
da6 (PASSIVE)
multipath/disk5 OPTIMAL da26 (ACTIVE)
da5 (PASSIVE)
multipath/disk4 OPTIMAL da25 (ACTIVE)
da4 (PASSIVE)
multipath/disk3 OPTIMAL da24 (ACTIVE)
da3 (PASSIVE)
multipath/disk2 OPTIMAL da23 (ACTIVE)
da2 (PASSIVE)
multipath/disk1 OPTIMAL da22 (ACTIVE)
da1 (PASSIVE)


sas info:

root@freenas:/script # sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.


Adapter Vendor Device SubSys SubSys
Index Type ID ID Pci Address Ven ID Dev ID
----- ------------ ------ ------ ----------------- ------ ------
0 SAS2008 1000h 72h 00h:06h:00h:00h 103ch 3371h
SAS2IRCU: Utility Completed Successfully.

root@freenas:/script # sas3ircu list
Avago Technologies SAS3 IR Configuration Utility.
Version 15.00.00.00 (2016.11.21)
Copyright (c) 2009-2016 Avago Technologies. All rights reserved.




any help resolving that issue will be great...if any more info needed it is not a problem...thanks.
 
Joined
Jul 3, 2015
Messages
926
In my experience of multipathed systems then its quite common to see one path fail as a result of an impending disk failure. At this time I would assume its the drive as thats the easiest and replace it. If however the problem should re-appear with a new drive in the same location then that should start sending alarm bells.
 

Tomer

Dabbler
Joined
Jun 26, 2017
Messages
17
well can anyone advice for the correct steps to replace the bad disk?
 
Joined
Jul 3, 2015
Messages
926
Just noticed this in one of your outputs:

multipath/disk19 OPTIMAL da41 (ACTIVE)
da20 (PASSIVE)
multipath/disk19-65239471 OPTIMAL da40 (ACTIVE)
da19 (PASSIVE)

This looks like like two multipathed devices have been given the same name albeit with a small difference '-65239471'.

Its a bit late now for this system but I think its always worth naming your multipath devices before building your pool and giving them a better naming scheme reflecting their physical location in the system. So for example on your system with one JBOD 'J1S5'. FreeNAS will auto name the multipath device disk1, disk2, etc but you can do the following:

gmultipath destroy disk1
gmultipath label -v J1S5 /dev/da4 /da27

Do this BEFORE you build your pool and then once all the devices are named you can create the pool from the UI as you would normally.

Now based on the initial output you gave at the top of the thread you would know without any guess work that 'J1S5' is the drive with the issue. Offline the drive, remove and replace. Then name your new device using its associated da numbers (which are often the same as the last but do check to be sure) gmultipath label -v J1S5 /dev/da4 /da27 then replace the drive via the UI and all your names are back to good again.

Its worth noting that these drive names are stored on the drive itself so its not a god idea to move drives from one bay to another without renaming the drives otherwise you'll get very confused.
 
Joined
Jul 3, 2015
Messages
926
well can anyone advice for the correct steps to replace the bad disk?
Like @Chris Moore said, offline the drive via the UI, physically remove it and replace it with a new one. Then via the UI select the off-lined drive and click replace selecting your newly installed drive.
 

Tomer

Dabbler
Joined
Jun 26, 2017
Messages
17
ok,so i replaced the faulty drive..
ill wait and see if any new error on that drive will pop up again..its may be the disk netapp draw?
thanks everyone for support.
 
Status
Not open for further replies.
Top