Unable to identify drive with sas2ircu

Status
Not open for further replies.
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
Hi, I inherited a Freenas Server 9.3 running on Supermicro server with a LSI SAS2008 card and 3 JBOD enclosures attached, 2 of the enclosures are 60 drive capacity and the other one is 48 drive capacity, with 8TB drives installed.

We've recently experienced a drive failure. I am unable to find the serial number in order to replace the drive from the GUI as well as the CLI (screenshots enclosed).

I am unable to retrieve the serial number through "smartctl" and "camcontrol". I tried using the LSI Logic's configuration tool "sas2ircu" with negative results.

I am at least trying to narrow down the location of the drive to the enclosure, the information from sas2ircu display is not accurate as there missing serial drive numbers, wrong enclosure information and and wrong slot numbers.

I am also getting this error when issuing the "sas2ircu 0 status" command.

[root@freenas01] /dev# sas2ircu 0 status
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

SAS2IRCU: The STATUS command is not supported by the firmware currently loaded on controller 0.
[root@freenas01] /dev#


Help,
jinno
 

Attachments

  • 2017-11-27_10-03-36.png
    2017-11-27_10-03-36.png
    99.2 KB · Views: 578
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Does the server that FreeNAS is running on have drives in it also?
We need to get a little more information on the topology of the SAS controller.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
The more information you can give us, the better possibility someone will be able to help you.
Can you SSH into a console or are you using the web interface?
If you can, please give us the results of zpool status inside
Code:
  blocks
You will need a SSH terminal to do this because the results will scroll out of view, having that many drives.
 
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
Hi Chris,

The server contains boot drives only, no data drives. My situation is close to what he describes https://annvix.com/blog/replacing-a-freenas-drive. sample output (truncated due to number of drives) from my server.

Code:
[root@freenas01] /dev# zpool status -v | more
  pool: cRaid
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
		Sufficient replicas exist for the pool to continue functioning in a
		degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
		repaired.
  scan: scrub repaired 0 in 228h10m with 0 errors on Tue Nov 28 12:10:39 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		cRaid										   DEGRADED	 0	 0	 0
		  raidz2-0									  DEGRADED	 0	 0	 0
			gptid/bd5737ba-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c0a800b7-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c1670fa6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c4b61cd1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c845eb15-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/cbcb6d7a-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/cf7578b4-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d0369857-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d38017f2-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d71bdc77-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d7e4fc72-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/db28ef5c-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/dfb4dc47-aac7-11e6-a712-000e1e52b160  FAULTED	 35	37	 0  too many errors
			gptid/e35609cc-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0



[root@freenas01] ~# glabel status | grep dfb4dc47
[root@freenas01] ~#


Thanks,
Jinno
 
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
What is the output of sas2flash -listall? Also is that a single vdev with all those disks? If that's how things are configured you need to double check your backup because you probably will not survive a drive replacement.
 
Last edited:
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
jinno said:
truncated due to number of drives
Click to expand...
I actually wanted to see the whole thing because it matters how the pool is organized.
 
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
Chris Moore said:
Take a look at the script that is referenced in this thread:
https://forums.freenas.org/index.ph...-devices-disktype-serial-num-multipath.59319/
I think that the output of that script might be very helpful to you.

Unfortunately, to get the script, you will need to go to another forum:
https://forums.servethehome.com/ind...nvenient-way-to-report-installed-disks.16310/
Click to expand...


Hi Chris,

Thanks for the link. Here is the requested output from

Code:
[root@freenas01] /dev# zpool status

  pool: cRaid

 state: DEGRADED

status: One or more devices are faulted in response to persistent errors.

	Sufficient replicas exist for the pool to continue functioning in a

	degraded state.

action: Replace the faulted device, or use 'zpool clear' to mark the device

	repaired.

  scan: scrub repaired 0 in 228h10m with 0 errors on Tue Nov 28 12:10:39 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	cRaid										   DEGRADED	 0	 0	 0

	  raidz2-0									  DEGRADED	 0	 0	 0

		gptid/bd5737ba-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c0a800b7-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c1670fa6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c4b61cd1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c845eb15-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cbcb6d7a-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cf7578b4-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d0369857-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d38017f2-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d71bdc77-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d7e4fc72-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/db28ef5c-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dfb4dc47-aac7-11e6-a712-000e1e52b160  FAULTED	 35	37	 0  too many errors

		gptid/e35609cc-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/e6dd9716-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ea841121-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ee1a9f23-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/eee0ab52-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ef5d5fc9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f2a865c1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f633f828-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f9dbb8b9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fa9deac6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fb1c3918-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fe6e200b-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02140771-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02f4b928-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/06430cb3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/09d5c62a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0d51e59b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11107239-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11d14280-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/12552ab2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/15a7fbd7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1937835e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1ccd0f7f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1d93336a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/20e6cb65-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/21aa80f9-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/24f3cd46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/25c5f189-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2917d4ed-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/29f1726d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2a77f2de-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2dd74b4e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/31712836-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3526ab46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/38b72da4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3c4e0838-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3feb4bf3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/40bbc3d3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/441b6ae2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/44eacab0-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/45797d18-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/48d559c5-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4c769f0f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4d41a9e2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/509e7cc3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5164f85d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/51f5f0d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/555f4dca-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58f2b297-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5c856e4d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5d539516-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5de78069-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6152f8e1-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/64f4dc1a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/688ef3d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c1e4d69-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6fc5b404-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735c9a08-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7431759f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/77925be8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/786217ff-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7bc9b1cd-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7f61e68a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/830eb51f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/86ac47aa-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8a4f61c4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8e05825c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/91892dea-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9263edc6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95c80717-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9960a00c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9d10c918-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a0b1e1ad-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a4593c87-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5330625-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5d4fe5b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a931b655-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ace6d7f8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b07e57b6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b424fa53-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b4ff7486-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: dRaid

 state: ONLINE

  scan: scrub repaired 0 in 116h35m with 0 errors on Thu Nov  9 19:36:00 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	dRaid										   ONLINE	   0	 0	 0

	  raidz2-0									  ONLINE	   0	 0	 0

		gptid/fcb63901-1e24-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0056d583-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/03cbc0a5-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/073cda83-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0a8735df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0de6f39e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1140e9c2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/149ef27c-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/17fb2128-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1b5d677e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1eb3151a-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/220bca19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2561fc11-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/28c6b36d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2c1c7562-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2f7252ca-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/32f30dbe-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/362ec4df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/398bf719-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3cf90530-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/405e7cb4-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/43bc94d2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/471fed3b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4a86b92b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4de5dfa6-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/516bf6e9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/54cfbfbc-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58366017-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5b8bdaf7-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5eef2b5d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/625c66b0-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/65c4bc19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/69498ac1-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c90e7f9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6ff3a493-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735d2b7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/76c0180b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7a169abd-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7d7d4822-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/80f54104-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/845a0f78-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/87c87d99-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8b419e3e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8eb5aa7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/922504b2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95afe024-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9920dad8-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9c9ac102-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: freenas-boot

 state: ONLINE

  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Nov 13 03:45:03 2017

config:


	NAME										  STATE	 READ WRITE CKSUM

	freenas-boot								  ONLINE	   0	 0	 0

	  gptid/ae531cef-92fd-11e5-a800-0cc47a6bf7ca  ONLINE	   0	 0	 0


errors: No known data errors

[root@freenas01] /dev#


Thanks,
Jinno
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
Wow. That's 96 disks in one vdev. That's simply an awful pool configuration, and might explain why your last scrub took the better part of two weeks. But that doesn't address why you can't find the serial number. The script @Chris Moore linked above should help. Also, does the disk show on the volume status page in the web GUI, with its identifier (e.g., da17p2)? With that, the output of smartctl (e.g., smartctl -a /dev/da17) should get you the serial number, though it won't tell you enclosure or slot.
 
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
@danb35 counts faster than me, I lost count at 60 and had to start over, but wow. Then the second pool is a single vdev with 48 more drives.
This is a sub-optimal configuration. It means in that first pool of 96 drives, you only have 2 "extra" disks for parity and to cover you if you have a drive failure. Likewise in the second pool of 48 disks, there are only 2 parity disks. This is not a recommended configuration, and you are over 80% of capacity, which means you will need more storage soon.
Based on what you told us about your enclosures, the first two being 60 drives, I would guess that the failed drive is in the first of those two enclosures and based on the drive count, either they are not full of drives or they are not actually 60 drive capacity? I would love to know more.
Please keep us informed. I am curious.

On the issue of the system being over 80% capacity, and having a very low redundancy configuration, what is it used for? If this is mission critical corporate data, the time has come to do something better.
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
Chris Moore

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
SweetAndLow said:
What is the output of sas2flash --listall? Also is that a single vdev with all those disks? If that's how things are configured you need to double check your backup because you probably will not survive a drive replacement.
Click to expand...

Hi SweetAndLow,

I am still learning at lots about ZFS and do not fully comprehend all the concepts (vdev, multipath). We do have backup, but hoping that is a last ditch effort. Here's the output you requested.

Code:
[root@freenas01] ~# sas2flash

LSI Corporation SAS2 Flash Utility

Version 16.00.00.00 (2013.03.01) 

Copyright (c) 2008-2013 LSI Corporation. All rights reserved


Thanks,
Jinno
 
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
jinno said:
Hi Chris,

Thanks for the link. Here is the requested output from

Code:
[root@freenas01] /dev# zpool status

  pool: cRaid

 state: DEGRADED

status: One or more devices are faulted in response to persistent errors.

	Sufficient replicas exist for the pool to continue functioning in a

	degraded state.

action: Replace the faulted device, or use 'zpool clear' to mark the device

	repaired.

  scan: scrub repaired 0 in 228h10m with 0 errors on Tue Nov 28 12:10:39 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	cRaid										   DEGRADED	 0	 0	 0

	  raidz2-0									  DEGRADED	 0	 0	 0

		gptid/bd5737ba-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c0a800b7-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c1670fa6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c4b61cd1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c845eb15-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cbcb6d7a-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cf7578b4-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d0369857-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d38017f2-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d71bdc77-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d7e4fc72-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/db28ef5c-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dfb4dc47-aac7-11e6-a712-000e1e52b160  FAULTED	 35	37	 0  too many errors

		gptid/e35609cc-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/e6dd9716-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ea841121-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ee1a9f23-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/eee0ab52-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ef5d5fc9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f2a865c1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f633f828-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f9dbb8b9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fa9deac6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fb1c3918-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fe6e200b-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02140771-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02f4b928-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/06430cb3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/09d5c62a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0d51e59b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11107239-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11d14280-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/12552ab2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/15a7fbd7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1937835e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1ccd0f7f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1d93336a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/20e6cb65-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/21aa80f9-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/24f3cd46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/25c5f189-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2917d4ed-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/29f1726d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2a77f2de-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2dd74b4e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/31712836-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3526ab46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/38b72da4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3c4e0838-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3feb4bf3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/40bbc3d3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/441b6ae2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/44eacab0-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/45797d18-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/48d559c5-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4c769f0f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4d41a9e2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/509e7cc3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5164f85d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/51f5f0d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/555f4dca-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58f2b297-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5c856e4d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5d539516-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5de78069-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6152f8e1-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/64f4dc1a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/688ef3d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c1e4d69-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6fc5b404-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735c9a08-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7431759f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/77925be8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/786217ff-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7bc9b1cd-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7f61e68a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/830eb51f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/86ac47aa-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8a4f61c4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8e05825c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/91892dea-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9263edc6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95c80717-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9960a00c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9d10c918-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a0b1e1ad-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a4593c87-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5330625-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5d4fe5b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a931b655-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ace6d7f8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b07e57b6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b424fa53-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b4ff7486-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: dRaid

 state: ONLINE

  scan: scrub repaired 0 in 116h35m with 0 errors on Thu Nov  9 19:36:00 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	dRaid										   ONLINE	   0	 0	 0

	  raidz2-0									  ONLINE	   0	 0	 0

		gptid/fcb63901-1e24-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0056d583-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/03cbc0a5-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/073cda83-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0a8735df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0de6f39e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1140e9c2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/149ef27c-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/17fb2128-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1b5d677e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1eb3151a-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/220bca19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2561fc11-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/28c6b36d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2c1c7562-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2f7252ca-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/32f30dbe-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/362ec4df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/398bf719-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3cf90530-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/405e7cb4-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/43bc94d2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/471fed3b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4a86b92b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4de5dfa6-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/516bf6e9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/54cfbfbc-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58366017-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5b8bdaf7-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5eef2b5d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/625c66b0-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/65c4bc19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/69498ac1-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c90e7f9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6ff3a493-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735d2b7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/76c0180b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7a169abd-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7d7d4822-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/80f54104-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/845a0f78-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/87c87d99-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8b419e3e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8eb5aa7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/922504b2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95afe024-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9920dad8-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9c9ac102-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: freenas-boot

 state: ONLINE

  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Nov 13 03:45:03 2017

config:


	NAME										  STATE	 READ WRITE CKSUM

	freenas-boot								  ONLINE	   0	 0	 0

	  gptid/ae531cef-92fd-11e5-a800-0cc47a6bf7ca  ONLINE	   0	 0	 0


errors: No known data errors

[root@freenas01] /dev#


Thanks,
Jinno
Click to expand...
Chris Moore said:
Take a look at the script that is referenced in this thread:
https://forums.freenas.org/index.ph...-devices-disktype-serial-num-multipath.59319/
I think that the output of that script might be very helpful to you.

Unfortunately, to get the script, you will need to go to another forum:
https://forums.servethehome.com/ind...nvenient-way-to-report-installed-disks.16310/
Click to expand...


Hi Chris,

Thanks for the info. I ran the script but that particular drive is missing from the output.

Code:
multipath/disk13p2  gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  cRaid  da111,da12	 512  HGST HUH728080AL5200  8001		V000000	   SAS2008(0):0#0   multipath/disk13	  

multipath/disk14p2  gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  cRaid  da13,da112	 512  HGST HUH728080AL5200  8001		V000001	   SAS2008(0):0#0   multipath/disk14	  

multipath/disk16p2  gptid/e35609cc-aac7-11e6-a712-000e1e52b160  cRaid  da114,da15	 512  HGST HUH728080AL5200  8001		V000002	   SAS2008(0):0#0   multipath/disk16	  

multipath/disk17p2  gptid/e6dd9716-aac7-11e6-a712-000e1e52b160  cRaid  da105,da16	 512  HGST HUH728080AL5200  8001		V000003	   SAS2008(0):0#0   multipath/disk17	  

multipath/disk18p2  gptid/ea841121-aac7-11e6-a712-000e1e52b160  cRaid  da104,da17	 512  HGST HUH728080AL5200  8001		V000004	   SAS2008(0):0#0   multipath/disk18


I changed the serial numbers for obvious reasons, but as you can see disk15 is missing form the output.


Thanks again,
Jinno
 
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
Chris Moore said:
@danb35 counts faster than me, I lost count at 60 and had to start over, but wow. Then the second pool is a single vdev with 48 more drives.
This is a sub-optimal configuration. It means in that first pool of 96 drives, you only have 2 "extra" disks for parity and to cover you if you have a drive failure. Likewise in the second pool of 48 disks, there are only 2 parity disks. This is not a recommended configuration, and you are over 80% of capacity, which means you will need more storage soon.
Based on what you told us about your enclosures, the first two being 60 drives, I would guess that the failed drive is in the first of those two enclosures and based on the drive count, either they are not full of drives or they are not actually 60 drive capacity? I would love to know more.
Please keep us informed. I am curious.

On the issue of the system being over 80% capacity, and having a very low redundancy configuration, what is it used for? If this is mission critical corporate data, the time has come to do something better.
Click to expand...


Hi Dan,

Thanks for your response and interest. Yes, that was my first observation concerning only two parity drives. Also, we have don't even extra drives on premise in case of emergency, I was always under the impression to have at least 2 extra non spinning drives in house. You are correct, the first 2 two are 60 drive capacity but not fully populated. We are in the process of getting moving data to tape to go below 80%.


BEst,
Jinno
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
Try sas2flash -listall.

And from the output above, it almost looks like the disk has dropped offline, but it also looks like it is (or was) da14. smartctl -a /dev/da14 may give you the serial number, as well as some more detail about what's wrong with it.
 
SweetAndLow

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
jinno said:
Hi SweetAndLow,

I am still learning at lots about ZFS and do not fully comprehend all the concepts (vdev, multipath). We do have backup, but hoping that is a last ditch effort. Here's the output you requested.

Code:
[root@freenas01] ~# sas2flash

LSI Corporation SAS2 Flash Utility

Version 16.00.00.00 (2013.03.01)

Copyright (c) 2008-2013 LSI Corporation. All rights reserved


Thanks,
Jinno
Click to expand...
I was looking for the output of "sas2flash -listall" this will tell us what controllers you are using and what firmware they have on them. smartctl -a /dev/da14 should give you the info you need to find the serial number.
 
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
danb35 said:
Wow. That's 96 disks in one vdev. That's simply an awful pool configuration, and might explain why your last scrub took the better part of two weeks. But that doesn't address why you can't find the serial number. The script @Chris Moore linked above should help. Also, does the disk show on the volume status page in the web GUI, with its identifier (e.g., da17p2)? With that, the output of smartctl (e.g., smartctl -a /dev/da17) should get you the serial number, though it won't tell you enclosure or slot.
Click to expand...


Hi Dan,

I did try the script but couldn't find the serial number of the drive in question but, I think I can narrow it down with that information.


Thanks,
Jinno
 
J

jinno

Dabbler
Joined
Nov 28, 2017
Messages
13
jinno said:
Hi Dan,

I did try the script but couldn't find the serial number of the drive in question but, I think I can narrow it down with that information.


Thanks,
Jinno
Click to expand...


Hi Dan,

Correct...I did try that earlier but, couldn't get the info, here's the output.

Code:
[root@freenas01] ~# smartctl -a /dev/da14

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)

Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


/dev/da14: Unable to detect device type

Please specify device type with the -d option.


Use smartctl -h to get a usage summary


[root@freenas01] ~#


Jinno
 
Status
Not open for further replies.

Similar threads

L
  • Locked
How to find a failed drive if you have an LSI controller
Replies
10
Views
20K
Scharbag
Scharbag
Z
ACTUALLY finding the failed drive
Replies
1
Views
3K
Chris Moore
Chris Moore
W
  • Locked
Reliable way of locating a dead drive for replacement?
Replies
5
Views
2K
Ericloewe
Ericloewe
H
Help... SAS HBA are driving me mad...
Replies
3
Views
2K
Hanry
H
M
  • Locked
find the correct drive
Replies
3
Views
1K
Stux
Stux
Top