Unable to identify drive with sas2ircu

[jinno]

Hi, I inherited a Freenas Server 9.3 running on Supermicro server with a LSI SAS2008 card and 3 JBOD enclosures attached, 2 of the enclosures are 60 drive capacity and the other one is 48 drive capacity, with 8TB drives installed.

We've recently experienced a drive failure. I am unable to find the serial number in order to replace the drive from the GUI as well as the CLI (screenshots enclosed).

I am unable to retrieve the serial number through "smartctl" and "camcontrol". I tried using the LSI Logic's configuration tool "sas2ircu" with negative results.

I am at least trying to narrow down the location of the drive to the enclosure, the information from sas2ircu display is not accurate as there missing serial drive numbers, wrong enclosure information and and wrong slot numbers.

I am also getting this error when issuing the "sas2ircu 0 status" command.

[root@freenas01] /dev# sas2ircu 0 status
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

SAS2IRCU: The STATUS command is not supported by the firmware currently loaded on controller 0.
[root@freenas01] /dev#


Help,
jinno

 

[Chris Moore]

Does the server that FreeNAS is running on have drives in it also?
We need to get a little more information on the topology of the SAS controller.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Chris Moore]

The more information you can give us, the better possibility someone will be able to help you.
Can you SSH into a console or are you using the web interface?
If you can, please give us the results of zpool status inside

Code:

  blocks

You will need a SSH terminal to do this because the results will scroll out of view, having that many drives.

 

[jinno]

Hi Chris,

The server contains boot drives only, no data drives. My situation is close to what he describes https://annvix.com/blog/replacing-a-freenas-drive. sample output (truncated due to number of drives) from my server.

Code:

[root@freenas01] /dev# zpool status -v | more
  pool: cRaid
 state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
		Sufficient replicas exist for the pool to continue functioning in a
		degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
		repaired.
  scan: scrub repaired 0 in 228h10m with 0 errors on Tue Nov 28 12:10:39 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		cRaid										   DEGRADED	 0	 0	 0
		  raidz2-0									  DEGRADED	 0	 0	 0
			gptid/bd5737ba-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c0a800b7-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c1670fa6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c4b61cd1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/c845eb15-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/cbcb6d7a-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/cf7578b4-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d0369857-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d38017f2-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d71bdc77-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/d7e4fc72-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/db28ef5c-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0
			gptid/dfb4dc47-aac7-11e6-a712-000e1e52b160  FAULTED	 35	37	 0  too many errors
			gptid/e35609cc-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0



[root@freenas01] ~# glabel status | grep dfb4dc47
[root@freenas01] ~# 

Thanks,
Jinno

 

[Chris Moore]

Take a look at the script that is referenced in this thread:
https://forums.freenas.org/index.ph...-devices-disktype-serial-num-multipath.59319/
I think that the output of that script might be very helpful to you.

Unfortunately, to get the script, you will need to go to another forum:
https://forums.servethehome.com/ind...nvenient-way-to-report-installed-disks.16310/

 

[SweetAndLow]

What is the output of sas2flash -listall? Also is that a single vdev with all those disks? If that's how things are configured you need to double check your backup because you probably will not survive a drive replacement.

 

[Chris Moore]

truncated due to number of drives

I actually wanted to see the whole thing because it matters how the pool is organized.

 

[jinno]

Take a look at the script that is referenced in this thread:
https://forums.freenas.org/index.ph...-devices-disktype-serial-num-multipath.59319/
I think that the output of that script might be very helpful to you.

Unfortunately, to get the script, you will need to go to another forum:
https://forums.servethehome.com/ind...nvenient-way-to-report-installed-disks.16310/

Hi Chris,

Thanks for the link. Here is the requested output from

Code:

[root@freenas01] /dev# zpool status

  pool: cRaid

 state: DEGRADED

status: One or more devices are faulted in response to persistent errors.

	Sufficient replicas exist for the pool to continue functioning in a

	degraded state.

action: Replace the faulted device, or use 'zpool clear' to mark the device

	repaired.

  scan: scrub repaired 0 in 228h10m with 0 errors on Tue Nov 28 12:10:39 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	cRaid										   DEGRADED	 0	 0	 0

	  raidz2-0									  DEGRADED	 0	 0	 0

		gptid/bd5737ba-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c0a800b7-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c1670fa6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c4b61cd1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c845eb15-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cbcb6d7a-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cf7578b4-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d0369857-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d38017f2-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d71bdc77-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d7e4fc72-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/db28ef5c-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dfb4dc47-aac7-11e6-a712-000e1e52b160  FAULTED	 35	37	 0  too many errors

		gptid/e35609cc-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/e6dd9716-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ea841121-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ee1a9f23-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/eee0ab52-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ef5d5fc9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f2a865c1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f633f828-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f9dbb8b9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fa9deac6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fb1c3918-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fe6e200b-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02140771-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02f4b928-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/06430cb3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/09d5c62a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0d51e59b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11107239-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11d14280-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/12552ab2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/15a7fbd7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1937835e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1ccd0f7f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1d93336a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/20e6cb65-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/21aa80f9-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/24f3cd46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/25c5f189-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2917d4ed-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/29f1726d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2a77f2de-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2dd74b4e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/31712836-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3526ab46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/38b72da4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3c4e0838-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3feb4bf3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/40bbc3d3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/441b6ae2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/44eacab0-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/45797d18-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/48d559c5-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4c769f0f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4d41a9e2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/509e7cc3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5164f85d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/51f5f0d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/555f4dca-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58f2b297-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5c856e4d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5d539516-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5de78069-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6152f8e1-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/64f4dc1a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/688ef3d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c1e4d69-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6fc5b404-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735c9a08-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7431759f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/77925be8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/786217ff-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7bc9b1cd-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7f61e68a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/830eb51f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/86ac47aa-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8a4f61c4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8e05825c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/91892dea-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9263edc6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95c80717-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9960a00c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9d10c918-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a0b1e1ad-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a4593c87-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5330625-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5d4fe5b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a931b655-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ace6d7f8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b07e57b6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b424fa53-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b4ff7486-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: dRaid

 state: ONLINE

  scan: scrub repaired 0 in 116h35m with 0 errors on Thu Nov  9 19:36:00 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	dRaid										   ONLINE	   0	 0	 0

	  raidz2-0									  ONLINE	   0	 0	 0

		gptid/fcb63901-1e24-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0056d583-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/03cbc0a5-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/073cda83-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0a8735df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0de6f39e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1140e9c2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/149ef27c-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/17fb2128-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1b5d677e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1eb3151a-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/220bca19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2561fc11-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/28c6b36d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2c1c7562-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2f7252ca-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/32f30dbe-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/362ec4df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/398bf719-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3cf90530-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/405e7cb4-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/43bc94d2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/471fed3b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4a86b92b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4de5dfa6-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/516bf6e9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/54cfbfbc-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58366017-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5b8bdaf7-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5eef2b5d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/625c66b0-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/65c4bc19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/69498ac1-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c90e7f9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6ff3a493-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735d2b7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/76c0180b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7a169abd-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7d7d4822-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/80f54104-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/845a0f78-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/87c87d99-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8b419e3e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8eb5aa7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/922504b2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95afe024-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9920dad8-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9c9ac102-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: freenas-boot

 state: ONLINE

  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Nov 13 03:45:03 2017

config:


	NAME										  STATE	 READ WRITE CKSUM

	freenas-boot								  ONLINE	   0	 0	 0

	  gptid/ae531cef-92fd-11e5-a800-0cc47a6bf7ca  ONLINE	   0	 0	 0


errors: No known data errors

[root@freenas01] /dev# 

Thanks,
Jinno

 

[danb35]

Wow. That's 96 disks in one vdev. That's simply an awful pool configuration, and might explain why your last scrub took the better part of two weeks. But that doesn't address why you can't find the serial number. The script @Chris Moore linked above should help. Also, does the disk show on the volume status page in the web GUI, with its identifier (e.g., da17p2)? With that, the output of smartctl (e.g., smartctl -a /dev/da17) should get you the serial number, though it won't tell you enclosure or slot.

 

[Chris Moore]

@danb35 counts faster than me, I lost count at 60 and had to start over, but wow. Then the second pool is a single vdev with 48 more drives.
This is a sub-optimal configuration. It means in that first pool of 96 drives, you only have 2 "extra" disks for parity and to cover you if you have a drive failure. Likewise in the second pool of 48 disks, there are only 2 parity disks. This is not a recommended configuration, and you are over 80% of capacity, which means you will need more storage soon.
Based on what you told us about your enclosures, the first two being 60 drives, I would guess that the failed drive is in the first of those two enclosures and based on the drive count, either they are not full of drives or they are not actually 60 drive capacity? I would love to know more.
Please keep us informed. I am curious.

On the issue of the system being over 80% capacity, and having a very low redundancy configuration, what is it used for? If this is mission critical corporate data, the time has come to do something better.

 

[danb35]

@danb35 counts faster than me,

wc -l is pretty fast...

 

[Chris Moore]

wc -l is pretty fast...

Where I am right now, the system I can access the internet with is Windows.

 

[danb35]

I didn't actually use wc either; I pasted the output into Word and used its word/line count function.

 

[jinno]

What is the output of sas2flash --listall? Also is that a single vdev with all those disks? If that's how things are configured you need to double check your backup because you probably will not survive a drive replacement.

Hi SweetAndLow,

I am still learning at lots about ZFS and do not fully comprehend all the concepts (vdev, multipath). We do have backup, but hoping that is a last ditch effort. Here's the output you requested.

Code:

[root@freenas01] ~# sas2flash

LSI Corporation SAS2 Flash Utility

Version 16.00.00.00 (2013.03.01) 

Copyright (c) 2008-2013 LSI Corporation. All rights reserved 

Thanks,
Jinno

 

[jinno]

Hi Chris,

Thanks for the link. Here is the requested output from

Code:

[root@freenas01] /dev# zpool status

  pool: cRaid

 state: DEGRADED

status: One or more devices are faulted in response to persistent errors.

	Sufficient replicas exist for the pool to continue functioning in a

	degraded state.

action: Replace the faulted device, or use 'zpool clear' to mark the device

	repaired.

  scan: scrub repaired 0 in 228h10m with 0 errors on Tue Nov 28 12:10:39 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	cRaid										   DEGRADED	 0	 0	 0

	  raidz2-0									  DEGRADED	 0	 0	 0

		gptid/bd5737ba-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c0a800b7-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c1670fa6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c4b61cd1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/c845eb15-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cbcb6d7a-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/cf7578b4-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d0369857-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d38017f2-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d71bdc77-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/d7e4fc72-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/db28ef5c-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/dfb4dc47-aac7-11e6-a712-000e1e52b160  FAULTED	 35	37	 0  too many errors

		gptid/e35609cc-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/e6dd9716-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ea841121-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ee1a9f23-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/eee0ab52-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ef5d5fc9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f2a865c1-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f633f828-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/f9dbb8b9-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fa9deac6-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fb1c3918-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/fe6e200b-aac7-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02140771-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/02f4b928-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/06430cb3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/09d5c62a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0d51e59b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11107239-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/11d14280-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/12552ab2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/15a7fbd7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1937835e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1ccd0f7f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1d93336a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/20e6cb65-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/21aa80f9-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/24f3cd46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/25c5f189-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2917d4ed-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/29f1726d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2a77f2de-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2dd74b4e-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/31712836-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3526ab46-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/38b72da4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3c4e0838-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3feb4bf3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/40bbc3d3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/441b6ae2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/44eacab0-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/45797d18-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/48d559c5-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4c769f0f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4d41a9e2-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/509e7cc3-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5164f85d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/51f5f0d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/555f4dca-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58f2b297-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5c856e4d-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5d539516-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5de78069-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6152f8e1-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/64f4dc1a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/688ef3d7-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c1e4d69-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6fc5b404-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735c9a08-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7431759f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/77925be8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/786217ff-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7bc9b1cd-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7f61e68a-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/830eb51f-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/86ac47aa-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8a4f61c4-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8e05825c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/91892dea-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9263edc6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95c80717-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9960a00c-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9d10c918-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a0b1e1ad-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a4593c87-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5330625-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a5d4fe5b-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/a931b655-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/ace6d7f8-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b07e57b6-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b424fa53-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0

		gptid/b4ff7486-aac8-11e6-a712-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: dRaid

 state: ONLINE

  scan: scrub repaired 0 in 116h35m with 0 errors on Thu Nov  9 19:36:00 2017

config:


	NAME											STATE	 READ WRITE CKSUM

	dRaid										   ONLINE	   0	 0	 0

	  raidz2-0									  ONLINE	   0	 0	 0

		gptid/fcb63901-1e24-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0056d583-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/03cbc0a5-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/073cda83-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0a8735df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/0de6f39e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1140e9c2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/149ef27c-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/17fb2128-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1b5d677e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/1eb3151a-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/220bca19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2561fc11-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/28c6b36d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2c1c7562-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/2f7252ca-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/32f30dbe-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/362ec4df-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/398bf719-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/3cf90530-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/405e7cb4-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/43bc94d2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/471fed3b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4a86b92b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/4de5dfa6-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/516bf6e9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/54cfbfbc-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/58366017-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5b8bdaf7-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/5eef2b5d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/625c66b0-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/65c4bc19-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/69498ac1-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6c90e7f9-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/6ff3a493-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/735d2b7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/76c0180b-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7a169abd-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/7d7d4822-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/80f54104-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/845a0f78-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/87c87d99-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8b419e3e-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/8eb5aa7d-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/922504b2-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/95afe024-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9920dad8-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0

		gptid/9c9ac102-1e25-11e7-85ac-000e1e52b160  ONLINE	   0	 0	 0


errors: No known data errors


  pool: freenas-boot

 state: ONLINE

  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Nov 13 03:45:03 2017

config:


	NAME										  STATE	 READ WRITE CKSUM

	freenas-boot								  ONLINE	   0	 0	 0

	  gptid/ae531cef-92fd-11e5-a800-0cc47a6bf7ca  ONLINE	   0	 0	 0


errors: No known data errors

[root@freenas01] /dev#

Thanks,
Jinno

Take a look at the script that is referenced in this thread:
https://forums.freenas.org/index.ph...-devices-disktype-serial-num-multipath.59319/
I think that the output of that script might be very helpful to you.

Unfortunately, to get the script, you will need to go to another forum:
https://forums.servethehome.com/ind...nvenient-way-to-report-installed-disks.16310/

Hi Chris,

Thanks for the info. I ran the script but that particular drive is missing from the output.

Code:

multipath/disk13p2  gptid/dbeecbfe-aac7-11e6-a712-000e1e52b160  cRaid  da111,da12	 512  HGST HUH728080AL5200  8001		V000000	   SAS2008(0):0#0   multipath/disk13	  

multipath/disk14p2  gptid/dc6ae552-aac7-11e6-a712-000e1e52b160  cRaid  da13,da112	 512  HGST HUH728080AL5200  8001		V000001	   SAS2008(0):0#0   multipath/disk14	  

multipath/disk16p2  gptid/e35609cc-aac7-11e6-a712-000e1e52b160  cRaid  da114,da15	 512  HGST HUH728080AL5200  8001		V000002	   SAS2008(0):0#0   multipath/disk16	  

multipath/disk17p2  gptid/e6dd9716-aac7-11e6-a712-000e1e52b160  cRaid  da105,da16	 512  HGST HUH728080AL5200  8001		V000003	   SAS2008(0):0#0   multipath/disk17	  

multipath/disk18p2  gptid/ea841121-aac7-11e6-a712-000e1e52b160  cRaid  da104,da17	 512  HGST HUH728080AL5200  8001		V000004	   SAS2008(0):0#0   multipath/disk18	  

I changed the serial numbers for obvious reasons, but as you can see disk15 is missing form the output.


Thanks again,
Jinno

 

[jinno]

@danb35 counts faster than me, I lost count at 60 and had to start over, but wow. Then the second pool is a single vdev with 48 more drives.
This is a sub-optimal configuration. It means in that first pool of 96 drives, you only have 2 "extra" disks for parity and to cover you if you have a drive failure. Likewise in the second pool of 48 disks, there are only 2 parity disks. This is not a recommended configuration, and you are over 80% of capacity, which means you will need more storage soon.
Based on what you told us about your enclosures, the first two being 60 drives, I would guess that the failed drive is in the first of those two enclosures and based on the drive count, either they are not full of drives or they are not actually 60 drive capacity? I would love to know more.
Please keep us informed. I am curious.

On the issue of the system being over 80% capacity, and having a very low redundancy configuration, what is it used for? If this is mission critical corporate data, the time has come to do something better.

Hi Dan,

Thanks for your response and interest. Yes, that was my first observation concerning only two parity drives. Also, we have don't even extra drives on premise in case of emergency, I was always under the impression to have at least 2 extra non spinning drives in house. You are correct, the first 2 two are 60 drive capacity but not fully populated. We are in the process of getting moving data to tape to go below 80%.


BEst,
Jinno

 

[danb35]

Try sas2flash -listall.

And from the output above, it almost looks like the disk has dropped offline, but it also looks like it is (or was) da14. smartctl -a /dev/da14 may give you the serial number, as well as some more detail about what's wrong with it.

 

[SweetAndLow]

Hi SweetAndLow,

I am still learning at lots about ZFS and do not fully comprehend all the concepts (vdev, multipath). We do have backup, but hoping that is a last ditch effort. Here's the output you requested.

Code:

[root@freenas01] ~# sas2flash

LSI Corporation SAS2 Flash Utility

Version 16.00.00.00 (2013.03.01)

Copyright (c) 2008-2013 LSI Corporation. All rights reserved

Thanks,
Jinno

I was looking for the output of "sas2flash -listall" this will tell us what controllers you are using and what firmware they have on them. smartctl -a /dev/da14 should give you the info you need to find the serial number.

 

[jinno]

Wow. That's 96 disks in one vdev. That's simply an awful pool configuration, and might explain why your last scrub took the better part of two weeks. But that doesn't address why you can't find the serial number. The script @Chris Moore linked above should help. Also, does the disk show on the volume status page in the web GUI, with its identifier (e.g., da17p2)? With that, the output of smartctl (e.g., smartctl -a /dev/da17) should get you the serial number, though it won't tell you enclosure or slot.

Hi Dan,

I did try the script but couldn't find the serial number of the drive in question but, I think I can narrow it down with that information.


Thanks,
Jinno

 

[jinno]

Hi Dan,

I did try the script but couldn't find the serial number of the drive in question but, I think I can narrow it down with that information.


Thanks,
Jinno

Hi Dan,

Correct...I did try that earlier but, couldn't get the info, here's the output.

Code:

[root@freenas01] ~# smartctl -a /dev/da14

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)

Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org


/dev/da14: Unable to detect device type

Please specify device type with the -d option.


Use smartctl -h to get a usage summary


[root@freenas01] ~# 

Jinno