6 of 15 Drives Detected

[netsvcs]

I hope this is the correct spot to put this. I'm having issues with FreeNAS 11.1-U4 detecting all of my drives as it will only detect 6 of 15.

I have a Dell R710 with an MD1000 and all 15 slots are filled with 2 TB drives. The MD1000 is connected with an External Mini-SAS HD SFF-8644 to Mini-SAS SFF-8470 Cable via an LSI SAS8300-8e. All 15 drives are detected from the BIOS and are available for boot selection or when using other operating systems such as FreeBSD 11.x or CentOS 7.

FreeNAS, either during the installation or when it's installed, will only detect 6 of the drives in the MD1000 shelf.

I have updated the firmware and BIOS for the LSI card (Firmware 15.00.02.00 and BIOS 08.35.04.00). I have also ensured the R710 firmware for various components is entirely up to date using the various Dell utilities to perform the update.

Any thoughts on what I can do to resolve this issue? I've included some additional information below.

Code:

root@freenas:~ # camcontrol devlist
<FUJITSU MBA3147RC D306>		   at scbus0 target 0 lun 0 (pass0)
<FUJITSU MBA3147RC D306>		   at scbus0 target 1 lun 0 (pass1)
<SEAGATE ST3300657SS ES65>		 at scbus0 target 3 lun 0 (pass2)
<SEAGATE ST3300657SS ES65>		 at scbus0 target 4 lun 0 (pass3)
<SEAGATE ST3300657SS ES65>		 at scbus0 target 5 lun 0 (pass4)
<DP BACKPLANE 1.07>				at scbus0 target 32 lun 0 (ses0,pass5)
<ATA WDC WD2002FYPS-0 1G01>		at scbus1 target 2 lun 0 (pass6,da0)
<ATA WDC WD2002FYPS-0 5G05>		at scbus1 target 3 lun 0 (pass7,da1)
<ATA WDC WD2002FYPS-0 1G01>		at scbus1 target 4 lun 0 (pass8,da2)
<ATA WDC WD2002FYPS-0 1G01>		at scbus1 target 5 lun 0 (pass9,da3)
<ATA WDC WD2000F9YZ-0 1A01>		at scbus1 target 6 lun 0 (pass10,da4)
<ATA WDC WD2002FYPS-0 1G01>		at scbus1 target 7 lun 0 (pass11,da5)
<DELL MD1000 A.04>				 at scbus1 target 8 lun 0 (ses1,pass12)
<TEAC DVD-ROM DV28SV D.0L>		 at scbus2 target 0 lun 0 (cd0,pass13)

Code:

root@freenas:~ # dmesg | grep mpr
mpr0: <Avago Technologies (LSI) SAS3008> port 0xdc00-0xdcff mem 0xdf5b0000-0xdf5bffff,0xdf5c0000-0xdf5fffff irq 40 at device 0.0 on pci7
mpr0: Firmware: 15.00.02.00, Driver: 18.03.00.00-fbsd
mpr0: IOCCapabilities: 7a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc,FastPath,RDPQArray>
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a90c
mpr0: Found device <6201<StpTarg,AtapiDev,SepDev>,End Device> <3.0Gbps> handle<0x000c> enclosureHandle<0x0002> slot 0
mpr0: At enclosure level 0 and connector name (	)
mpr0: SAS Address for SATA device = 9086371fa0aa8ea3
mpr0: SAS Address for SATA device = 91823520a8a690a9
mpr0: SAS Address for SATA device = 90863626a4a88ea2
mpr0: SAS Address for SATA device = 9086371fa1a993a8
mpr0: SAS Address for SATA device = ac973622aa9f9753
mpr0: SAS Address for SATA device = 90863626a3a496a7
mpr0: SAS Address for SATA device = 90863626a4a58eab
mpr0: SAS Address for SATA device = 9086371fa0a197aa
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a942
mpr0: SAS Address from SATA device = 9086371fa0aa8ea3
mpr0: Found device <81<SataDev>,End Device> <3.0Gbps> handle<0x000d> enclosureHandle<0x0000> slot 0
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a943
mpr0: SAS Address from SATA device = 91823520a8a690a9
mpr0: Found device <81<SataDev>,End Device> <3.0Gbps> handle<0x000e> enclosureHandle<0x0000> slot 0
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a944
mpr0: SAS Address from SATA device = 90863626a4a88ea2
mpr0: Found device <81<SataDev>,End Device> <3.0Gbps> handle<0x000f> enclosureHandle<0x0000> slot 0
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a945
mpr0: SAS Address from SATA device = 9086371fa1a993a8
mpr0: Found device <81<SataDev>,End Device> <3.0Gbps> handle<0x0010> enclosureHandle<0x0000> slot 0
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a946
mpr0: SAS Address from SATA device = ac973622aa9f9753
mpr0: Found device <81<SataDev>,End Device> <1.5Gbps> handle<0x0011> enclosureHandle<0x0000> slot 0
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a947
mpr0: SAS Address from SATA device = 90863626a3a496a7
mpr0: Found device <81<SataDev>,End Device> <3.0Gbps> handle<0x0012> enclosureHandle<0x0000> slot 0
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a948
mpr0: SAS Address from SATA device = 90863626a4a58eab
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a949
mpr0: SAS Address from SATA device = 9086371fa0a197aa
mpr0: SAS Address for SATA device = 9086371fa2a48ea5
mpr0: SAS Address for SATA device = 9086371fa1a897a3
mpr0: SAS Address for SATA device = 90863626a4a497a8
mpr0: SAS Address for SATA device = 90863626a3a696a8
mpr0: SAS Address for SATA device = 9086371fa0a197a9
mpr0: SAS Address for SATA device = 9086371fa1aa97ab
mpr0: SAS Address for SATA device = 9086371fa4aa8fa2
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a982
mpr0: SAS Address from SATA device = 9086371fa2a48ea5
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a983
mpr0: SAS Address from SATA device = 9086371fa1a897a3
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a984
mpr0: SAS Address from SATA device = 90863626a4a497a8
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a985
mpr0: SAS Address from SATA device = 90863626a3a696a8
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a986
mpr0: SAS Address from SATA device = 9086371fa0a197a9
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a987
mpr0: SAS Address from SATA device = 9086371fa1aa97ab
mpr0: SAS Address from SAS device page0 = 5001e4f241c1a988
mpr0: SAS Address from SATA device = 9086371fa4aa8fa2
da4 at mpr0 bus 0 scbus1 target 6 lun 0
da1 at mpr0 bus 0 scbus1 target 3 lun 0
da2 at mpr0 bus 0 scbus1 target 4 lun 0
da5 at mpr0 bus 0 scbus1 target 7 lun 0
da0 at mpr0 bus 0 scbus1 target 2 lun 0
da3 at mpr0 bus 0 scbus1 target 5 lun 0
ses1 at mpr0 bus 0 scbus1 target 8 lun 0

Code:

root@freenas:~ # sas3ircu 0 DISPLAY
Avago Technologies SAS3 IR Configuration Utility.
Version 16.00.00.00 (2017.04.26)
Copyright (c) 2009-2017 Avago Technologies. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type						 : SAS3008
  BIOS version							: 8.35.04.00
  Firmware version						: 15.00.02.00
  Channel description					 : 1 Serial Attached SCSI
  Initiator ID							: 0
  Maximum physical devices				: 1023
  Concurrent commands supported		   : 9664
  Slot									: 3
  Segment								 : 0
  Bus									 : 7
  Device								  : 0
  Function								: 0
  RAID Support							: No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #							 : 0
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a944
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 1907729/3907029167
  Manufacturer							: ATA
  Model Number							: WDC WD2002FYPS-0
  Firmware Revision					   : 1G01
  Serial No							   : WDWCAVY7284700
  Unit Serial No(VPD)					 : WD-WCAVY7284700
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 0
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a942
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 1907729/3907029167
  Manufacturer							: ATA
  Model Number							: WDC WD2002FYPS-0
  Firmware Revision					   : 1G01
  Serial No							   : WDWCAVY7310901
  Unit Serial No(VPD)					 : WD-WCAVY7310901
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 0
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a943
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 1907729/3907029167
  Manufacturer							: ATA
  Model Number							: WDC WD2002FYPS-0
  Firmware Revision					   : 5G05
  Serial No							   : WDWCAVY3129747
  Unit Serial No(VPD)					 : WD-WCAVY3129747
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 0
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a947
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 1907729/3907029167
  Manufacturer							: ATA
  Model Number							: WDC WD2002FYPS-0
  Firmware Revision					   : 1G01
  Serial No							   : WDWCAVY7283385
  Unit Serial No(VPD)					 : WD-WCAVY7283385
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 0
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a945
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 1907729/3907029167
  Manufacturer							: ATA
  Model Number							: WDC WD2002FYPS-0
  Firmware Revision					   : 1G01
  Serial No							   : WDWCAVY7311856
  Unit Serial No(VPD)					 : WD-WCAVY7311856
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 0
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a946
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 1907729/3907029167
  Manufacturer							: ATA
  Model Number							: WDC WD2000F9YZ-0
  Firmware Revision					   : 1A01
  Serial No							   : WDWCC1P1243988
  Unit Serial No(VPD)					 : WD-WCC1P1243988
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Enclosure services device
  Enclosure #							 : 2
  Slot #								  : 0
  SAS Address							 : 5001e4f-2-41c1-a90c
  State								   : Standby (SBY)
  Manufacturer							: DELL
  Model Number							: MD1000
  Firmware Revision					   : A.04
  Serial No							   : 6W0Q9G1
  Unit Serial No(VPD)					 : N/A
  GUID									: N/A
  Protocol								: SATA
  Device Type							 : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#							  : 1
  Logical ID							  : 500605b0:0ab5fa60
  Numslots								: 8
  StartSlot							   : 0
  Enclosure#							  : 2
  Logical ID							  : 5001e40b:c700004e
  Numslots								: 1
  StartSlot							   : 0
------------------------------------------------------------------------
SAS3IRCU: Command DISPLAY Completed Successfully.
SAS3IRCU: Utility Completed Successfully.

 

[Ericloewe]

Is there any pattern to the disks that do show up?

 

[netsvcs]

They appear to be the same disks every time.

 

[Ericloewe]

Are they next to each other? Alternating? First n disks?

 

[Stux]

FWIW, I think there was a similar post recently which had a happy ending...

Might be able to find it with some searching...

 

[netsvcs]

They are next to eachother, it's the last 6 disks (da0 in slot 14, da1 in slot 13, etc). Disk da5 was showing has having issues so I've pulled it and restarted the server. Still only 6 disks shown, da5 is now in slot 08. Another odd thing is that if I swap one of the drives, it's not automatically showing up. It's not really ideal if I have to restart the server to get new or replaced drives to be detected.

Closest related issue to this I've found searching so far is an issue due to bad drives.

 

[Ericloewe]

Try a different OS, like Windows or Linux. Let's see if we can narrow this down a bit.

 

[netsvcs]

I've already tried both CentOS and FreeBSD 11. Both show and detect all 15 drives and are usable. It seems to be only FreeNAS that has issues.

 

[Ericloewe]

FreeBSD 11.0?

 

[netsvcs]

FreeBSD 11.1.

 

[Ericloewe]

That makes it even more mysterious. Try filing a bug report so that one of the devs can take a look.

 

[westinc]

Wow, I just started to use Google to help solve my problem. Netsvcs is having the exact same problem as I am. Fresh install of FreeNAS 11.1 on a Dell 610. Which is connected to a Dell MD1000. Each Hard Drive is 2TB. FreeNAS is only detecting 7 drives out of the 15. Tracing the serial numbers I found that the 7 drives are spread randomly throughout the 15 slots on the MD1000. I can grab some pictures of showing the output of the above commands and update this thread. I didn't want to make a new thread for a same/similar problem.

 

[netsvcs]

I think the issue is related to the drives or maybe interposers on the drives. I removed the original shelf and connected a second MD1000 shelf that was filled with 250 GB drives and all 15 drives were detected from there.

At this point, due to the headaches, I have to abandon FreeNAS and will move to using CentOS. Unfortunately I don't have a whole lot of time left to devote to this issue and this was a hail mary attempt to see if someone else had thoughts on a fix. Thank you for the assistance.

 

[sretalla]

Have you looked at gmultipath list ? the geom_multipath kernel module may have snatched the disks if it thinks they are attached via a bus that has redundant paths (not always correctly)

if the above command shows a disk or more than one disk, you could destroy those disks and you would have your individual drives back (obviously ensure that you aren't using any of the multipath disks first if you do that).

 

[westinc]

Sretalla Here are some images. Hopefully this can help out a bit.

During the R610's boot the server see's all disk (one disk needs replacing, so the 14 disk is correct).
https://imgur.com/xzHRyGE
Camcontrol devlist shows 6 disks on the MD1000. I traced the disks serial numbers to the physical disks and found that they are spread randomly throughout the MD1000 chassis.
https://imgur.com/wimMUQ8
As seen in the image above gmultipath list does not show anything.

 

[sretalla]

So it seems gmultipath is not interfering here.

I guess you have ruled out a cabling issue to the backplane if the HBA POST can see all of the disks.

 

[westinc]

Correct. Additionally I just installed Nexentastor 4 and it detects all disks.