LSI BIOS order and /dev/daX order not aligned

Status
Not open for further replies.

seanacais

Cadet
Joined
Jan 24, 2017
Messages
6
I've recently installed FreeNAS 9.10 onto a system containing 12 drives (no real data)

MB: A1SAI-2758F/32 GB ECC RAM
LSI (Avago) 9207-8i (P20 Firmware)

Four of the drives are attached to the MB SATA controller and were given /dev names ada0-3
Eight drives are attached to the HBA and were given /dev/da0 thru /dev/da7

All is good!

I noticed that I had accidentally swapped the connectors to the HBA and during boot the configuration from the LSI utility shows disks physically in slots 0-3 are listed as slots 4-7 (and are given names /dev/da4 thru /dev/da7) and disks physically in slots 4-7 show as enclosure slots 0-3 and were given /dev/da0 thru /dev/da3.

No problem! I open the case, and swap the SFF cables and reboot. The LSI listing during boot looks fine but the /dev/hdX names stay the same.

I rearrange the drives in the slots, names stay the same (pretty cool). I pull some of the drives and the designations remap to /dev/da0 thru /dev/daX depending on the number of drives I've left in the system. but whenever I put all the drives back, it goes back to the original designations.

hmm! OK, OS probably has a list of ID's so let's reinstall the FreeNAS. Wipe the boot disk, reinstall, no joy! The drives are given their origin /dev/daX designations.

Now I'm pretty surprised! I guess there's some information written on the partitions. I boot gpartd, wipe the partition table off every drive, reinstall and still the drive designations stays the same.

So my question is what is happening? I get the value of keeping the same name once FreeNAS is installed but I'm really surprised that this is persisting across a partition wipe of all the drives in the system and a reinstall. I'm guessing that the 9207-8i itself is mapping a drive serial number to some manufactured SAS address and that's being passed to the driver. I took a quick look at sas2flash and did a -reset since that seems to be closest to a factory reset. I don't know if that guess is correct so I wasn't about to start erasing memory segments on the HBA to try to fix this.

Question 2: How do I force a rescan and reassignment? Or am I stuck having the physical and logical order forever out of alignment?

Thanks in advance for the help!

Kevin
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
So my question is what is happening? I get the value of keeping the same name once FreeNAS is installed but I'm really surprised that this is persisting across a partition wipe of all the drives in the system and a reinstall. I'm guessing that the 9207-8i itself is mapping a drive serial number to some manufactured SAS address and that's being passed to the driver. I took a quick look at sas2flash and did a -reset since that seems to be closest to a factory reset. I don't know if that guess is correct so I wasn't about to start erasing memory segments on the HBA to try to fix this.

Question 2: How do I force a rescan and reassignment? Or am I stuck having the physical and logical order forever out of alignment?
Welcome to the forums!

FreeBSD (the OS on which FreeNAS is based) assigns device names in no particular order, and I don't believe there's any way to force it to remap them.

For example, when I added an Intel DC S3700 SSD as a SLOG device in one of my servers, it got poked into the middle of the 4 existing HDD's at /dev/da3. Before I added it, the four HDD's had been designated as devices da1-da4:
Code:
[root@bacon] ~# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00>  at scbus1 target 0 lun 0 (pass0,cd0)
<VMware Virtual disk 1.0>  at scbus2 target 0 lun 0 (pass1,da0)
<ATA WDC WD20EARX-00P AB51>  at scbus3 target 0 lun 0 (pass2,da1)
<ATA WDC WD20EARX-00P AB51>  at scbus3 target 2 lun 0 (pass3,da2)
<ATA INTEL SSDSC2BA10 0265>  at scbus3 target 4 lun 0 (pass4,da3)
<ATA WDC WD30EZRX-00D 0A80>  at scbus3 target 5 lun 0 (pass5,da4)
<ATA WDC WD20EFRX-68E 0A82>  at scbus3 target 6 lun 0 (pass6,da5)
[root@bacon] ~#
FreeNAS doesn't care about the device order. And it doesn't matter if the device order changes, either. As long as all the disks in your pool are present, you can reshuffle the ID's all you want: FreeNAS will load your pool just fine.

I know... if you're slightly obsessive, like I am, this can be frustrating. :smile:
 

seanacais

Cadet
Joined
Jan 24, 2017
Messages
6
This goes beyond being obsessive (but I will admit, guilty as charged). :smile:

I'm concerned about identification of the drive in response to an error. If a drive fails and you get a message that says /dev/da2 how do you positively identify the physical location if the names are assigned arbitrarily?

Kevin
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
This goes beyond being obsessive (but I will admit, guilty as charged). :)

I'm concerned about identification of the drive in response to an error. If a drive fails and you get a message that says /dev/da2 how do you positively identify the physical location if the names are assigned arbitrarily?

Kevin
The ID's generally 'stay put' unless you add or remove devices.

FreeNAS provides a nice list of devices and associated serial numbers at Storage->View Disks:
view-disks-2.jpg

Armed with this list, you can make the association between device ID and serial number and thus determine which disk to replace in the event of failure.
 

seanacais

Cadet
Joined
Jan 24, 2017
Messages
6
The issue is that I won't be the one replacing the disk. I see the error and I need to be able to tell another (non-technical) person which caddy to pull. I don't see how knowing the serial number helps unless I'm going to stick a tag with the serial number on the drive caddy so they pull the correct disk.

Maybe I'm looking at this incorrectly. There is a physical connection from the HBA to a specific location in the case. That doesn't change. There should be some identifier in BSD that maps it to the /dev/daX name? It's difficult to believe it just makes it up every time it boots. How did it decide in the first place? It appears to me that it took the drives in BIOS order on first boot. It's only because I had the cables backwards that I even noticed the issue.

Kevin
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
stick a tag with the serial number on the drive caddy so they pull the correct disk.
Yes, do this. :)
Maybe I'm looking at this incorrectly. There is a physical connection from the HBA to a specific location in the case. That doesn't change. There should be some identifier in BSD that maps it to the /dev/daX name? It's difficult to believe it just makes it up every time it boots. How did it decide in the first place? It appears to me that it took the drives in BIOS order on first boot. It's only because I had the cables backwards that I even noticed the issue.
I honestly don't know how it determines the order, and it doesn't seem to change unless you add or remove a device (as I mentioned above). For details and deeper understanding, you'll have to consult a FreeBSD guru, which I am not.
 

RickH

Explorer
Joined
Oct 31, 2014
Messages
61
I've seen similar behavior on a couple of my FreeNAS servers, and while I wasn't ever able to get the detected order to match the 'physical order' I did find a bit of a workaround using sas2ircu.

sas2ircu is a small program that will allow you to get physical device information from anything connected to a LSI based controller and do things like flash indicator lights on backplanes... I can't remember if it was ever officially released for FreeBSD, but someone ported it from the .nix build - send me a PM if you can't find it, and I'll send it to you (it wouldn't let me upload it as an attachment).

Running ./sas2ircu list will output a list of all of your LSI controllers along with their ID's (sounds like you'll just have one with an ID of 0)
Code:
[root@Storage01] /scripts# ./sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.


		 Adapter	  Vendor  Device					   SubSys  SubSys
 Index	Type		  ID	  ID	Pci Address		  Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------	------  ------
   0	 SAS2008	 1000h	72h   00h:04h:00h:00h	  1000h   3020h
SAS2IRCU: Utility Completed Successfully.
[root@Storage01] /scripts#



Then run ./sas2ircu <controller #> display to display all of the devices connected - this should provide with a physical mapping to serial number list:
Code:
[root@Storage01] /scripts# ./sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

Read configuration has been initiated for controller 0
------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type						 : SAS2008
  BIOS version							: 7.31.00.00
  Firmware version						: 20.00.04.00
  Channel description					 : 1 Serial Attached SCSI
  Initiator ID							: 0
  Maximum physical devices				: 255
  Concurrent commands supported		   : 3432
  Slot									: 1
  Segment								 : 0
  Bus									 : 4
  Device								  : 0
  Function								: 0
  RAID Support							: No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 0
  SAS Address							 : 500065b-3-6789-abe0
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291DR1Q
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 1
  SAS Address							 : 500065b-3-6789-abe1
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291AYAV
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 2
  SAS Address							 : 500065b-3-6789-abe2
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291D2YB
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 3
  SAS Address							 : 500065b-3-6789-abe3
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z2917Z6Z
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 4
  SAS Address							 : 500065b-3-6789-abe4
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291B6ND
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 5
  SAS Address							 : 500065b-3-6789-abe5
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291CZBC
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 6
  SAS Address							 : 500065b-3-6789-abe6
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291C5HT
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 7
  SAS Address							 : 500065b-3-6789-abe7
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z2919R8T
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 8
  SAS Address							 : 500065b-3-6789-abe8
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: Hitachi HUA72303
  Firmware Revision					   : A8C0
  Serial No							   : MK0361YHJP7VAD
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 9
  SAS Address							 : 500065b-3-6789-abe9
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z290NAH0
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 10
  SAS Address							 : 500065b-3-6789-abea
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291DTAZ
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Hard disk
  Enclosure #							 : 2
  Slot #								  : 11
  SAS Address							 : 500065b-3-6789-abeb
  State								   : Ready (RDY)
  Size (in MB)/(in sectors)			   : 2861588/5860533167
  Manufacturer							: ATA
  Model Number							: ST33000650NS
  Firmware Revision					   : RA05
  Serial No							   : Z291AQMS
  GUID									: N/A
  Protocol								: SATA
  Drive Type							  : SATA_HDD

Device is a Enclosure services device
  Enclosure #							 : 2
  Slot #								  : 255
  SAS Address							 : 500065b-3-6789-abfd
  State								   : Standby (SBY)
  Manufacturer							: LSI CORP
  Model Number							: Bobcat
  Firmware Revision					   : 6A07
  Serial No							   : x3654654855
  GUID									: N/A
  Protocol								: SAS
  Device Type							 : Enclosure services device
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#							  : 1
  Logical ID							  : 500605b0:02af8df0
  Numslots								: 8
  StartSlot							   : 0
  Enclosure#							  : 2
  Logical ID							  : 500065b3:6789abff
  Numslots								: 12
  StartSlot							   : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.
[root@Storage01] /scripts# ./sas2ircu 0 display



You should then be able to match the serial number of the failed disk to the physical connection.

Although it's probably easier just to stick serial number on the front of your drive trays...
 

seanacais

Cadet
Joined
Jan 24, 2017
Messages
6
Thank you for the replies and assistance.

The sas2ircu is one of the utilities you can download from LSI/Avago but I hadn't given it much thought since it's for RAID configurations. After it was mentioned I ran it to get the output of sas2ircu 0 display and noticed that each drive has a SAS ID and GUID. The SAS ID's are basically identical and incremented by 100 in the third section (like so).

SAS Address : 4433221-1-0500-0000
SAS Address : 4433221-1-0400-0000


And they were incremented in the order that the drives were in the first time I connected the drives to the 9207-8i. So that was confirmation (to me) that the HBA had created the identifier.

So running on the theory that there was a mapping in the HBA and having read further about switching from IR to IT modes I decided to erase the flash as fully as possible and reflash the firmware and bios.

So I disconnected all the drives and ran the following

sas2flash -o -c 0 -e 6
sas2flash -o -c 0 -b mpt2sas.rom -f 9207-8.bin


When I then reconnected the drives and installed FreeNAS the physical and logical orders matched.

What was surprising was that the SAS ID of the drives didn't change. When I ran sas2ircu 0 display the SAS ID's of the drives had remained the same as prior to the flash erasure. I'm guessing that the created SAS ID was written into the drive itself which makes sense if you want to be able to identify drives across controllers. Unfortunately I don't have another 9207-8i to test that theory.

So for the moment the physical and logical orders match. But since it's still not clear to me how all this works I guess I'm still going to have to do something like the serial numbers on the caddy. Still doesn't seem right though :).

Does this count as solved?

Again, thanks a ton for the assist.

Kevin
 

seanacais

Cadet
Joined
Jan 24, 2017
Messages
6
Wanted to give one last update to this. It turned out that the 9207-8i controller I was using was damaged in some way. I kept receiving errors on the eighth drive connected to the controller no matter what drive, cable, or chassis it was connected to.

I contacted the vendor who replaced the card and the data errors and all the weird stuff with matching physical and logical order went away.

Thank you again for all the help

Kevin
 
Status
Not open for further replies.
Top