Can't see all my drives

[Brian Blater]

I have a FreeNAS 9.3.1 setup running on a Supermicro MB with two IBM M1015 cards in IT mode. I have two arrays. One with 5 @ 1TB drives and another with 8 @ 1TB drives. System has been running for sometime now, but of course has been giving me the error that I should be running firmware P20 instead of P16. This morning I finally had some time and figured I go about upgrading the firmware and see about upgrading to 9.10.

Anyways, I booted to a RufusDOS USB stick and proceeded to upgrade both BIOS and Firmware on the two cards. I basically ran these commands:
sas2flsh.exe -biosall mptsas2.rom
sas2flsh.exe -fwall 2118it.bin

As far as I could tell everything went just fine with the upgrade of the BIOS and the firmware. I did a sas2flsh.exe -listall and I could see the cards running the new versions of each.

So, I reboot and I noticed during the reboot that there was an error that it couldn't find one of the arrays. Sure enough when checking the GUI it shows the array with 5 drives not available and the array with 8 drives as degraded. I checked the drives and it shows only 7 of the 13 drives. If when getting into the BIOS of the IBM card one card has 3 drives listed and the other has 4 drives listed.

Any ideas on what I can do to get all my drives back or what the problem is? I've ready through some of the forum posting and tried to upgrade the cards again and it still is the same. I could really use the help on getting everything back to normal.

Thank you,
Brian

 

[Ericloewe]

Full system specs, please.

Also, what's the output of sas2flash -listall?

 

[Spearfoot]

I have a FreeNAS 9.3.1 setup running on a Supermicro MB with two IBM M1015 cards in IT mode. I have two arrays. One with 5 @ 1TB drives and another with 8 @ 1TB drives. System has been running for sometime now, but of course has been giving me the error that I should be running firmware P20 instead of P16. This morning I finally had some time and figured I go about upgrading the firmware and see about upgrading to 9.10.

Anyways, I booted to a RufusDOS USB stick and proceeded to upgrade both BIOS and Firmware on the two cards. I basically ran these commands:
sas2flsh.exe -biosall mptsas2.rom
sas2flsh.exe -fwall 2118it.bin

As far as I could tell everything went just fine with the upgrade of the BIOS and the firmware. I did a sas2flsh.exe -listall and I could see the cards running the new versions of each.

So, I reboot and I noticed during the reboot that there was an error that it couldn't find one of the arrays. Sure enough when checking the GUI it shows the array with 5 drives not available and the array with 8 drives as degraded. I checked the drives and it shows only 7 of the 13 drives. If when getting into the BIOS of the IBM card one card has 3 drives listed and the other has 4 drives listed.

Any ideas on what I can do to get all my drives back or what the problem is? I've ready through some of the forum posting and tried to upgrade the cards again and it still is the same. I could really use the help on getting everything back to normal.

Thank you,
Brian

Yikes!

Did you disconnect the drives from the controllers before you flashed them?

Don't know how your system is configured, but if FreeNAS is resilvering any pool(s), let it finish doing that before you do anything else...

Just to eliminate the obvious, I would power the system off and then check and reconnect all of the controller and drive connections.

It would be interesting to see detailed info about the cards. Would you please boot from your Rufus stick and run these commands? Then post the results here, in CODE tags.

Code:

sas2flsh -c 0 -list -l hba-1.txt
sas2flsh -c 1 -list -l hba-2.txt

Or use the -listall command as @Ericloewe suggested. Either way, this should yield the following information for the two controllers, much like this:

Code:

sas2flsh -c 0 -list -l hba-1.log
   Adapter Selected is a LSI SAS: SAS2008(B2)   

   Controller Number  : 0
   Controller  : SAS2008(B2)   
   PCI Address  : 00:01:00:00
   SAS Address  : 5xxxxxx-x-xxxx-xxxx
   NVDATA Version (Default)  : 14.01.00.08
   NVDATA Version (Persistent)  : 14.01.00.08
   Firmware Product ID  : 0x2213 (IT)
   Firmware Version  : 20.00.07.00
   NVDATA Vendor  : LSI
   NVDATA Product ID  : SAS9211-8i
   BIOS Version  : N/A
   UEFI BSD Version  : N/A
   FCODE Version  : N/A
   Board Name  : SAS9211-8i
   Board Assembly  : N/A
   Board Tracer Number  : N/A

   Finished Processing Commands Successfully.
   Exiting SAS2Flash.

Since you've installed the BIOS on the controllers, you should be able to press Ctrl-C at startup and then poke around and see which drives the controllers are seeing. There's an option to have the controllers display all of the drives connected to them at startup, which may be handy.

Hopefully the controllers can be made to detect all of your drives and FreeNAS will either load your pools without problems, or be able to resilver them successfully.

If not... you'll have to figure out the problem, re-create your pool(s), and then restore your data from backup.

 

[Mirfster]

If when getting into the BIOS of the IBM card one card has 3 drives listed and the other has 4 drives listed.

That tells me the issue is not in FreeNAS. The cards themselves are not seeing all the drives.

 

[Brian Blater]

Full system specs, please.

Also, what's the output of sas2flash -listall?

Yikes!

Did you disconnect the drives from the controllers before you flashed them?

Don't know how your system is configured, but if FreeNAS is resilvering any pool(s), let it finish doing that before you do anything else...

Just to eliminate the obvious, I would power the system off and then check and reconnect all of the controller and drive connections.

It would be interesting to see detailed info about the cards. Would you please boot from your Rufus stick and run these commands? Then post the results here, in CODE tags.

Code:

sas2flsh -c 0 -list -l hba-1.txt
sas2flsh -c 1 -list -l hba-2.txt

Or use the -listall command as @Ericloewe suggested. Either way, this should yield the following information for the two controllers, much like this:

Code:

sas2flsh -c 0 -list -l hba-1.log
   Adapter Selected is a LSI SAS: SAS2008(B2) 

   Controller Number  : 0
   Controller  : SAS2008(B2) 
   PCI Address  : 00:01:00:00
   SAS Address  : 5xxxxxx-x-xxxx-xxxx
   NVDATA Version (Default)  : 14.01.00.08
   NVDATA Version (Persistent)  : 14.01.00.08
   Firmware Product ID  : 0x2213 (IT)
   Firmware Version  : 20.00.07.00
   NVDATA Vendor  : LSI
   NVDATA Product ID  : SAS9211-8i
   BIOS Version  : N/A
   UEFI BSD Version  : N/A
   FCODE Version  : N/A
   Board Name  : SAS9211-8i
   Board Assembly  : N/A
   Board Tracer Number  : N/A

   Finished Processing Commands Successfully.
   Exiting SAS2Flash.

Since you've installed the BIOS on the controllers, you should be able to press Ctrl-C at startup and then poke around and see which drives the controllers are seeing. There's an option to have the controllers display all of the drives connected to them at startup, which may be handy.

Hopefully the controllers can be made to detect all of your drives and FreeNAS will either load your pools without problems, or be able to resilver them successfully.

If not... you'll have to figure out the problem, re-create your pool(s), and then restore your data from backup.

Thank you for the suggestions. I had to take a break from the system before I made it even worse or just got to frustrated. When I get back to the system later tonight I'll post up what everyone is asking for.

I guess I should have read more or asked questions before updating the BIOS/Firmware. Didn't even think that I would need to unplug all the drives from the controllers first. Hopefully it isn't too far gone and I can get the arrays back.

That tells me the issue is not in FreeNAS. The cards themselves are not seeing all the drives.

Yeah, I'm pretty sure it isn't a FreeNAS issue at this point, but I would figure this is the best place with the best people to help figure it out as I bet someone here has seen this exact problem before.

Thanks again for the help.
Brian

 

[Mirfster]

Hopefully it isn't too far gone and I can get the arrays back.

Nah, my guess is the data is safe. Once you can access the drives FreeNAS should see the Pool(s) fine.

Yeah, I'm pretty sure it isn't a FreeNAS issue at this point, but I would figure this is the best place with the best people to help figure it out as I bet someone here has seen this exact problem before.

NP, wasn't meaning that you shouldn't have posted here or anything like that. Apologies if it sounded that way.

While it *may* not make a difference, I normally apply the Firmware and BIOS at the same time ( sas2flash -o -f 2118it.bin -b mptsas2.rom). I noticed that you did the BIOS first and then the Firmware. Not saying that the order is incorrect, but maybe try one card at a time:
sas2flash -o -c 0 -f 2118it.bin -b mptsas2.rom
sas2flash -o -c 1 -f 2118it.bin -b mptsas2.rom

Lastly, I don't think having the drive connected should have been an issue. I have flashed them from within FreeNAS itself with drives attached (using "sas2flash").

 

[Brian Blater]

Nah, my guess is the data is safe. Once you can access the drives FreeNAS should see the Pool(s) fine.


NP, wasn't meaning that you shouldn't have posted here or anything like that. Apologies if it sounded that way.

While it *may* not make a difference, I normally apply the Firmware and BIOS at the same time ( sas2flash -o -f 2118it.bin -b mptsas2.rom). I noticed that you did the BIOS first and then the Firmware. Not saying that the order is incorrect, but maybe try one card at a time:
sas2flash -o -c 0 -f 2118it.bin -b mptsas2.rom
sas2flash -o -c 1 -f 2118it.bin -b mptsas2.rom

Lastly, I don't think having the drive connected should have been an issue. I have flashed them from within FreeNAS itself with drives attached (using "sas2flash").

No worries... It's always hard to know exactly what someone means when it is just words on a screen :)

I actually used those commands the second time I tried the firmware and BIOS upgrade.

Now, I'm using the DOS sas2flsh.exe utility as I don't believe my board has efi capabilities.

 

[Brian Blater]

Ok, here is what I have regarding the questions asked.

The specs are as follows: Supermicro 16bay X7DB8+ with 2 @ Intel Quad Core L5420 processors and 16GB RAM part# 836TQ-R800B.

Code:

sas2flsh.exe -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2) 

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)     20.00.07.00    14.01.00.08    07.39.02.00     00:0a:00:00
1  SAS2008(B2)     20.00.07.00    14.01.00.08    07.39.02.00     00:0b:00:00

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Code:

sas2flsh.exe -c 0 -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2) 

    Controller Number              : 0
    Controller                     : SAS2008(B2) 
    PCI Address                    : 00:0a:00:00
    SAS Address                    : 500605b-0-044d-0900
    NVDATA Version (Default)       : 14.01.00.08
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9211-8i
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9211-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Code:

sas2flsh.exe -c 1 -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2) 

    Controller Number              : 1
    Controller                     : SAS2008(B2) 
    PCI Address                    : 00:0b:00:00
    SAS Address                    : 500605b-0-044c-6220
    NVDATA Version (Default)       : 14.01.00.08
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9211-8i
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9211-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

I had a photo of what the cards see for drives when booting, but I can't get stupid Google Images to create a correct link for the forum software (it was easier with picasa).

Thanks,
Brian

 

[Ericloewe]

Didn't even think that I would need to unplug all the drives from the controllers first.

You generally don't, it's just a matter of paranoia - or risk aversion.

I had a photo of what the cards see for drives when booting, but I can't get stupid Google Images to create a correct link for the forum software (it was easier with picasa).

No need, just paste them into your message. External image hosts are discouraged.

As for the output, show us the output with FreeNAS (note that it's sas2flash and not sas2flsh).
At least the cards look fine under DOS.

 

[Ericloewe]

Very important: What drives are these?

 

[Spearfoot]

Ok, here is what I have regarding the questions asked.

The specs are as follows: Supermicro 16bay X7DB8+ with 2 @ Intel Quad Core L5420 processors and 16GB RAM part# 836TQ-R800B.

Code:

sas2flsh.exe -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)     20.00.07.00    14.01.00.08    07.39.02.00     00:0a:00:00
1  SAS2008(B2)     20.00.07.00    14.01.00.08    07.39.02.00     00:0b:00:00

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Code:

sas2flsh.exe -c 0 -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2)

    Controller Number              : 0
    Controller                     : SAS2008(B2)
    PCI Address                    : 00:0a:00:00
    SAS Address                    : 500605b-0-044d-0900
    NVDATA Version (Default)       : 14.01.00.08
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9211-8i
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9211-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

Code:

sas2flsh.exe -c 1 -list
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2008(B2)

    Controller Number              : 1
    Controller                     : SAS2008(B2)
    PCI Address                    : 00:0b:00:00
    SAS Address                    : 500605b-0-044c-6220
    NVDATA Version (Default)       : 14.01.00.08
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 20.00.07.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9211-8i
    BIOS Version                   : 07.39.02.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9211-8i
    Board Assembly                 : N/A
    Board Tracer Number            : N/A

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

I had a photo of what the cards see for drives when booting, but I can't get stupid Google Images to create a correct link for the forum software (it was easier with picasa).

Thanks,
Brian

The boards look good. Did you try loading the Avago/LSI option ROM at startup by pressing Ctrl-C? This will let you see the drives attached to the cards.

 

[Brian Blater]

Very important: What drives are these?

The first set of 5 are Hitachi and the second set of 8 are WD REDs.

The boards look good. Did you try loading the Avago/LSI option ROM at startup by pressing Ctrl-C? This will let you see the drives attached to the cards.

Yes, I can get into the Avago ROM with Ctrl-C. I don't see all the drives there either.

 

[Brian Blater]

So, this has me puzzled. For giggles I decided to return to the previous BIOS and P16 firmware. After doing that the cards show all drives are back again.

I've even booted the system and in FreeNAS it shows that both arrays are up and healthy.

So, why can't I upgrade the BIOS and firmware? Is it a problem with the HDDs or something else. This is just strange.

Brian

 

[Ericloewe]

Which HDDs are missing, specifically? An even mix of both models?

 

[Spearfoot]

Is there any kind of backplane involved here?

 

[Mirfster]

This is weird. Wonder what happens if you use say P19?

Another oddity:
Looking at the downloads from Avago in the "Current" the latest has a date of 04/04/2016. However, in the "Archive" they have the P20 with a date of 05/06/2016... So it looks like they archived a version that was released at a later date?

Might want to double check the version you downloaded as well..

 

[Brian Blater]

Which HDDs are missing, specifically? An even mix of both models?

Not quite. I get 3 drives on one controller and 4 drives on the other. Since the arrays are spread across the controllers it ends up being all drives from the iSCSI array (5 @ 1TB) and 1 (8 @ 1TB) from the NFS array.

This is weird. Wonder what happens if you use say P19?

Another oddity:
Looking at the downloads from Avago in the "Current" the latest has a date of 04/04/2016. However, in the "Archive" they have the P20 with a date of 05/06/2016... So it looks like they archived a version that was released at a later date?

Might want to double check the version you downloaded as well..

I may have to try and download a newer version of P20 then. I don't have anything but P15, P16 and P20 on hand right now.

The other problem I'm seeing now is that my ESX box will not connect to the NFS array, but connects to the iSCSI array just fine. Boy, how I love computers.

 

[Mirfster]

When you say "array", are you talking about a Hardware Raid Array or the FreeNAS Pool/Volume? Might just be me but when I hear the word "array" I always assume Hardware Raid...

/In case anyone want to be a smarty pants... Yes; I do know what Raid stands for and am aware that "array" is part of the acronym... :P

 

[Brian Blater]

When you say "array", are you talking about a Hardware Raid Array or the FreeNAS Pool/Volume? Might just be me but when I hear the word "array" I always assume Hardware Raid...

/In case anyone want to be a smarty pants... Yes; I do know what Raid stands for and am aware that "array" is part of the acronym... :p

Yes, in this case Array and FreeNAS Pool/Volume mean the same thing. I'm definitely not doing a Hardware RAID.

So, I went ahead and downloaded the P20 and BIOS with the 4/5/16 date and installed those on both cards. Again, I'm only seeing 7 of the 13 drives and the only drives that are showing are the WD Red drives. Could be a problem then with the firmware on the other 6 drives I guess??? I may have to see if there is a firmware update for these.

Another question about updating the cards. After flashing the BIOS and firmware do I have to run the command:
sas2flsh.exe -o -sasadd 500605b##### ?

I see that in several postings but wasn't sure that needs to be done each time I flash.

 

[Mirfster]

Nah you should be fine; since you did not wipe the card. If concerned, just run:

From within FreeNAS (or via Putty):
sas2flash -c 0 -list and look for the "SAS Address" entry to verify your card has one.

Could also do sas2flash -c 0 -list | grep "SAS Address" to just grab that line from the output

For from within DOS:
sas2flsh -c 0 -list and look for the "SAS Address" entry to verify your card has one

*** Note: I used "-c 0" in my examples to select card number zero. Since you have two cards, you would also want to run the commands with "-c 1" to check the other card.