9400-16e - Which firmware?

[Jorsher]

I currently have 3 x 8-disk U-NAS 810A enclosures and am in the process of doing the one-disk-at-a-time method of upgrading to larger disks. The first disk took 8 days to complete and scrubs also take forever. I assume a large part of that is due to the SAS HBAs being a bottleneck, so I've decided to replace the controllers with the Broadcom/LSI/Avago/whatever 9400-16e SAS3 HBA.

I've flashed HBAs before to get IT mode, but in this case it's not clear to me what I should be flashing:

>>>9400_16e_Pkg_P24_SAS_SATA_FW_BIOS_UEFI<<<
.\Firmware\HBA_9400-16e_SAS_SATA_Profile.bin
.\SAS35BIOS_Rel\mpt35sas_legacy.rom
.\UEFI_BSD_HII_SAS3.5_IT_X64\mpt35sas_x64.rom
.\UEFI_BSD_HII_SAS3.5_IT_X64\Signed\mpt35sas_x64.rom

I assume I'd flash HBA_9400-16e_SAS_SATA_Profile.bin and one of the UEFI_BSD_HII_SAS3.5_IT_X64? Signed, unsigned, or does it matter? Does the 'IT' in the filename mean it will support target/IT mode?

Thanks

 

[Ericloewe]

I assume a large part of that is due to the SAS HBAs being a bottleneck

Why would you think that? Unless you're using a mountain of HDDs behind expanders (not the case here) or SSDs, I can't imagine even a SAS2 HBA being a bottleneck. Scrubs and resilvers are slowish by virtue of being fairly-random workloads - background tasks, too, to make things slower.

I assume I'd flash HBA_9400-16e_SAS_SATA_Profile.bin

Sounds right, but does Broadcom not include a readme or something in the archive so we can be sure?

one of the UEFI_BSD_HII_SAS3.5_IT_X64? Signed, unsigned, or does it matter

Well, there are two parts to this:
1) Is it necessary? No, unless you want to boot from the cards.
2a) If you do, you'll need a legacy BIOS expansion ROM to boot in BIOS mode, a UEFI expansion ROM to boot in UEFI mode, or both.
2b) What about signed/unsigned? I assume this is for secure boot, so I won't even pretend to understand the intricacies. I expect unsigned would work.

 

[Jorsher]

Thanks for the response.

Why would you think that? Unless you're using a mountain of HDDs behind expanders (not the case here) or SSDs, I can't imagine even a SAS2 HBA being a bottleneck. Scrubs and resilvers are slowish by virtue of being fairly-random workloads - background tasks, too, to make things slower.

Right now I have (3x) 8 x 16TB Exos on a single SAS2 link that only supports SATA2. As I understand from other user's experiences, this HP HBA seems to top out at ~137MB/s per disk with a single link. Whatever the case, the small 8-bay enclosures are only temporary and I'd like to use these cards with 24+ bay enclosures in the future. I'm using 64 drives at the moment and not sure if that qualifies as a mountain, but they aren't all behind expanders yet.

Sounds right, but does Broadcom not include a readme or something in the archive so we can be sure?

Well, there are two parts to this:
1) Is it necessary? No, unless you want to boot from the cards.
2a) If you do, you'll need a legacy BIOS expansion ROM to boot in BIOS mode, a UEFI expansion ROM to boot in UEFI mode, or both.
2b) What about signed/unsigned? I assume this is for secure boot, so I won't even pretend to understand the intricacies. I expect unsigned would work.

No, I'll be booting from TrueNAS on a flash drive, and it'll load these drives. I assumed the ROM was for accessing the card configuration from BIOS or Legacy.

The readme doesn't tell me much to make me feel confident that these are the things I should be flashing. It's clear the ARM architecture ROM is not for me, but knowledge from people that use these already is always helpful.

***********************************************************************
Broadcom Inc SAS3.5 MPT IT UEFI BSD & HII Configuration Utility
***********************************************************************

Supported Chip Names:
=====================
SAS3408
SAS3508
SAS3508_1
SAS3416
SAS3516
SAS3516_1
SAS3616

Release Component:
==================
Binary image name: mpt35sas_x64.rom.

Installation Instruction:
=========================
Use storcli.efi to install the SAS3.5 UEFI BSD & HII Configuration Utility.
UEFI version of storcli can be downloaded from the Support & Downloads section of www.broadcom.com
The command line installation instruction to flash the SAS3.5 UEFI BSD & HII Configuration Utility is:
storcli.efi /c<N> download efibios file=mpt35sas_x64.rom
where <N> is the controller number (starting with zero (0)).
If you need further help, please contact the Broadcom FAE associated with your Organization.

Notes:
1) The SAS3.5 UEFI BSD & HII Configuration Utility does not require Legacy BIOS to be loaded on to the controller.
2) The firmware in the controller should be fully operational for flashing the SAS3.5 UEFI BSD & HII Configuration Utility.
3) The release supports X64 platforms.

 

[Ericloewe]

No, I'll be booting from TrueNAS on a flash drive, and it'll load these drives.

A real SSD, I hope. Most USB flash drives are too sucky to be worth the trouble.

Right now I have (3x) 8 x 16TB Exos on a single SAS2 link that only supports SATA2.

Can you sketch this out? I'm a bit confused, since your original post suggested that you had groups of 8 disks connected up to one HBA each. Even with a single connector and an expander that's just 2-to-1 oversubscription. Between SAS being full-duplex and SATA half-duples and mechanical disks being slow, I doubt a real application would notice the difference.
That said, you mentioned HP HBAs. It could be that you're stuck with some of the crappy stuff HP uses, which has pretty poor drivers.

I assumed the ROM was for accessing the card configuration from BIOS or Legacy.

Well, that is half of the story. The ROMs have two parts:

1. A driver for the HBA that plugs in to UEFI.
2. A configuration utility (a standalone real-mode binary for BIOS and a plugin for UEFI that integrates with the native menus). For BIOS boot, this includes custom boot code with the required driver to boot from the HBA.

The configuration utility doesn't allow for much for non-RAID cards:

• SAS topology viewer
• Basic SAS device info
• Config options for staggered spinup

Since they're of little use and can cause a lot of weird problems, most people agree that the ROMs are best skipped unless you want to boot from the cards. And if you do, don't flash multiple cards with the ROM.

 

[Jorsher]

A real SSD, I hope. Most USB flash drives are too sucky to be worth the trouble.

Yes, a real SSD. Have had a USB boot fail on me before and have old spare SSDs laying around.

Can you sketch this out? I'm a bit confused, since your original post suggested that you had groups of 8 disks connected up to one HBA each. Even with a single connector and an expander that's just 2-to-1 oversubscription. Between SAS being full-duplex and SATA half-duples and mechanical disks being slow, I doubt a real application would notice the difference.
That said, you mentioned HP HBAs. It could be that you're stuck with some of the crappy stuff HP uses, which has pretty poor drivers.

I have one box with the motherboard, onboard LSI-3008 for 8 drives, and an LSI 92xx-8e. There are two other 8-disk boxes, each with a 468406-B21 connected via single SAS link to the LSI 92xx-8e. I understand that it should support 8 drives through that single link without being a bottleneck, but I've seen a user report in testing they only got ~130-140MB/s using this card with a similar configuration. Worst case the new cards will give me 0 benefit now, but I will need them for the next build anyway.

Well, that is half of the story. The ROMs have two parts:

1. A driver for the HBA that plugs in to UEFI.
2. A configuration utility (a standalone real-mode binary for BIOS and a plugin for UEFI that integrates with the native menus). For BIOS boot, this includes custom boot code with the required driver to boot from the HBA.

The configuration utility doesn't allow for much for non-RAID cards:

• SAS topology viewer
• Basic SAS device info
• Config options for staggered spinup

Since they're of little use and can cause a lot of weird problems, most people agree that the ROMs are best skipped unless you want to boot from the cards. And if you do, don't flash multiple cards with the ROM.

Gotcha. I usually use the configuration utility to configure staggered spin-up since these enclosures just have SFF PSUs. Is that not an option without the ROM?

 

[Ericloewe]

each with a 468406-B21

Ok, that could begin to explain some bottlenecks, since it's a SAS1 expander. I'm not sure it'd make a huge difference on scrub performance, though - aside from any non-obvious bottlenecks caused by a crappy expander IC, which is a distinct possibility with this generation.

I usually use the configuration utility to configure staggered spin-up since these enclosures just have SFF PSUs.

Unfortunately, no. But 8 disks should not be a huge deal, what are the PSUs rated for?

 

[Jorsher]

Ok, that could begin to explain some bottlenecks, since it's a SAS1 expander. I'm not sure it'd make a huge difference on scrub performance, though - aside from any non-obvious bottlenecks caused by a crappy expander IC, which is a distinct possibility with this generation.

Unfortunately, no. But 8 disks should not be a huge deal, what are the PSUs rated for?

350w Flex PSU

Thanks for the info. It answered my questions.

 

[Ericloewe]

Sounds to me like you don’t really need to worry about staggered spin up on the external disks. 8 disks at ~30W peak per disk is 240W, well below the PSU’s rating (and it’s Seasonic, so it’s credible). The expander might add 10W at most, so it should be comfortable.
Things would get tight in the server proper, though, depending on the hardware.