LSI 9300-8i / 9300-16i

[kdaaawg]

Am a noob and am building my first new TrueNAS Core system. I acquired a used SuperMicro Dual Xeon, 12 bays, 128GB ECC, with an LSI 9300-8i that appeared be previously used for FreeNAS. Want to do it "right" and verify my system configuration/versions of everything before putting into production. Your comments and feedback are appreciated.

1) How do I verify the firmware, etc on the LSI 9300-8i are at the correct levels? Seems like the best practice is to boot into the EFI shell, but once there, what steps are needed to verify versions vs flash? What else might need to be updated beside the firmware (BIOS, other?) Can I get some direction as to where to find the most recent downloads for this chip-set?

2) I also acquired a LSI-9300-16i in case of trouble with the LSI-9300-8i. Would the exact same firmware as 8i apply to the 16i, or are there different files? Again, where do I get the appropriate firmware files. The -16i that I purchased came from eBay supposedly came pre-loaded as follows:

Firmware has been updated to version 16.00.12.00.​

BIOS has been updated to version 8.37.00.00.​

x64 BIOS (signed) has been updated to version 18.00.00.00.​

Do these versions seem good for the latest TrueNAS? Is it important to update firmware and BIOS and x64 BIOS of the LSI card? Everything I've read on the forums just speaks to updating the LSI firmware but nothing about updating a BIOS, hence the question. I've never heard of a diff between BIOS and x64 BIOS, can someone explain that as well?

Thanks, this will be a fun project and appreciate all the info many of you have diligently put out there to pave the road for me.

 

[Ericloewe]

1) How do I verify the firmware, etc on the LSI 9300-8i are at the correct levels?

sas3flash -listall should report the firmware and extension ROM versions.

2) I also acquired a LSI-9300-16i in case of trouble with the LSI-9300-8i. Would the exact same firmware as 8i apply to the 16i, or are there different files? Again, where do I get the appropriate firmware files. The -16i that I purchased came from eBay supposedly came pre-loaded as follows:

Firmware has been updated to version 16.00.12.00.BIOS has been updated to version 8.37.00.00.x64 BIOS (signed) has been updated to version 18.00.00.00.

Good question, I could see it going either way. Best to check with a good copy from Broadcom's site, although 16.00.12.00 is not publicly available, and iX is, as far as I know, the only company openly distributing that version. I guess they must have used the -8 firmware and it's the same image.

Do these versions seem good for the latest TrueNAS? Is it important to update firmware and BIOS and x64 BIOS of the LSI card? Everything I've read on the forums just speaks to updating the LSI firmware but nothing about updating a BIOS, hence the question. I've never heard of a diff between BIOS and x64 BIOS, can someone explain that as well?

The vendor is using poor terminology. These "BIOS" references should read "BIOS extension ROM" or "UEFI extension ROM" (also known as Option ROM/OpROM). They are, strictly speaking, optional. On HBAs, they only provide:

• A custom card configuration application (for BIOS)
• Configuration menus to extend the native system firmware setup menu (for UEFI)
• SAS Topology viewer
• Staggered spinup configuration
• Interrupt 13h hook to allow BIOS to access the card for booting (BIOS)
• UEFI LSI/Broadcom SAS3 driver for UEFI, to allow booting and all sorts of UEFI functionality (UEFI)

Long story short, if you don't boot from the card (which I recommend you don't, it's a pain in the ass in some cases), you don't need the extension ROMs. I wouldn't worry about the versions of those if they got the firmware right.

 

[Patrick M. Hausen]

Long story short, if you don't boot from the card (which I recommend you don't, it's a pain in the ass in some cases

Not on point to the topic but a nice "war story" that might still be interesting ...

Among our approx. 100 physical machines are a handful of systems by Supermicro with an LSI controller on board and SAS/SATA backplane. Great, innit?

Now with my involvement in the TrueNAS community and our experience with FreeBSD in general we flashed all these controllers to IT firmware. Still everything working as intended.

Customer asking for an SSD capacity extension. Our base configuration runs from a single 2-way mirror, so we add two more SSDs and expand the pool with another mirrored vdev. Everything's cool and froody.

Fast forward, time for a FreeBSD upgrade - including kernel - on the host. System does not boot. Error message (from my memory) "insufficient replicas" or similar. Get the data centre remote hands plug in a FreeBSD USB drive, boot, import pool, everything's perfectly fine.
Yet, system will not boot from the zroot pool.

OK, cutting short - the FreeBSD boot loader relies on the BIOS to actually read the kernel from disks. The controller in IT mode presents only the first disk to the BIOS.

As long as you are running from a 2 disk mirror, that is not a problem. All information is available on the first disk. If that one fails and the BIOS is not capable of booting from the second, or the FreeBSD boot loader doesn't - no problem. Ask your remote hands to remove the broken SSD and place the remaining one into slot #1, boot, done.

Now what happens when you add another mirrored vdev? Nothing much for existing files in the pool. But as soon as you write a new kernel to the pool these blocks will be spread out across all four disks.

And because the BIOS can read only the first of them it will not be able to load the kernel.

This happened to us with one customer system. We still have a somewhat akward workaround in place. We moved all SSDs one slot forward and added a new one as ada0. Then installed a current FreeBSD with UFS on that medium and finally instructed the bootloader like this:

Code:

vfs.root.mountfrom="zfs:zroot/ROOT/13.2-RELEASE-p2"

Grandad telling war stories over

 

[Ericloewe]

OK, cutting short - the FreeBSD boot loader relies on the BIOS to actually read the kernel from disks. The controller in IT mode presents only the first disk to the BIOS.

You know, I have a Dell R6515 at work, with an HBA330/SAS9300 in a Scooby Doo mask that reminds me of that, but instead of FreeBSD it's ZFS Boot Menu for Linux. Despite much time spent on it, I just could not convince ZBM to mount its stuff, and it would panic. Meanwhile, the box is fine, the install was fine, and I have two R630s with the same HBA330 booting just fine from GRUB (legacy installs). So I gave up and ordered NVMe bits and pieces to just boot the box from NVMe - which is fine, I didn't want SATA/SAS, it was just cheaper this way because Dell charges an arm, a leg and a major organ for NVMe disks, even the relatively puny 960 GB Intel P4610s, so configuring the server with an HBA330 and SATA disks for booting ended up being a few hundred bucks cheaper.

Now I'm wondering if I just had to move the disks to the other end of the backplane. Did you ever check if the behavior was the same with the BIOS and UEFI OpROMs?

Curiously, also at work, an old Supermicro 2U NAS came with the rear 2.5" drive bays connected to the SAS expander and thus the SAS3008. I had to leave it that way because I did not have SATA cables on hand that day and the server had to be shipped off to our remote site ASAP, and since it booted fine, I left it as is. Though they do show up as da0 and da1, so maybe I either got lucky or the Supermicro distributor actually knew what they were doing.

This happened to us with one customer system. We still have a somewhat akward workaround in place. We moved all SSDs one slot forward and added a new one as ada0. Then installed a current FreeBSD with UFS on that medium and finally instructed the bootloader like this:

That's why industrial USB flash drives are a thing and the likes of Swissbit are in business. Of course, they make U.2 carriers for M.2 SSDs plus said SSDs look like a bargain in comparison.