LSI-9305-24i and automatic firmware update to wrong version

ajgnet

Explorer
Joined
Jun 16, 2020
Messages
65
I re-installed TrueNAS Core this morning from the source ISO (TrueNAS-12.0-BETA2.1) and on installation, the firmware of my 2x LSI-9305-24i controllers were re-flashed to a version that did not allow any drives to be detected. The only way I was able to restore functionality was to re-flash back to the stock version of the firmware from the Broadcom website (16.00.11.00 -- same version as before). Now, everything works as normal.

Am I crazy to think that TrueNAS automatically updates HBA firmware on installation? The card may have been misidentified in this particular case, breaking it.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
TrueNAS Enterprise, formerly known as simply TrueNAS, did that to so extent, since the hardware configurations were tightly controlled by iX and support was standing by to help if something broke.

This may have leaked out into TrueNAS Core.

@Kris Moore, @mav@, are the automatic firmware updates for LSI HBAs still a thing?
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
@Ericloewe Yes, in TrueNAS Core 12 should also update HBA firmware now on first boot after install/update.

@ajgnet Could you specify what version you had there originally and what was flashed to cause the problem? Upcoming 12.0-RC1 should flash the same 16.00.11.00-IT from the Broadcom site.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
It was not intentional, mostly a side effect of the code unification, but we decided to keep it, since every time somebody report disk-related issue, the firmware version is the first question.
 

ajgnet

Explorer
Joined
Jun 16, 2020
Messages
65
@ajgnet Could you specify what version you had there originally and what was flashed to cause the problem? Upcoming 12.0-RC1 should flash the same 16.00.11.00-IT from the Broadcom site.
I was running 09.00.101.00. I have 2x 9305-24i HBAs in the machine. After the automatic update during installation, both failed to detect the connected drives. Only after a manual re-flash to 16.00.11.00-IT did the controllers operate normally. I wish I could re-test and report more information, but it took me by surprise and this is all I can report, unfortunately.
 

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
It was not intentional, mostly a side effect of the code unification, but we decided to keep it, since every time somebody report disk-related issue, the firmware version is the first question.


I would disagree with that approach; it could have unintentional consequences and actually lead to legal trouble if you're making that kind of modification to a user's system.

Such as, what if that system is still under warranty, and the vendor, noting that it was not the firmware that shipped with the device, decides to deny the claim? For example, Supermicro Support told me flat out that the 3008-IR controller on my X11 board, if flashed to IT, would void the warranty. Should my board go south for any other reason, I'm out that expense.
Also consider the case where should a firmware update go wrong; is iX prepared to remedy the situtation?

I would strongly recommend not having automatic firmware 'adjustments' be part of any operation that involves non-controlled hardware; IE, leave it out of 12 Core; and perhaps consider leaving it out of the operating system entirely.
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
The update script is very particular about the board model name. It does not touch anything it does not know, i.e. for what there are firmware images on Broadcom site. It does not forcefully wipe flash, it does not cross-flash and it should not touch onboard controllers, since they usually have different board names for different firmware images.
 

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
The update script is very particular about the board model name. It does not touch anything it does not know, i.e. for what there are firmware images on Broadcom site. It does not forcefully wipe flash, it does not cross-flash and it should not touch onboard controllers, since they usually have different board names for different firmware images.

While that does drop the scope a bit to whoever is unlucky enough to have a controller that matches what iX has used in TrueNAS systems before; I still have to question the approach.

Sure it can cut down on ticket churn, but the legal implications could be significant. I'd recommend putting this through to iX's legal team before others run into the issue because I imagine if this is not in beta / use-at-your-own-risk-disclaimer (and even then...), then there's some opportunities for not only bad experiences, but also bad press, and potentially someone seeking damages; especially if the process borks a controller card or nukes a system; even then, say the firmware you include with TrueNAS Core ends up with a vulnerability or bug; user later removes TrueNAS and that becomes a highlight. Who is at fault?

I would take this thread's existence to imply how bad of an idea it would be to include firmware updates/flash. Sorry to be negative on this, but take a poll or something, then get legal to sign off.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
I get where you're coming from, but I don't think I've ever heard of anyone who managed to brick an LSI SAS controller.
 

jasonsansone

Explorer
Joined
Jul 18, 2019
Messages
79
While that does drop the scope a bit to whoever is unlucky enough to have a controller that matches what iX has used in TrueNAS systems before; I still have to question the approach.

Sure it can cut down on ticket churn, but the legal implications could be significant. I'd recommend putting this through to iX's legal team before others run into the issue because I imagine if this is not in beta / use-at-your-own-risk-disclaimer (and even then...), then there's some opportunities for not only bad experiences, but also bad press, and potentially someone seeking damages; especially if the process borks a controller card or nukes a system; even then, say the firmware you include with TrueNAS Core ends up with a vulnerability or bug; user later removes TrueNAS and that becomes a highlight. Who is at fault?

I would take this thread's existence to imply how bad of an idea it would be to include firmware updates/flash. Sorry to be negative on this, but take a poll or something, then get legal to sign off.

Regardless of it being a good or bad idea, there isn't any concern from a legal standpoint. The software is already free, being used as-is. The EULA covers this.

Code:
THE PRODUCT IS PROVIDED "AS IS" AND WITH ALL FAULTS AND DEFECTS WITHOUT WARRANTY OF ANY KIND. TO THE MAXIMUM EXTENT PERMITTED UNDER APPLICABLE LAW, IXSYSTEMS, ON ITS OWN BEHALF AND ON BEHALF OF ITS AFFILIATES AND ITS AND THEIR RESPECTIVE LICENSORS AND SERVICE PROVIDERS, EXPRESSLY DISCLAIMS ALL WARRANTIES, WHETHER EXPRESS, IMPLIED, STATUTORY, OR OTHERWISE, WITH RESPECT TO THE PRODUCT, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, AND NON-INFRINGEMENT, AND WARRANTIES THAT MAY ARISE OUT OF COURSE OF DEALING, COURSE OF PERFORMANCE, USAGE, OR TRADE PRACTICE. WITHOUT LIMITATION TO THE FOREGOING, IXSYSTEMS PROVIDES NO WARRANTY OR UNDERTAKING, AND MAKES NO REPRESENTATION OF ANY KIND THAT THE PRODUCT WILL MEET THE LICENSEE'S REQUIREMENTS, ACHIEVE ANY INTENDED RESULTS, BE COMPATIBLE, OR WORK WITH ANY OTHER SOFTWARE, APPLICATIONS, SYSTEMS, OR SERVICES, OPERATE WITHOUT INTERRUPTION, MEET ANY PERFORMANCE OR RELIABILITY STANDARDS OR BE ERROR FREE, OR THAT ANY ERRORS OR DEFECTS CAN OR WILL BE CORRECTED.

TO THE FULLEST EXTENT PERMITTED UNDER APPLICABLE LAW: (A) IN NO EVENT WILL IXSYSTEMS OR ITS AFFILIATES, OR ANY OF ITS OR THEIR RESPECTIVE LICENSORS OR SERVICE PROVIDERS, BE LIABLE TO LICENSEE, LICENSEE’S AFFILIATES, OR ANY THIRD PARTY FOR ANY USE, INTERRUPTION, DELAY, OR INABILITY TO USE THE PRODUCT; LOST REVENUES OR PROFITS; DELAYS, INTERRUPTION, OR LOSS OF SERVICES, BUSINESS, OR GOODWILL; LOSS OR CORRUPTION OF DATA; LOSS RESULTING FROM SYSTEM OR SYSTEM SERVICE FAILURE, MALFUNCTION, OR SHUTDOWN; FAILURE TO ACCURATELY TRANSFER, READ, OR TRANSMIT INFORMATION; FAILURE TO UPDATE OR PROVIDE CORRECT INFORMATION; SYSTEM INCOMPATIBILITY OR PROVISION OF INCORRECT COMPATIBILITY INFORMATION; OR BREACHES IN SYSTEM SECURITY; OR FOR ANY CONSEQUENTIAL, INCIDENTAL, INDIRECT, EXEMPLARY, SPECIAL, OR PUNITIVE DAMAGES, WHETHER ARISING OUT OF OR IN CONNECTION WITH THIS AGREEMENT, BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, REGARDLESS OF WHETHER SUCH DAMAGES WERE FORESEEABLE AND WHETHER OR NOT IXSYSTEMS WAS ADVISED OF THE POSSIBILITY OF SUCH DAMAGES; (B) IN NO EVENT WILL IXSYSTEMS’ AND ITS AFFILIATES', INCLUDING ANY OF ITS OR THEIR RESPECTIVE LICENSORS' AND SERVICE PROVIDERS', COLLECTIVE AGGREGATE LIABILITY UNDER OR IN CONNECTION WITH THIS AGREEMENT OR ITS SUBJECT MATTER, UNDER ANY LEGAL OR EQUITABLE THEORY, INCLUDING BREACH OF CONTRACT, TORT (INCLUDING NEGLIGENCE), STRICT LIABILITY, AND OTHERWISE, EXCEED THE TOTAL AMOUNT PAID TO IXSYSTEMS PURSUANT TO THIS AGREEMENT FOR THE PRODUCT THAT IS THE SUBJECT OF THE CLAIM; (C) THE LIMITATIONS SET FORTH IN THIS SECTION SHALL APPLY EVEN IF THE LICENSEE'S REMEDIES UNDER THIS AGREEMENT FAIL OF THEIR ESSENTIAL PURPOSE.


Even if someone was motivated to sue, get past a 12(b)(6) Motion to Dismiss, and make it to discovery, the claimant would still need to prove up damages. Every HBA I am aware of sells used for less than one billable hour of attorney's fees. A new, top of the line Broadcom tri-mode HBA is less than four billable hours. I wouldn't suggest anyone retain a lawyer to sue a corporation for such a small amount in controversy with virtually no chance of recovery.

Personally, I love to hear that TrueNAS saves me the headache of tracking firmware updates for various HBA's. Reboot, update, move on. I think it is a great feature.
 
Last edited:

jenksdrummer

Patron
Joined
Jun 7, 2011
Messages
250
OK, sounds like you guys have it handled. Still think it's some shady shit.
 
Last edited:
Top