Updating mpt3sas?

[im.thatoneguy]

In the last version I believe I used dpkg to just install an updated Broadcom driver. The current mpt3sas version is very old and I had an issue that was solved with a combination of updating driver and firmware. (Not sure which fixed it). Did I dream that? I can't seem to be able to update my hba drivers anymore.

 

[jgreco]

TrueNAS is an appliance firmware and you are not supposed to be trying to install updated drivers. This makes technical support very difficult because you create a unicorn platform that behaves differently from everyone else's. The developers at iXsystems put a lot of time and effort into validating particular combinations of OS driver/card firmware and do not always run the latest driver or firmware; the drivers and firmware specified are certified to work with the hardware that iXsystems sells as TrueNAS Enterprise. You are best off buying hardware that closely resembles what iX is selling, rather than picking some rando card (even if it is LSI/Broadcom) and then needing to bang on different drivers or firmware to make it work.

A list of the chipsets expected to work is at

Building new TrueNAS server with Broadcom SAS3016 PCIe card

Hi, I am new in TrueNAS community, but I am not new in server/storage world. I have on my disposal new Supermicro AOC-S3816L-L16iT, AOC-S3216L-L16iT, AOC-S3616L-L16iT cards, which works in IT/HBA mode. Can I use any or all of these controllers, or they are too new and/or not supported? I am...

www.truenas.com

But I will also note that it claims SAS3108 which is something I wouldn't expect to work. Mmm.

 

[im.thatoneguy]

Kind of a Catch-22 aint it then though?

"You need to be on the exact drivers as shipped with the platform for support. But we don't support any hardware but our own."

I get that they don't want people changing ix-Systems drivers because that's part of the support agreement. "We've thoroughly tested these exact drivers with this exact hardware." but if you aren't running their hardware and they haven't tested that hardware we need to be able to run "unicorn" software to find a stable driver version for *our* hardware.

 

[jgreco]

Kind of a Catch-22 aint it then though?

No. And I gave you the solution:

You are best off buying hardware that closely resembles what iX is selling,

"You need to be on the exact drivers as shipped with the platform for support. But we don't support any hardware but our own."

This could be interpreted as entitled and rude. I'm going to assume you did not mean it this way, but I will still comment that iXsystems is providing you the use of their enterprise grade storage platform for free. There's no reason to assume that they even COULD make it work with everything out there; there's discussion in my article about RAID controllers about why that is a fallacy. Hardware and software manufacturers actually do spend a lot of time trying to validate what's known to work; if you decide to go offroad and try other random stuff, you're really on your own. VMware maintains one of the most complex HCL systems that I am aware of in the industry, identifying specific card variants, firmware revs, driver releases, and ESXi versions, all of which must be lined up for success. Lots of hardware is validated to work only within its own ecosystem, with obvious examples being Dell and HP servers. By comparison, iX isn't purposely tying you down to specific hardware. Let me be clear. They don't give a fsck what you use. But ZFS is a fickle bastard, and ZFS works best -- by far -- when you follow certain rules for operational behaviours, such as using true HBA firmware rather than RAID firmware. And it's not just HBA firmware that matters, but HBA firmware that operates correctly with all drives to pass I/O reliably to the drives. The field rapidly narrows.

My off-the-cuff advice to people is to look at what iX is selling and duplicate that. The reason is simple: they've got a development team that will make it work. Do you have to do this? Obviously not. But when you wander on in with some off-brand HBA and stuff isn't working for you, it puts you at a disadvantage. iX is unlikely to be able to duplicate your env. Forum users are unlikely to be able to duplicate your env. Your problem may not get solved soon, or even ever.

It's basically about being smart when you buy hardware. Buy what is known to work, and your purchase is likely to pay dividends.

 

[im.thatoneguy]

if you decide to go offroad and try other random stuff, you're really on your own.

Which is fine. I don't expect IX-Systems to support me. But getting locked out from supporting myself after extensive testing on my own hardware is what's confusing me. Yeah, it takes a lot of time and effort to validate and test a good firmware/driver/hardware combination. Which is why I want to update my driver to a known-good configuration for my extensively tested setup.

No. And I gave you the solution:

If the solution now is effectively "only use IX Systems Hardware or IX System Clones" then that's a pretty radical shift from what TrueNAS/FreeNAS used to be. Which again, they aren't obligated to be that anymore, but if the only way now to deal with driver bugs is to be on first party hardware (or identical clones) then I need to transition off of TrueNAS to RHEL/Rocky or such where I still have control over my system for such basic things as driver updates where needed.

There's a difference between "This is the validated configuration of hardware\drivers\firmware" and "This is the only allowed configuration of hardware\drivers\firmware" and unless I'm mistaken this lockdown is a new change for Cobia which wasn't documented and has pretty large implications for system maintenance.

 

[HoneyBadger]

@im.thatoneguy What's the specific hardware/firmware/driver combination that's in question here where the included mpt3sas driver isn't cutting it? If there's an optimization or functionality gap in our current setups we'd love to hear about it.

 

[im.thatoneguy]

@im.thatoneguy What's the specific hardware/firmware/driver combination that's in question here where the included mpt3sas driver isn't cutting it? If there's an optimization or functionality gap in our current setups we'd love to hear about it.

All Disks in VDEV Faulted all at once? But no other drives on Backplane?

So I have a pool of 4VDEVs with 7 drives each in RAIDZ2. 2 of the drives suddenly were marked Faulted yesterday. But all of the drives are reporting Read/write Errors. It seems really strange for two disks to simultaneously fail. And even stranger that all 7 drives in VDEV reported R/W errors...

www.truenas.com

This was the discussion previously. In short one VDEV started erroring like crazy.

I simultaneously made a few changes (some physical) but it's run without error since (and I haven't updated TrueNAS for fear of disrupting anything).

System: Supermicro SSG-540P-E1CTR45L
HBA: Broadcom 3808 (IT mode) AOC-S3808L-L8IT-P
Desired Driver version: 45.0 (43.1 ships with Cobia)
Firmware version 25.0
Main Storage (4 x 7 Wide RAIDZ2): Western Digital UltraStar DC HC550 | WDC WUH721816ALE6L4

 

[jgreco]

"This is the only allowed configuration of hardware\drivers\firmware"

This has always been the way it is, since FreeNAS 8 more than a decade ago. The tools to build kernels and do linking are not included with the system, so you were always required to use a particular driver that was already built in, which then dictated firmware, which in turn advised the hardware you should be using. iXsystems does not lock you out from using whatever your little heart desires, but the system and design philosophy behind an appliance OS does not allow for users to willy-nilly pick whatever hardware they want; you need to pick hardware that is known to work with the system. And as has been said probably tens of thousands of time on these forums, that's mostly Intel SATA AHCI or LSI HBA with specific firmware. Also a few other things like Intel PCH SCU that happen to be highly reliable.

Truth be told, you can use ANY FSCKING THING you desire as long as it works correctly. It's just that the solution set there isn't real large. There are far more things that do not work correctly. That's why the Resources section is full of guidance.

I need to transition off of TrueNAS to RHEL/Rocky or such where I still have control over my system for such basic things as driver updates where needed.

See ya.

unless I'm mistaken this lockdown is a new change for Cobia which wasn't documented and has pretty large implications for system maintenance.

Everyone errs. You're mistaken. This has been design policy since the beginning.

 

[Arwen]

I think I summed it up in the "Welcome to TrueNAS SCALE - Beginners Intro";

SCALE is a specialized & targeted OS based on Linux, so installing additional packages on the base OS will likely lead to frustration as they will disappear during an update

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux distro, and updates are by full replacement with a...

www.truenas.com

It might be better to say;

TrueNAS SCALE, (and Core), are NOT OS distros with a NAS application. They are a specialized package that combines an Open Source OS, (Linux or FreeBSD), and NAS application where both have been thoroughly tested together. Updating or changing a distro package, like Python or SSL library, would NOT have been tested with TrueNAS. Thus, it is not supported.


We are seeing a TON of new users, mostly to SCALE, who think TrueNAS is a Linux distro with a NAS application. Then are disappointed or out right angry that is not the case. Some even think they need to force SCALE to be a Linux distro with NAS application.


However, what most disappointed or angry new users to SCALE over look, is that SCALE is designed from the ground up, to have a fully replaceable boot OS. Meaning assuming the configuration has been saved, loss of the boot drive is nothing except lost time. You can restore your NAS to EXACTLY where it was before, by loading up a new boot device and restoring the configuration.

All documented, and clearly supported, because that is exactly what is needed in the Enterprise space.


Trying to replace a normal working Linux distro boot drive from the distro vendor's boot image, is impossible. Their are no mainstream Linux distros that allow saving the entire configuration. Yes, you can perform a full backup of the OS partitions on a Linux server. Then use that for later restore. But, one main point of TrueNAS was easy restore from boot drive failure.

 

[im.thatoneguy]

Everyone errs. You're mistaken. This has been design policy since the beginning.

1) was I imagining being able to install drivers manually in every previous version?
2) was there not a change in Cobia that now prevents that?

You're being a condescending dick and continuously taking the worst possible interpretation of everything I say in response to a very simple and reasonable question.

* My system is unstable.
* I want to install a driver.

I don't want moral judgement. Or useless replies like "don't build your own TrueNAS systems" I want to know how to return to how TrueNAS has worked for years until yesterday.

 

[sfatula]

1) was I imagining being able to install drivers manually in every previous version?
2) was there not a change in Cobia that now prevents that?

1. Yes, I am on non Cobia and I cannot install/build anything without some effort as necessary things are disabled or non existent.
2. No

As noted, Truenas is not a distribution, it is an appliance.

 

[im.thatoneguy]

Well then I'm confused how I did it back in May. I'm not a trueNAS expert so I'm not sure how I just waltzed past all of the previous roadblocks.

I left myself no notes because the process was so trivial that I just expected it to be straightforward and painless again.

 

[HoneyBadger]

I simultaneously made a few changes (some physical) but it's run without error since (and I haven't updated TrueNAS for fear of disrupting anything).

I'm not trying to say specifically what did or didn't fix it, but the "multiple simultaneous changes" means it's hard to nail down exactly which change corrected the issue. It may have been the driver, it may have been HBA firmware, it may have been the physical cable change (if all impacted drives were sharing the lanes of one cable on a non-expanding backplane, this might be it)

If we had a known bad combination (eg: mpt3sas 43.1 + SAS3808) it would likely have cropped up for multiple users. I'm not discounting the possibility of that, and that it's been sheer luck to not encounter it in other builds; just worth considering the possibility that the driver may be a red herring here.

Regardless of all that though, Cobia does have some "higher guardrails" against modification of the base OS. It is possible to toggle a developer mode switch to give you access to all of the necessary pieces, but note that debugs or problem tickets filed with modified drivers/modules will likely be asked to be reproduced on a clean system.

Show : Here Be Dragons

sudo install-dev-tools should give you the powers you seek.

 

[sfatula]

Regardless of all that though, Cobia does have some "higher guardrails" against modification of the base OS. It is possible to toggle a developer mode switch to give you access to all of the necessary pieces, but note that debugs or problem tickets filed with modified drivers/modules will likely be asked to be reproduced on a clean system.

Show : Here Be Dragons

sudo install-dev-tools should give you the powers you seek.

This, it's actually easier in my view on Cobia due to the dev tools "script".

 

[im.thatoneguy]

I'm not trying to say specifically what did or didn't fix it, but the "multiple simultaneous changes" means it's hard to nail down exactly which change corrected the issue. It may have been the driver, it may have been HBA firmware, it may have been the physical cable change (if all impacted drives were sharing the lanes of one cable on a non-expanding backplane, this might be it)

If we had a known bad combination (eg: mpt3sas 43.1 + SAS3808) it would likely have cropped up for multiple users. I'm not discounting the possibility of that, and that it's been sheer luck to not encounter it in other builds; just worth considering the possibility that the driver may be a red herring here.

Regardless of all that though, Cobia does have some "higher guardrails" against modification of the base OS. It is possible to toggle a developer mode switch to give you access to all of the necessary pieces, but note that debugs or problem tickets filed with modified drivers/modules will likely be asked to be reproduced on a clean system.

Show : Here Be Dragons

sudo install-dev-tools should give you the powers you seek.

Thanks, that's what I needed.

Note dkms isn't installed by default. Might be a good inclusion in the "dev tools".

And I agree, it very well may not be the driver. But I'll take an excessive change to the old status quo with near-total-pool-loss.

 

[jgreco]

was I imagining being able to install drivers manually in every previous version?

Pedantically, it's a UNIX system. You can force stuff that is not intended to happen to nevertheless happen by overwriting files. So yes you are ALWAYS "able to install drivers manually" for some value of that term.

But practically, in no version of FreeNAS or TrueNAS ever was this an intended, desired, or offered capability for end users.

was there not a change in Cobia that now prevents that?

No. Nothing now prevents you from doing it for the reasons I outline in the first paragraph above. The manual steps needed to manually do it may now be somewhat more complicated as iXsystems may be trying harder to discourage users because of what I outline in the second paragraph.

You're being a condescending dick and continuously taking the worst possible interpretation of everything I say in response to a very simple and reasonable question.

Is super ironic in light of:

I want to know how to return to how TrueNAS has worked for years until yesterday.

Because TrueNAS didn't work like that for years until yesterday. As outlined by Honeybadger, it may be somewhat more difficult now, but it was never intended functionality for you. And as I pointed out earlier in the thread, this has been this way since at least FreeNAS 8. Please try being polite especially when you're being gently corrected with facts for your incorrectness. It's in the rules.

 

[LarsR]

You propably started out on an early version of scale angelfish, where apt was still enabled by default. During it's lifetime there were many users who borked their system by intalling additional packages or simply runnig apt update && upgrade, because "it's just another linux distro". Therefore iX made the decision to disable apt on later angelfish and bluefin. You could still re-enable it because it just didn't have execute permissions. But now on cobia there are additional security checks in place to prevent the installation of additional packages.

 

[sfatula]

But now on cobia there are additional security checks in place to prevent the installation of additional packages.

Well, it's made harder by the additional checks but then that is overruled and actually made simple by the dev tools command.

So, early Angelfish was when you could in fact install packages and such. I was never on it so was unaware that was ever a thing, thanks!

I think the dev tools command is a good thing as people posting for support can be easily detected. It's a great compromise. At that point, it's on them to deal with lack of support or reverting to standard behavior.

 

[Arwen]

Well, it's made harder by the additional checks but then that is overruled and actually made simple by the dev tools command.

So, early Angelfish was when you could in fact install packages and such. I was never on it so was unaware that was ever a thing, thanks!

I think the dev tools command is a good thing as people posting for support can be easily detected. It's a great compromise. At that point, it's on them to deal with lack of support or reverting to standard behavior.

Yes, I agree that having the dev tools command will help in some cases.

The unfortunate thing is that some people will break their SCALE installation using the dev mode. But, demand or get aggressive for help. When the response is save the configuration and re-install, they may go ballistic because that removes some functionality they added and MUST have working. (Even though that added functionality may very well be the thing that broke SCALE.)

And yes, we can remind them they are in unsupported / uncharted waters. However, as has been seen over the last 2 years, their have been too many SCALE users that just won't accept that answer.


I can sum up the problem:

Linux people think that because SCALE is based on Linux, it MUST be a Linux distro and manageable as such.

 

[im.thatoneguy]

No. Nothing now prevents you from doing it for the reasons

IX Systems just confirmed that cobia has additional blocks. So maybe take this experience to be a little less dismissive of other people having issues.

"I want to know how to return to how TrueNAS has worked for years until yesterday." is not a dickish request. I'm stating a super basic desire. "I just want to be able to do what I could do yesterday" in the face of "No you can't and no you never could."

Anyway, thank you to the people who gave me the real answer instead of gaslighting me pretending that nothing changed and withholding the solution to a basic problem.