Freenas 11.x upgrade to Truenas 12 leaves Aquantia broken.

[woutervddn]

This month it's exactly a year ago that we upgraded to a new NAS with 10Gbps at our (small) company this time running FreeNAS instead of Windows.
Mainly due to the cost we did some testing what the impact would be in the months before that using some Asus xg-c100c 10G cards. They worked perfectly.
On November 11th I installed the NAS and loaded the Aquantia driver myself. Everything worked perfectly for almost a year. Yesterday I made the poor decision of upgrading to TrueNAS 12 during the weekend. This has given me short night and some stares from the wife.

I think the old .ko file was still somewhere inside the system and it didn't get upgraded but I might be mistaken. In any case I got greeted that there was a version mismatch between the atlantic driver and the kernel version. Last time it was easy to fix this issue by just grabbing a .ko that was found online. Now I found somebody who said that I should just add the if_atlantic_load="YES" tuneable. So I did, but I was still not seeing the card.

Thats when I started freewheeling, I removed my old .ko file from my user home, downloaded the amd64 tar archive with the driver and copied it to:
- `/usr/`
- `/boot/modules/`
- `/boot/kernel/`

Obviously with complete disregard of whether or not there already was such a file to begin with in those locations.
At this point I lost remote connection to the system and I know I'm going in to work early tomorrow.
Any thoughts on how to get this back up and running?

I'm fluent in Debian based Linux systems, but BSD is like your second girlfriend at the beginning of the relationship. She's gorgeous and all, but you're still figuring out what makes her tick and what makes her mad...

 

[woutervddn]

I solved it.

In the end I just downloaded the latest version of TrueNAS to a thumb drive and reinstalled it to the boot drive. Next I re-uploaded my config (which I downloaded right before the upgrade) and got it back up again.

Thank god for the formidable config export & import function!

That being said, I'm still at the point that I don't know how what exactly went wrong and/or how I could have mitigated it without reinstalling. So advice on how to handle this in the future is still very much welcome!

 

[jgreco]

The best advice is to remove the Aquantia card and install a supported card.

FreeNAS and TrueNAS are appliances. You are not supposed to be tinkering inside the appliance, it is designed as a black box with "no user serviceable parts inside" and we warn people that things can and will break if you go modifying the appliance "firmware". You just experienced that.

Because the developers cannot possibly anticipate all the changes people might want to make and have made over the years, it is entirely up to you to make sure that your offroad adventures are safe. There's no particular guarantee that there will be an updated Aquantia driver available, the company might go out of business, or the driver might develop a problem on newer FreeBSD versions as things change inside the kernel. I doubt any of the developers at iXsystems are testing on this card, because it isn't even supported by FreeBSD.

That said, as long as you're willing to own the results of your offroad adventures, such as the problem you just encountered, that's fine.

Generally I don't suggest this sort of offroading for commercial applications, because there is more of a chance for something unusual to go wrong. Driver crashes, kernel panics, pool is left in a damaged state due to a ZFS bug ... not out of the realm of stuff that's happened. Personally I'd rather just stick in a nice Chelsio T520-CR which can often be found for about $150, which is a card that iXsystems ships with their TrueNAS systems, or a used Intel X520 for about $80, or even a used Solarflare SFN6122 for about $25.

The downside to picking random hardware is that you never know how well supported it is. If you pick a T520-CR or X520 card, you know that there are thousands of these installed in FreeNAS systems around the world that have billions and billions of aggregate problem-free run hours.