TrueNas 13 Core Upgrade Challenge - ib_mthca Hangup

[mbartho]

I recently attempted and upgrade from TrueNAS-12.0-U8.1 to TrueNas 13. After the upgrade, I was not able to get the system to boot. The boot process gets hung up at “ib_mthca: Initalizing ib_mtcha”. I even attempted a fresh install of TrueNas 13, and had the same issue.

Since this is happening on a fresh install I’m assuming it is a hardware comparability issue. This is being installed on a EMC Isilon NL400. Its a SuperMicro built system.

I am not the most proficient in FreeBSD, so any ideas to guide me in the right direction would be appropriated.

 

[Samuel Tai]

In all likelihood, the kernel doesn't have a compiled-in driver for the Mellanox InfiniBand NIC, and there's not a kernel loadable driver shipped in /boot/kernel or /boot/modules. Does your system boot without the InfiniBand NIC?

 

[mav@]

Considering the "autoloading" message, TrueNAS probably does have the driver for some matching Mellanox NIC of yours, but something does not go right there. Unfortunately we can't provide proper support for Infiniband, if that it indeed what your NIC is, we are only including drivers readily available in FreeBSD, so that community could experiment themselves. You may try to set loader tunable hint.ib_mthca.0.disabled=1 via the loader prompt to see whether it allow your system to boot. If yes, you may try to reenable it after via `devctl enable ib_mthca0` (not sure whether you still need to load it manually first with `kldload mthca` or autoloader already do it) to see what happen than.

 

[mbartho]

Thanks for the follow-up. Everything works fine with TrueNAS-12.0-U8.1. Just not TrueNas 13. Were the drivers for this NIC removed in FreeBSD 13?

 

[mav@]

I don't think so. Infiniband (if that is what you are really using) was in modules for quite a while, but all should be available. We have to strip kernel heavily to still have DTrace working. Would there be no driver -- there would be nothing to hang. I'd just like to be sure that you are really using Infiniband (it is not some false-positive) and the problem is really caused by it, not that it is just a random last message on the console.

 

[mbartho]

The motherboard is a Supermicro X8DT6 variant using Intel® 82574L and the 10g NIC is a Chelsio 110-1088-30 10GB 2-Port Adapter. After doing a little digging I found it does have an Infiniband adapter installed. I'll attempt to pull the card out and see what happens.

 

[Volts]

TrueNAS probably does have the driver

TrueNAS 13 provides /boot/kernel/mthca.ko.

I don't have a similar device, but I can kldload it on the nightly truenas/13-stable-74831eea5 just fine. Maybe it really is just hanging the hardware on initialization?

The message KLD '/boot/kernel/mthca.ko' is newer than linker.hints is interesting. Could kldxref have been missed in the build/packaging process?

@mbartho what's the output of `ls -la /boot/kernel/mthca.ko /boot/kernel/ibcore.ko /boot/kernel/linker.hints` on -RELEASE?

 

[mbartho]

I removed the adapter and the system booted fine into TrueNas 13. The part number for the card is 415-0017-08 E.

@Volts I ran the requested on TrueNas 13 with the card removed. Below is the output:

Code:

root@freenas[~]# ls -la /boot/kernel/mthca.ko /boot/kernel/ibcore.ko /boot/kernel/linker.hints
-r-xr-xr-x  1 root  wheel  744600 May 13 18:00 /boot/kernel/ibcore.ko
-rw-r--r--  1 root  wheel  190116 May 13 18:15 /boot/kernel/linker.hints
-r-xr-xr-x  1 root  wheel  266544 May 13 18:00 /boot/kernel/mthca.ko

 

[mav@]

@mbartho You may report it to FreeBSD. Mellanox generally supports their drivers. I am just not sure many care about so old ones. I probably don't, even considering the price of $10 on eBay.

 

[mbartho]

@mbartho You may report it to FreeBSD. Mellanox generally supports their drivers. I am just not sure many care about so old ones. I probably don't, even considering the price of $10 on eBay.

I'm not sure it is worth the trouble. I do not use the adapter and I doubt many do. Thanks for helping me identify the issue.