Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

Mellanox passthrough to FreeNAS guest on ESXi

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

wormified

Newbie
Joined
Aug 19, 2020
Messages
3
Hi, I have a FreeNAS guest running on ESXi with a single 50 GbE port on a ConnectX-5 card passed through to it. FreeNAS doesn't show the interface and dmesg -a | grep mlx returns:

mlx5_core0: <mlx5_core> mem 0xe4000000-0xe5ffffff irq 19 at device 0.0 on pci6
mlx5_core0: INFO: firmware version: 12.24.1000
mlx5_core0: ERR: enable msix failed
mlx5_core0: ERR: mlx5_load_one failed -6

I've attached a debug dump from the install as well. Any help would be greatly appreciated!
 

Attachments

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,538
Moderator note: Moved from Networking Forum to Hypervisors subforum because this is not a Networking question, as it is clearly an issue between FreeNAS, ESXi, and a bit of hardware. The fact that the bit of hardware is a network card does not make it a Networking question (sorry!).

Virtualization guru comment: Try disabling msi. See for ex. https://nvidia.custhelp.com/app/answers/detail/a_id/4135/~/vmware-vdga-/-gpu-passthrough-requires-that-msi-is-disabled-on-vms but your specifics may be different.
 

wormified

Newbie
Joined
Aug 19, 2020
Messages
3
I'm sorry! Will try to be more mindful of the categorization in the future.

I tried disabling msi for all passthru devices (for simplicity) and it didn't affect this behavior. One thought is that perhaps the Mellanox drivers expect msix? That seems to be the case for some devices with Windows guests:


There also seems to be some discussion of whether or not MSI-X and its implementation in BSD vs ESXi is broken in some way? Although this discussion is a bit beyond me tbh:

The MLX4 driver documentation suggests it's possible to disable MSI-X:
But I'm not sure how this interacts with passthru, especially since I'm only passing through one port on a two port adapter.

This card was working under ESXi and with traffic routed through a vswitch, but FreeNAS saw that portgroup only as 10 Gb and I get line rates at that speed (~1.1-1.3 GB/s). I'm hoping to get more out of the 50 Gb pipe that's available hence why I'm trying to pass the 50 Gb port in directly, but I wonder if I need to either pass the whole card through or enable SR-IOV (which I can't without upgrading beyond my essentials license) to do this properly. At that point it would be cheaper just to add an additional 50 Gb card dedicated for the FreeNAS guest.
 

wormified

Newbie
Joined
Aug 19, 2020
Messages
3
Confirmed that this error persists even when passing through both ports of the same card.
 
Top