Intel VF NICs broken in Cobia

[bryan_v]

Just did a test migration of Bluefin to Cobia and when trying to bring the up the VFs for the virtual machine, I got hit with a super random error:

Code:

root@truenas[~]# echo 4 > /sys/class/net/enp1s0f0/device/sriov_numvfs
echo: write error: cannot allocate memory
root@truenas[~]# dmesg
...
[220313.650288] ixgbe 0000:01:00.0 enp1s0f0: SR-IOV enabled with 4 VFs
[220313.653027] ixgbe 0000:01:00.0: removed PHC on enp1s0f0
[220314.005310] ixgbe 0000:01:00.0: Multiqueue Enabled: Rx Queue count = 4, Tx Queue count = 4 XDP Queue count = 0
[220314.033706] pps pps0: new PPS source ptp0
[220314.034851] ixgbe 0000:01:00.0: registered PHC device on enp1s0f0
[220314.163844] ixgbe 0000:01:00.0: can't enable 4 VFs (bus 02 out of range of [bus 01])
[220314.164395] ixgbe 0000:01:00.0: Failed to enable PCI sriov: -12

A couple of quick questions so I can try to track down the source:

• Does anyone know what linux kernel version Cobia is using and which version of the ixgbe driver it got bumped to?
  (Should also help determine why the latest version of the ixgbe driver cannot be recompiled for Cobia - I get a build error: 'VXLAN_HEADROOM' undeclared - it's a known bug in certain version of Debian 11)
• Also what is the best way to add additional kernel parameters? sysctl gets angry whenI try to tell it to add the param "pci=realloc,assign-busses" saying 'pci' is not a valid param

We use VFs in our TrueNAS systems that underpin our storage and development VMs/containers. VFs allow us to create network level isolation via VXLAN without too much messing arround with TrueNAS's network stack.

FYI the test systems is:

• AsRock Rack X470D4U-2T
• Ryzen 5 3600
• Intel X550-2T NIC

Thanks in advance!
Bryan