SR-IOV NIC + jail/bhyve

[Meirick]

Hello,

I need help for my future configuration, today it is just a project.
My target is to use FreeNAS and 4 jails + maybe 1 bhyve VM.

My problem is the network for the host and the guests. I would like to separate everything by using 1 Vlan for each VM/host. For the bandwidth, we need 3GBE for the NAS and around 1GBE for each jails/VM.

For the hardware, I would like to use a 10GBE NIC (like X520-SR2). I would like to know if we can use a sr-iov capable NIC to:
- Create xx VF :
one VF for the NAS
one VF for each jail and bhyve's virtualization
- assign 1 Vlan per VF
- set bandwidth limit for each VF

I found this document, but I need more information that I didn't found.
How to set up the Vlan ID and limit the bandwidth for each VF? Could these features be done directly by the card sr-iov feature?
If not possible, what is the best way to respect my request?

In some thread, they are issue by using VF attached to Freebsd guest and a ESXi host. Do will we have the same issue with Freenas?

Meirick

 

[dlavigne]

Have you tried to configure this yet? If so, what luck did you have with it?

 

[jgreco]

Hello,

I need help for my future configuration, today it is just a project.
My target is to use FreeNAS and 4 jails + maybe 1 bhyve VM.

My problem is the network for the host and the guests. I would like to separate everything by using 1 Vlan for each VM/host. For the bandwidth, we need 3GBE for the NAS and around 1GBE for each jails/VM.

For the hardware, I would like to use a 10GBE NIC (like X520-SR2). I would like to know if we can use a sr-iov capable NIC to:
- Create xx VF :
one VF for the NAS
one VF for each jail and bhyve's virtualization
- assign 1 Vlan per VF
- set bandwidth limit for each VF

I found this document, but I need more information that I didn't found.
How to set up the Vlan ID and limit the bandwidth for each VF? Could these features be done directly by the card sr-iov feature?
If not possible, what is the best way to respect my request?

In some thread, they are issue by using VF attached to Freebsd guest and a ESXi host. Do will we have the same issue with Freenas?

Meirick

This isn't how the technology works.

A virtual function is essentially a virtual ethernet card passed through to a virtual machine. The silicon of a real ethernet card just splits itself up into a number of virtual functions so that it looks like a bunch of physical ethernet cards, which may then be passed through to hypervisor guests. This gives each hypervisor guest direct access to the physical ethernet silicon. (I'm trying to keep the idea clear here so let's not have any unnecessary pedantry please.)

I don't think you can attach multiple VF's to a single FreeBSD host. This may not be a technical impossibility. If you mean that the FreeBSD host is actually an ESXi guest, then that might be more complicated. I don't think ESXi will let you do this but I don't think I've tried either.

You don't "assign [..] vlan per VF". It's an ethernet card. The VF consumer is responsible for figuring out any vlan complications.

You don't "set bandwidth limit for each VF". It's an ethernet card. There are no ethernet card settings that say "limit my 10GbE card to 1GbE". The VF consumer is responsible for managing this.

What is the reason for wanting to assign each VF to a different vlan? Without some idea of what the actual intent/goal is here, any advice is just as likely to be useless or meaningless.

 

[Ericloewe]

I don't think you can attach multiple VF's to a single FreeBSD host.

Note from the future: you should be able to - SR-IOV merely exposes Virtual Functions, these are typically passed-through via VT-d, but are fully functional from the host. This is only really useful when used in combination with containers, such as jails.

You don't "assign [..] vlan per VF". It's an ethernet card. The VF consumer is responsible for figuring out any vlan complications.

Some NICs allow for the host to impose limitations on what VFs can do. This gets deep into device-specific territory, so treat it accordingly.