SR-IOV functionality broken

[Hash]

As posted previously http://downloadmirror.intel.com/20927/eng/e1000.htm talks about the e1000 driver. It is a linux version but doubt that would change what a driver does.

If you look at the sr-iov section it describes how sr-iov works and what drivers it needs to work - ill quote the part that makes me believe that is the correct driver.

The igbvf driver supports 82576-based virtual function devices that can only be activated on kernels that support SR-IOV. SR-IOV requires the correct platform and OS support.

The igbvf driver requires the igb driver, version 2.0 or later. The igbvf driver supports virtual functions generated by the igb driver with a max_vfs value of 1 or greater. For more information on the max_vfs parameter refer to the section on the the igb driver.
The guest OS loading the igbvf driver must support MSI-X interrupts.

Although this isn't the igb driver showing the version, the igbvf driver is showing Version 1.1.4. That is why I wanted to update the driver. Hope this clears some things up =]

 

[cyberjock]

As posted previously http://downloadmirror.intel.com/20927/eng/e1000.htm talks about the e1000 driver. It is a linux version but doubt that would change what a driver does.

That is an assumption that is more often wrong than right from my experience.

I've read up more on this SR-IOV technology and it seems that this is a relationship that is established between your hardware, hypervisor, the guest OS, and the driver used. From everything I read the driver for that hardware is the last link in the chain, and the correct driver for the hardware needs to be used. It also requires that the driver be written to properly support SR-IOV. I can find nothing that a different driver should be used other than the one for that hardware. So I'm going to agree with jgreco on this. The igb driver seems to be the correct driver for the hardware in question. So the igb driver should be used and any thought revolving around using the e1000 driver is going to end in disappointment because it's the wrong driver for the hardware.

I also wouldn't be the least bit surprised if FreeNAS is doing something weird that is making SR-IOV impossible. I don't think that the developers have ever tested SR-IOV to ensure it would work. As a rule, they don't support FreeNAS in a VM at all. So while I commend your determination to figure this out, I think you are fighting something that you may not be able to win just by the nature that it isn't(shouldn't?) work anyway.

 

[jgreco]

ill quote the part that makes me believe that is the correct driver.[...]Hope this clears some things up =]

Words fail me.

The igbvf driver supports 82576-based virtual function devices

PLEASE READ THAT SENTENCE. DO YOU HAVE AN 82576-BASED VIRTUAL FUNCTION DEVICE? NO. YOU HAVE AN X540 BASED VIRTUAL FUNCTION DEVICE.

I guess words actually don't fail me.

 

[jgreco]

I've read up more on this SR-IOV technology and it seems that this is a relationship that is established between your hardware, hypervisor, the guest OS, and the driver used. From everything I read the driver for that hardware is the last link in the chain, and the correct driver for the hardware needs to be used. It also requires that the driver be written to properly support SR-IOV.

Yeah, so, what you've probably read is likely as confusing as hell, so I'll see if I can clarify.

PCI-Passthrough (VT-d hereinafter) was designed to allow the hypervisor to pass through a single hardware device to a guest OS. This was very useful in environments where latency or performance as a result of using a virtual disk or virtual Ethernet device caused poor performance for a virtual machine. We here also understand that by giving direct access to the HBA controller you get all the other juicy benefits (S.M.A.R.T. etc) from that controller. The takeaway that we need for the purposes of this discussion is that direct access to the hardware (instead of a VMware virtual driver or whatever) offers potentially large performance benefits.

Of course someone tried to be clever, and wanted each of their key VM's to have a VT-d Ethernet interface. So then sometimes we'd see this big monster hypervisor with three quad ports in it eating up a quarter of a switch worth of ports. Ewww.

Worse, the cost-per-port of data-center-grade switch ports isn't that cheap, and this was never going to be sustainable with 10GbE. It isn't that you couldn't do it, it was that it'd cost a small fortune.

Now a PCI device is (incredibly oversimplified for the sake of discussion) a set of registers and a bus interface; you poke some values into the registers that tell it to transmit an Ethernet packet and it does so, right? So someone made the trite observation that you could have more than one set of registers; you could arrange it so that there were multiple sets of registers, so that you could hand a set of registers to a VM and give it access to the physical hardware, but ALSO still be able to hand other sets of registers to OTHER VM's and the hypervisor too. In practice it is a little bit more complex than that but you're on the right track if you think of it as simply being able to VT-d a bunch of secondary sets of registers on a card. This picture may help make sense of it.

 

[jgreco]

Oops. And no edit on IE11, haha, that's progress.

Ok anyways if you look there, the big NIC with SR-IOV offers its "physical function" (PF) to the virtual machine manager (VMM) which for our discussion would be ESXi.

Now the thing is that the NIC also has a bunch of VF's ("virtual functions"), which can then be offered to VM's. The possible benefit here is that you're avoiding dealing with virtual networking on ESXi, and are getting nearly direct access to the hardware... but in reality, just as with VT-d, ESXi gets to coordinate MSI/MSIX and some other aspects.

So ideally what they would have done is to make the Ethernet card VF look exactly like a physical Ethernet card, but the realities are a bit more complex. The driver knows it is talking to a VF and therefore knows it is unable to affect some things such as the physical port status, IIRC.

If I wasn't so lazy, I'd go try it. VMware does list the X540 (but not the 82576) as being compatible

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2038739

and I do actually have a potential candidate FreeNAS box.

 

[cyberjock]

That's exactly how I read it. The resources that a given PCIe card would normally make to a single OS is basically duplicated to appear to be 20 individual cards to 20 different VMs. But the cheat is that you didn't buy 20 cards. You bought a single card and got to use it in 20 VMs and enjoy the performance benefits you'd normally expect from VT-d. It's like having your cake and eating it too.

 

[Hash]

Really nice diagram and explanation. I did wonder about how it would communicate to other vms using the same network card.

Alright I will say im wrong about the driver xD. I think you are correct this is a battle I cant win cyberjock. The issue is the msix config and ill just have to wait and hope it implemented to work with sr-iov.

Thank you everyone for your input.

 

[cyberjock]

I'm not saying you are wrong. I think you are wrong, but I'm definitely no expert in SR-IOV. Heck, you had to tell me what it was... LOL.

If it makes you happy to continue down the path you are going, you should keep doing it. Maybe you'll learn something that will be helpful later?

 

[spidernas]

Did anyone ever get this to work? I was considering picking up some SR-IOV cards myself to try and improve a few performance issues. I'm running FreeNAS virtualized so I did a quick search first to see if anyone tried this.

Btw, here is an excellent video from Intel explaining SR-IOV for NICs:

​

The benefits are much more than just virtual NICs, it also stops your vCPUs from getting interrupted, and allows for DMA transfers for networking. In fact, it should allow near DMA from VM to VM as the NIC has it's own L2 switch.

I'm actually surprised that there has been an issue, I read somewhere that VMXNET3 support SR-IOV. However, it's not clear if you have to pass through the virtual functions to ensure the hypervisor is not using CPU cycles at all. See this screenshot:

Cheers.

 

[Hash]

I lost interest in sr-iov for freenas and just got a dedicated box. I didn't want to wait for a fix but I doubt it is fixed yet I guess I could load up freenas on my hypervisor and see if your really interested but don't be surprised the bug tracker hasn't been touched in quite some time.

https://bugs.freenas.org/issues/4614

 

[Hash]

Just verified the bug still exists in 9.2.1.6

 

[spidernas]

Thanks for checking Hash.

What did you end up using for a 10G NIC in your dedicated box? Was the driver issue only in terms of VFs?

 

[Hash]

Well I think the issue is that freenas thinks that the 10G NIC is the physical hardware and loads the main driver instead of the VF driver and thats why the error is thrown but I'm not 100% sure. I did try to use the full nic in passthrough mode to freenas to see if the physical hardware would work, and it does, however there is wierd problems with it making it unusable. Changing the ip address wont take affect, then changing it will just add another ip address. It is really odd.

I am using x540-t1. It works fine when it isn't in a vm lol.

 

[spidernas]

That's really strange that passthrough did not work (correctly).

I just ordered a Broadcom BCM57711 from eBay to see if I have better luck with that over your experience with the Intel drivers. VMWare HCL says the BCM57711 supports SR-IOV in vSphere 5.5 and FreeBSD 9 supposedly supports that chipset with the bxe drivers.

I was planning on trying a Neterion card, but vSphere 5.x does not support them and the vendor is no longer providing drivers. Real shame, as they support SR-IOV and you can find them for as low as $35 on ebay.

 

[Hash]

Wow! that's super cheap for an sr-iov card!

I wish you luck. Post back once you work on it. I'm interested to know if its intel or sr-iov in general. Let me know how it worked out. =]