SR-IOV functionality broken

[Hash]

Currently running ESXI 5.5 with an intel x540-t1. I wanted to give it a vf however this is the error I get below.

dmesg, it shows this x3 times.

VMware memory control driver initialized
ix0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.1.4> mem 0xfd3f8000-0xfd3fbfff,0xfd3fc000-0xfd3fffff at device
0.0 on pci11
ix0: MSIX config error
ix0: Allocation of PCI resources failed
device_attach: ix0 attach returned 6


Any Ideas?


I made a bug ticket https://bugs.freenas.org/issues/4614 but I wanted to post on the forums to see if maybe anyone else has had any issues? I haven't seen much about sr-iov but prob cause most people don't use it XD

 

[cyberjock]

Correct me if I'm wrong, but isn't SR-IOV exclusive of PCIe Passthrough?

The reason I ask is because PCIe Passthrough is the only recommended way of using FreeNAS in a VM.

 

[Hash]

Basically SR-IOV is a virtual PCIe Passthrough, where you can have a single device show up multiple times as a Passthrough. Im using a 10gb network card and currently my cpu is my limit for transfer speeds in esxi at around 2.4gb. With sr-iov I expect much higher network speeds as it will bypass my cpu and have direct access to my memory.

 

[cyberjock]

Ok. Now I'm wondering.. what the difference between PCIe passthrough and SR-IOV? I'm trying to understand this better.

To be honest, right now you've got my "ZOMG this is bad news" meter at like 9/10.

 

[Hash]

So pretty much some devices, network cards mainly, but also im sure there are other PCIe devices that will have it sooner or later if not already, have a function that can make multiple instances of a PCIe passthrough. This is known as SR-IOV. It requires a motherboard that has IOMMU, PCIe device that supports it, as well as an operating system.

In my case, I am using a network card x540-t1, it is a 10Gb card. The only way for all my vm's to utilize this card is by means of SR-IOV, because it will make multiple instances of the 10Gb card.

The reason I want this is because in ESXI, or as far as I know any bare metal virtualization solutions, is because the way networking works is more or less like this....

note may not be 100% accurate but you'll get the point.

in this example DP=data packet going in to a vm from a outside source

DP --> network card --> CPU host machine --> VM CPU --> VM memory

using a network card in passthrough, including SR-IOV would do this

DP --> network card --> VM memory

Because you are bypassing the CPU, your bandwidth would not be limited to or saturated by your CPU.


The main reason a virtual network card has to go to CPU is because it has no physical address to send the data to, so it has to be sent to the CPU to process the data to the correct place. The reason SR-IOV can turn a single device into multiple instances is because it creates virtual addresses for each vm to use for the network card instead of the single physical address it has.



Hope I'v explained it well enough. Think it is clear why I would want this working but highly doubt there are people that would use it so I understand if it isn't really something at the top of the list. I think i'm going to try and work on it this weekend when I have some time. don't know much about the MSIX config however so it may take some time.



*Edit*
Eh, should have read your post better.... you only asked about the difference of PCIe passthrough and SR-IOV... did answer the question but most of it is not what you asked sorry. I don't really want to delete everything I just posted so Im just going to leave it in case anyone asks any questions about anything xD


PCIe passthrough = passed through with the physical address of a PCIe device. Can only have one instance.

SR-IOV = passed through with virtual addresses of the PCIe device. Can have like up to 40 instances.

Well it was around 40. Can't remember the exact number and don't feel like looking it up xD

 

[cyberjock]

Ok. Thanks for the explanations. Interesting technology to say the least. So I'll say these few things and let it be what it is:

1. I don't know if FreeBSD 9 has SR-IOV support(or if you even need it from the OS itself).
2. This really pegs my "this is scary" meter when it comes to FreeNAS in a VM. As it is, the number of people that have lost their pool to the number of people that haven't that used FreeNAS in a VM is probably like 10 to 1. VMs are just a nightmare as it is right now.
3. Considering how new this technology is, I'm not sure I'd want to drop this into my FreeNAS box in the next year or two. It takes time for the kinks to get worked out of new technology. If PCIe passthrough is any example, the amount of hardware that claims to do PCIe passthrough versus the combinations that actually work reliably is incredibly small. I'm kind of viewing SR-IOV as a subset of PCIe passthrough, so I can only imagine this will be even worse. People usually want FreeNAS because they want a box that will be non-stop performance and reliability. It's fairly well documented(and has plenty of examples) that a VM isn't the best idea in the world for storage servers- ZFS based servers in particular.
4. I dealt with my own ESXi issues. In particular my issues were with MSI and interrupt storms. Your error, if I saw that with the system I was testing, generally meant that the relationship between your motherboard, the card you are intending to pass through, and the driver that FreeNAS uses don't have a working relationship. In my case, it should have worked, but didn't. I never dug deeper. I simply abandoned the whole idea and went with a whole different system and reused none of the parts from the old system.

I can see how, for networking, this could be totally awesome. I've got a single 10Gb link in my house between my server and my desktop. Being able to do what you are explaining could be really awesome for situations like mine. But, my pucker factor would be really high before I'd actually try it. And I'd definitely have a full backup of my data before I even considered it.

 

[Hash]

Thanks for the reply, had a feeling it maybe the driver. Saw that someone has been compiling the vmxnet3 driver for multiple builds of freenas, it may be a good start.

Have had the interrupt storms when using multiple cores. Not really sure whats up with that. It seems to work great with a single core. I use raid card in PCIe passthrough. Have had multiple issues but I don't think most of them has been due to ESXI. I do keep a backup that syncs daily so Im not too worried about losing my pools. Overall really happy with this setup. When I get more space I will prob move it to a dedicated machine but that isn't an option at the moment.

Do think freebsd added support for it, remember reading something about it. Think I read it was working in freenas as well somewhere.

If I make some progress with it ill be sure to post back!

 

[jgreco]

SR-IOV is neither new nor scary, basically just the next step past VT-d. Been around half a decade maybe, but usually only available on way crazy high end hardware like 10GbE where the virtualization hit hurts too much.

There should be nothing too risky about it assuming you can get it to work.

You know how pessimistic I am... getting from a random motherboard to a working VT-d is a bit rough, right? But completely doable. Getting to SR-IOV is another similar jump beyond VT-d.

 

[dlavigne]

1. I don't know if FreeBSD 9 has SR-IOV support(or if you even need it from the OS itself).

It looks like it is included in the e1000 driver: http://www.intel.com/support/network/adapter/pro100/sb/CS-031492.htm. While the FreeBSD link on that page points to the driver in HEAD (for ver. 11.0), it looks like there is a driver for 9.2: http://svnweb.freebsd.org/base/release/9.2.0/sys/dev/e1000/.

 

[Hash]

This driver already in freenas then right? Wonder if it has the 9.2.0 release version? Ill put it in and see if it fixes the problem when I get home but it sounds like it may not be a driver issue then unfortunately.

 

[dlavigne]

If the driver is not in FreeNAS (not all drivers are), you can create a feature request at bugs.freenas.org (post the issue number here if you do that). Otherwise, if the driver is there, let us know if it solves your issue.

 

[Hash]

Well I hit a bump.

How do I compile this driver? I have a 9.2 FreeBSD box set up Can prob figure it out when I have some time to spare but if anyone can guide me in the right direction that would be most appreciated. Looks like the e1000 driver here http://svnweb.freebsd.org/base/release/9.2.0/sys/dev/e1000/ has quite a few drivers I just need e1000_vf and if_igb.

 

[dlavigne]

Is the driver present on your FreeNAS system? If not, create a feature request at bugs.freenas.org and post the issue number here.

 

[jgreco]

I must be confused, because SR-IOV isn't a device. What driver are we talking about here, Dru?

 

[dlavigne]

Correct, but it is supposed to be in the e1000 driver. If that driver isn't on the system, a feature request should be made.

 

[cyberjock]

e1000 is definitely on FreeNAS.

 

[Hash]

Hmm where is the e1000 driver located? Or is it built into the kernel? I think the e1000 driver is suppost to have the igb_vf driver in it.

http://downloadmirror.intel.com/20927/eng/e1000.htm

The link above talks about the driver it states that in order to activate the igb_vf driver that would be the virtual function driver it requires the igb driver. Im not sure if the current e1000 driver has all this so was going to compile it and test to see if it works before requesting the feature. For all I know that may not be the issue.

 

[jgreco]

Okay, there's a bunch of weird shit going on in this thread and I'm really not anxious to sort it out. But I will see if I can do a rough course correction.

1) dlavigne seems unaware that FreeNAS includes the Intel e1000 driver, which is basically the Intel-supplied driver powering nearly every Intel based gigabit card out there (if_em, if_igb, etc).

2) That would not be the correct driver for an X540 based 10G card, so why are we interested in it again? My guess is that dlavigne may not know what SR-IOV is and arrived at the e1000 via Google.

3) The correct driver probed fine according to the first post in this thread. It included that it had probed as a Virtual Function Network Driver but then bombed with an MSIX problem of some sort and failed to attach. This will not and cannot be corrected by trying to load a driver for a substantially different card.

This discussion of the e1000 driver is far off in the weeds and I do not understand it. Suggest you focus your efforts on trying to figure out the MSIX configuration issue. Fix that and you'll probably find happiness.

 

[Hash]

Dont think the x540-t1 driver is the correct driver to load for sr-iov guest. The correct driver is the one I mentioned, which is part on the e1000 driver. Looking up updated versions to compile of the driver was my goal however it seems to be alot more complex being the config file not the driver like you suggested.

An update on the bug ticket. Guessing the msi-x config isn't something so easy to configure with the reply. I still have no idea where it is located. xD

 

[jgreco]

If you have an X540-T1, what makes you think that some random other driver would be the correct driver? SR-IOV is merely making a virtual instance of the physical hardware appear to the virtual machine. You still need the correct driver. Random other drivers will not function with your hardware; they were written for other hardware.