ESXi 5.5 + DirectPath = EHCI interrupt storm detected

[Cesar]

Hi ,

I'm new with FreeNAS (9.2) and I've setup it into vmware ESXi 5.5.
My objective with it is able share a external USB 3.0 Harddisk at our environment.

I've successfully installed , configured and shared this external HD using DirectPath/Passthrough at USB port.
All is working fine , except for this excessive messages at console and dmesg.
This IRQ are from ehci0 and ehci1 (which is two USB ports delivered at DirectPath).
If I remove this USBs from this VM configuration, the storm messages stop.

I already read other (old) topics about this issue, but all relationed with AHCI not EHCI.
Anyway , I try some workarounds, but no one works .

I missing something here ?
There is someway to stop this irq storm ?
I don't know if this is a FreeBSD issue and I need try solve at FreeBSD forums, not here.

Code:

 ~# dmesg | tail
interrupt storm detected on "irq16:"; throttling interrupt source
interrupt storm detected on "irq19:"; throttling interrupt source
interrupt storm detected on "irq16:"; throttling interrupt source
interrupt storm detected on "irq19:"; throttling interrupt source
interrupt storm detected on "irq16:"; throttling interrupt source
interrupt storm detected on "irq19:"; throttling interrupt source
interrupt storm detected on "irq16:"; throttling interrupt source
interrupt storm detected on "irq19:"; throttling interrupt source
interrupt storm detected on "irq16:"; throttling interrupt source
interrupt storm detected on "irq19:"; throttling interrupt source

----------------
~# uname -v
FreeBSD 9.2-RELEASE-p12 #0 r262572+b043649: Sun Sep 28 23:03:31 PDT 2014  root@build3.ixsystems.com:/tank/home/jkh/build/921/freenas/os-base/amd64/fusion/jkh/921/freenas/FreeBSD/src/sys/FREENAS.amd64


-------------
Motherboard : Gigabyte GA-Z77M-D3H
CPU : Model: 6.42.7 "Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz"

-----------
# uptime
 3:49PM  up 1 day,  4:14, 1 user, load averages: 1.18, 1.24, 1.24
# vmstat -i
interrupt  total  rate
irq1: atkbd0  6  0
irq0: attimer0  1  0
irq15: ata1  67841  0
irq16: ehci1  7386699691  72695
irq17: mpt0  9789  0
irq18: em0 alc0  53730851  528
irq19: ehci0  6964291019  68538
cpu0:timer  11400568  112
cpu1:timer  13974454  137
cpu3:timer  3037682  29
cpu2:timer  14100201  138
Total  14447312103  142181

------------
PCI conf

# pciconf -lv
hostb0@pci0:0:0:0:  class=0x060000 card=0x197615ad chip=0x71908086 rev=0x01 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '440BX/ZX/DX - 82443BX/ZX/DX Host bridge'
  class  = bridge
  subclass  = HOST-PCI
pcib1@pci0:0:1:0:  class=0x060400 card=0x00000000 chip=0x71918086 rev=0x01 hdr=0x01
  vendor  = 'Intel Corporation'
  device  = '440BX/ZX/DX - 82443BX/ZX/DX AGP bridge'
  class  = bridge
  subclass  = PCI-PCI
isab0@pci0:0:7:0:  class=0x060100 card=0x197615ad chip=0x71108086 rev=0x08 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '82371AB/EB/MB PIIX4 ISA'
  class  = bridge
  subclass  = PCI-ISA
atapci0@pci0:0:7:1:  class=0x01018a card=0x197615ad chip=0x71118086 rev=0x01 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '82371AB/EB/MB PIIX4 IDE'
  class  = mass storage
  subclass  = ATA
none0@pci0:0:7:3:  class=0x068000 card=0x197615ad chip=0x71138086 rev=0x08 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '82371AB/EB/MB PIIX4 ACPI'
  class  = bridge
none1@pci0:0:7:7:  class=0x088000 card=0x074015ad chip=0x074015ad rev=0x10 hdr=0x00
  vendor  = 'VMware'
  device  = 'Virtual Machine Communication Interface'
  class  = base peripheral
vgapci0@pci0:0:15:0:  class=0x030000 card=0x040515ad chip=0x040515ad rev=0x00 hdr=0x00
  vendor  = 'VMware'
  device  = 'SVGA II Adapter'
  class  = display
  subclass  = VGA
mpt0@pci0:0:16:0:  class=0x010000 card=0x197615ad chip=0x00301000 rev=0x01 hdr=0x00
  vendor  = 'LSI Logic / Symbios Logic'
  device  = '53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI'
  class  = mass storage
  subclass  = SCSI
pcib2@pci0:0:17:0:  class=0x060401 card=0x079015ad chip=0x079015ad rev=0x02 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI bridge'
  class  = bridge
  subclass  = PCI-PCI
pcib3@pci0:0:21:0:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib4@pci0:0:21:1:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib5@pci0:0:21:2:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib6@pci0:0:21:3:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib7@pci0:0:21:4:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib8@pci0:0:21:5:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib9@pci0:0:21:6:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib10@pci0:0:21:7:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib11@pci0:0:22:0:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib12@pci0:0:22:1:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib13@pci0:0:22:2:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib14@pci0:0:22:3:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib15@pci0:0:22:4:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib16@pci0:0:22:5:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib17@pci0:0:22:6:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib18@pci0:0:22:7:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib19@pci0:0:23:0:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib20@pci0:0:23:1:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib21@pci0:0:23:2:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib22@pci0:0:23:3:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib23@pci0:0:23:4:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib24@pci0:0:23:5:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib25@pci0:0:23:6:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib26@pci0:0:23:7:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib27@pci0:0:24:0:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib28@pci0:0:24:1:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib29@pci0:0:24:2:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib30@pci0:0:24:3:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib31@pci0:0:24:4:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib32@pci0:0:24:5:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib33@pci0:0:24:6:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
pcib34@pci0:0:24:7:  class=0x060400 card=0x07a015ad chip=0x07a015ad rev=0x01 hdr=0x01
  vendor  = 'VMware'
  device  = 'PCI Express Root Port'
  class  = bridge
  subclass  = PCI-PCI
em0@pci0:2:0:0: class=0x020000 card=0x075015ad chip=0x100f8086 rev=0x01 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = '82545EM Gigabit Ethernet Controller (Copper)'
  class  = network
  subclass  = ethernet
alc0@pci0:3:0:0:  class=0x020000 card=0xe0001458 chip=0x10831969 rev=0xc0 hdr=0x00
  vendor  = 'Atheros Communications'
  device  = 'AR8151 v2.0 Gigabit Ethernet'
  class  = network
  subclass  = ethernet
ehci0@pci0:11:0:0:  class=0x0c0320 card=0x50061458 chip=0x1e2d8086 rev=0x04 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = 'Panther Point USB Enhanced Host Controller'
  class  = serial bus
  subclass  = USB
ehci1@pci0:19:0:0:  class=0x0c0320 card=0x50061458 chip=0x1e268086 rev=0x04 hdr=0x00
  vendor  = 'Intel Corporation'
  device  = 'Panther Point USB Enhanced Host Controller'
  class  = serial bus
  subclass  = USB

 

[rs225]

Do you have Update 1 installed for ESXi 5.5?

Beyond that, it will be unlikely to find a solution here, since FreeNAS isn't supported in a VM, and it doesn't look like many other people solve this problem even on real hardware..

 

[Cesar]

Yes , even better, was installed the last patch which is newer than update 2 . (ESXi550-201410001.zip) .

That's sad ... I really like this FreeNAS , should be nice see versions runnig over other linux distro , since I believe they have better support of FreeBSD for lot of hardwares and VM...

 

[Ericloewe]

Yes , even better, was installed the last patch which is newer than update 2 . (ESXi550-201410001.zip) .

That's sad ... I really like this FreeNAS , should be nice see versions runnig over other linux distro , since I believe they have better support of FreeBSD for lot of hardwares and VM...

You seem to have misunderstood what FreeNAS is. It has nothing to do with Linux. It's an OS based on FreeBSD. "Moving" to Linux is not a trivial affair.

 

[cyberjock]

IRQ storms that involve hypervisors are almost always caused by a very complex set of circumstances. The hypervisor, the driver in the guest and the virtualization layer have a malformed relationship that leads to IRQ storms. For most people an IRQ storm is a message that "you are f*cked" with regards to a given configuration. I have had one or two and it wasn't entirely unexpected (definitely not desired though and the hardware definitely wouldn't function properly).

Unfortunately unless you are a debugger you aren't likely to find the problem and be able to fix it. So you will probably find that the only long-term answer is to give up on that given configuration. :(

 

[rs225]

I hadn't realized ESXi5.5 update 2 was already out.

You can give it a try on the FreeBSD forums, but they might also be a little difficult since it is FreeNAS rather than plain FreeBSD. It's worth a try.

 

[Cesar]

Just sharing with the community...

Today I've solved this problem.
After upgrade my ESXi to v6 , no more IRQ storm...