Getting lots if "Interrupt storm on irq16"

[zyberzero]

Hi,

Got an ESXi host with FreeNAS in a VM, and a LSI 3081E-R SAS card - with LSI 1068e chipset flashed in IT mode. The VM has 5 GB RAM (more to come, I just need buy some more), and host has a Xeon E3-1240 CPU.

When I boot FreeNAS, on the console I get the following message, over and over again:
"Interrupt storm detected on irq16 throttling interrrupt source"
vmstat -i tells me that mpt1 is irq16, and that´s my controller card.
Why is it doing this? Is this probably why I cant saturate a 1 Gbit connecten with 8 x 2TB disks in RAID-Z2 (stuck at approx 150 Mbit/s, 0.2% CPU-usage).

Any hints?

Thanks,
Zyber

 

[zyberzero]

Hi,

I've tested back and forth. When I set the VM to only have one CPU (instead of four or two) all the messages disappear. This is really odd, why does it do like that?

Thanks,
Zyber

 

[Jcrash]

I just registered to say thanks! Your thread helped me eliminate an "interrupt storm" error in my VM.

I'm running FreeNAS as a guest (VirtualBox) in a Windows 7 host (for Diablo 3 :P).

In my BIOS I wanted to keep my sata controller setting for RAID instead of AHCI.

Motherboard: P8Z68 Deluxe
2x Kingston SSDNow 64GB RAID0 in the Intel Sata3 ports
3x WD Caviar Green 2TB (Intel Sata2)


For those in a similar situation, I've come across two solutions to my "interrupt storm" error:

1. Check your BIOS, set your sata controller setting to AHCI if you don't need raid

2. If you want to keep it in raid: set your VM to only one processor

Thanks again

 

[relyt]

I came across this problem as well when I built my ESXi host a couple months ago. I solved it by booting into the BIOS of the FreeNAS vm and disabling all unnecessary devices (ie. serial, COM, etc.) I have no issue assigning 2 CPUs to my FreeNAS vm.

relyt

 

[zyberzero]

Hi Relyt, and thanks!

I've tried that as well, but without any look. I'm going to look into that once again, though.

Thanks!
Zyber

 

[mrsieb]

Interrupt storm on irq 16 too

Hi there,

I also have a lot of this Messages in my dmesg:

Code:

interrupt storm detected on "irq16:"; throttling interrupt source

What i find out in the mean time,
according to the boot message IRQ 16 is used by
PCI Bridge, internal VGA , USB-Hub, PCI-Slot where my SataController is pluged in

Code:

grep 'irq 16' /var/run/dmesg.boot
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xfe000000-0xfe3fffff,0xc0000000-0xcfffffff irq 16 at device 2.0 on pci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe503000-0xfe5033ff irq 16 at device 26.0 on pci0
siis0: <SiI3132 SATA controller> port 0xd000-0xd07f mem 0xfe484000-0xfe48407f,0xfe480000-0xfe483fff irq 16 at device 0.0 on pci4

Vmstat also show me a high rate on IRQ16

Code:

vmstat -i
interrupt                          total       rate
irq0:                                  1          0
stray irq0                             1          0
irq16: ehci0 siis0            6749764209      16526
irq23: ehci1                      961992          2
cpu0: timer                    816828995       1999
irq256: re0                    287873004        704
irq257: ahci0                    8005862         19
cpu3: timer                    816808959       1999
cpu1: timer                    816808959       1999
cpu2: timer                    816808959       1999
Total                        10313860941      25252

What i am asking could this also be triggered by a defected Harddisk?
My smart values are not the best but still nothing to worry about it..
But i find out normaly if i look at systat i se ALL my Harddisk.
Also after a fresh reboot when i start to copy something after a while the IRQ Storm again.

Below allready one disk is missing in this view... -> ada11
some times i only see 5 or less disks here

Code:

/0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   |

          /0%  /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
cpu  user|
     nice|
   system|
interrupt|
     idle|************************************************

          /0%  /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
da0   MB/s
      tps|
ada0  MB/s
      tps|
ada1  MB/s
      tps|
ada2  MB/s
      tps|
ada3  MB/s
      tps|
ada4  MB/s
      tps|
ada5  MB/s
      tps|
ada6  MB/s
      tps|
ada7  MB/s
      tps|
ada8  MB/s
      tps|
ada9  MB/s
      tps|
ada10 MB/s
      tps|

But Zfs Pool and Devices looks pretty good.

Code:

 pool: BLOCK-B
 state: ONLINE
 scrub: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        BLOCK-B                                         ONLINE       0     0     0
          raidz1                                        ONLINE       0     0     0
            gptid/e1f86b2c-f735-11e0-956b-80ee7316ce86  ONLINE       0     0     0
            ada5p2                                      ONLINE       0     0     0
            gptid/2a8b2044-40a7-11e1-be1b-80ee7316ce86  ONLINE       0     0     0
            gptid/4e5a8ea1-4167-11e1-afd3-80ee7316ce86  ONLINE       0     0     0
          raidz1                                        ONLINE       0     0     0
            ada0p2                                      ONLINE       0     0     0
            ada1p2                                      ONLINE       0     0     0
            ada2p2                                      ONLINE       0     0     0
            ada3p2                                      ONLINE       0     0     0
          raidz1                                        ONLINE       0     0     0
            gptid/d14615d6-442e-11e1-94ae-80ee7316ce86  ONLINE       0     0     0
            gptid/d1fde5bc-442e-11e1-94ae-80ee7316ce86  ONLINE       0     0     0
            gptid/d2519458-442e-11e1-94ae-80ee7316ce86  ONLINE       0     0     0
            gptid/d297be2b-442e-11e1-94ae-80ee7316ce86  ONLINE       0     0     0

errors: No known data errors

Code:

camcontrol devlist
<SAMSUNG HD203WI 1AN10002>         at scbus0 target 0 lun 0 (ada0,pass0)
<SAMSUNG HD203WI 1AN10002>         at scbus0 target 1 lun 0 (ada1,pass1)
<SAMSUNG HD203WI 1AN10002>         at scbus0 target 2 lun 0 (ada2,pass2)
<ST2000DL003-9VT166 CC32>          at scbus0 target 3 lun 0 (ada3,pass3)
<Port Multiplier 37261095 1706>    at scbus0 target 15 lun 0 (pass4,pmp0)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus1 target 0 lun 0 (ada4,pass5)
<ST2000DL003-9VT166 CC3C>          at scbus1 target 1 lun 0 (ada5,pass6)
<SAMSUNG HD203WI 1AN10002>         at scbus1 target 2 lun 0 (ada6,pass7)
<WDC WD20EARX-00PASB0 51.0AB51>    at scbus1 target 3 lun 0 (ada7,pass8)
<Port Multiplier 37261095 1706>    at scbus1 target 15 lun 0 (pass9,pmp1)
<SAMSUNG HD204UI 1AQ10001>         at scbus2 target 0 lun 0 (ada8,pass10)
<SAMSUNG HD204UI 1AQ10001>         at scbus3 target 0 lun 0 (ada9,pass11)
<SAMSUNG HD204UI 1AQ10001>         at scbus4 target 0 lun 0 (ada10,pass12)
<SAMSUNG HD204UI 1AQ10001>         at scbus5 target 0 lun 0 (ada11,pass13)
<Kingston DataTraveler G3 1.00>    at scbus8 target 0 lun 0 (da0,pass14)

I also tried to slow down the SATA Speed with loader hint.siisch.0.sata_rev=1, no change.
AHCI in Bios is on; IRQ 16 seams to be hardwired...not options for manual assignment

 

[mav@]

I can't say for sure about PCIe, but with usual PCI it was possible to change IRQ by moving card into different PCI slot.

 

[mrsieb]

I can't say for sure about PCIe, but with usual PCI it was possible to change IRQ by moving card into different PCI slot.

Yes normaly also at PCIe unfortunately in my case its a cheap Barboneboard only One 1xPCIe and one 16xPCIe-PEG the ban thing.
Normaly it should not be a problem to plug a 1x Card into a 16x Slot but this board can not deal with it only GPU cards working.

All leads to change the whole board :-(

this type i am using at the moment -> http://www.shuttle.eu/products/mini-pc/sh67h3/specification/