"CAM Status: Command Timeout" as well as "Interrupt Storm Detected" Errors

[Rpgwaiter]

Hopefully I'm posting this in the correct place.

I recently bought a used server off of eBay. Not sure if the hardware is relevant, so I'll just list it:

Motherboard: Supermicro H8DME-2
Backplane: SAS846TQ rev. 3.1
CPU: 2x AMD Opteron 2431
RAM: 64GB DDR2 Registered-ECC@667MHz
Hard Drive Controllers: 3x Supermicro SAT2-MV8 (BIOS 1.0)
Chassis: 24-bay Hotswap Supermicro (not sure on model number)
Not sure on the PSU's, although there are 2 of them

Anyways, I have 8x3TB WD Red drives running on a single RaidZ1 pool (I hope I'm using those terms correctly). Any time I transfer files to the server, I will *always* get CAM Status: Command Timeout errors on the monitor attached to the server, and every time this happens, the transfer speed will slow to a crawl (Less than 2MBytes/s on a Gigabit cable/NIC). I will also get "Interrupt Storm Detected" Errors occasionally, which seem to have the same effect.

I'm not sure how to get a copy of what is showing up on the monitor, and I'm sure it's needed to properly diagnose this issue. I have tried moving the drives around the chassis so they are all attached to a single controller card, as I was reading online that this may be a failing card, but I get these errors on all three of them. I have also tried a combination of everything on this post: (http://zfsguru.com/forum/zfsgurusupport/508), but still no luck.

I have updated my Motherboard BIOS to the latest version and it also did not fix the problem. I'm hoping this is a problem I'm able to fix in the console or by changing some BIOS settings, and not a hardware failure. Any ideas?

 

[BigDave]

FreeNAS is PURE software RAID, which means that ANYTHING in between FreeNAS and the disks (EVEN IF
IT'S JBOD) is going to cause issues like you are describing.

Hard Drive Controllers: 3x Supermicro SAT2-MV8 (BIOS 1.0)

My guess is the PCI based card with a Marvell chipset, is getting in the way of letting FreeNAS have direct access
and complete control of the hard disks.
Please read the recommended hardware stickie found in "Useful Links" in my signature below.

 

[Rpgwaiter]

I looked at the hardware recommendation post beforehand, and am considering getting new controller cards. Is there anything I can do with my current cards, either in the BIOS, flashing the cards themselves, etc that can improve my server in the mean time while I save up for the new cards?

 

[BigDave]

To be honest, I wouldn't know a thing about those old cards.
You can look at the Compatible Hardware List of FreeBSD for version 9.3 and
if you can't find those cards as being supported, I wouldn't bother... Sorry:(

 

[jgreco]

The Marvell cards are known to work poorly.

 

[cyberjock]

More than likely the interrupt storm is because of your controllers. IRQ storms *will* cause performance issues, which is why FreeBSD warns you when excessive interrupts are occurring.

Generally interrupt storms mean something like "your hardware is failing" or "your drivers suck". In this case, if the hardware has worked until you installed FreeNAS the issue is probably that the drivers just aren't 100% on FreeBSD. Wouldn't be the least bit surprising to me if that was the case. Easy fix though. Buy some M1015s or equivalent (see the sticky for the recommended hardware list) and as soon as you replace them the interrupt storms will probably go away and your performance issues that are the result of the storms will also go away. :)

If you want to dig deeper, you can run some commands using "vmstat -i" to see the IRQs being used excessively and identify the hardware on that IRQ, and therefore the hardware (or drivers) you can blame for the storm. Look at the rate during a file transfer and not the total interrupts.