Intel X520 - ixgbe hangs

[dvc9]

Hey, Thanks for the new and great release Freenas 9.1,
the performance is great without tuning an nothing. ill get like 400 MB r/w on my 10GbE.

But then i think i found a bug.

When 2 10GbE computers are accessing the server at the same time, over NFS (no string options)
then over time, accessing tiff sequences, the nic suddenly hangs. and no traffic gets through.

All is default, no tuning on both sides.

Any ideas ?

 

[cyberjock]

Is it only tiff sequences? If yes that doesn't sound like a FreeNAS problem.

Have you tried a different protocol? If the same thing happens on CIFS at least that rules out an issue with NFS specifically.

 

[dvc9]

This is not only tiff sequences, it looks like that´s the thing that initiates the hang.
We are working with video production, When we on one machine is working with tiff sequences, and the second machine is reading video files for rendering.

The disks is preforming at 600 mb r/w, and one machine is preforming 400 mb r/w
CIFS performance is really poor, compared to NFS, (like 80 - 120 mb r/w)

 

[dvc9]

I do think it´s something to do with a old ixgbe bugg with "flow controll" enabled.
witch will make the TX hangs, but i duno =)

 

[cyberjock]

Here's a link for some info on possible disabling flow control...

http://lists.freebsd.org/pipermail/freebsd-net/2012-July/032868.html

 

[dvc9]

Thanks,
ill test =)

 

[dvc9]

Looks like it´s disabled.
dev.ix.0.fc: 3
dev.ix.1.fc: 3

hmmm... you know how to disable TSO ?

 

[cyberjock]

sysctl net.inet.tcp.tso=0

1 will enable it.

Keep in mind that this will affect ALL network adapters on your system.

 

[dvc9]

Hmm, ok. Ill google a bit.

Thanks !

 

[dvc9]

After alot of testing,
it seems to be the ixgbe driver.

The same error happens on different machines, on different protocols.


is there a way to upgrade the ixgbe driver on freenas ?

 

[cyberjock]

What version of the driver do you need and what version does FreeNAS have? Also, are you using x86 or x64 version of FreeNAS?

It just so happens that I put a ticket in for the driver to be updated 2 weeks ago... https://support.freenas.org/ticket/2451

That ticket says to add the driver. I just read that the driver is already included with FreeNAS from another ticket, but may not be the latest based on your information.

 

[dvc9]

This issue started when we upgraded to Freenas 9.1 (newest) , we are using x64,
40 GB Ram, and 11 x Disks in raidZ1 + 1 Spare. (know abt RaidZ2 =) )

 

[morph027]

Same problem here...unluckily i've already upgraded zfs pool version, so i can't downgrade to 8.3 .... :(

I've tried to play with some parameters from intel driver (LRO, TSO, RXCSUM,....) and mtu settings, the problem still occurs after a while. I'll try to build the intel driver and see, how things are going.

 

[cyberjock]

Sorry, I'd try to add the driver but you have to have the card to test your test build before you request it to be committed to FreeNAS. :(

 

[morph027]

ah...just found: https://bugs.freenas.org/issues/3201

 

[cyberjock]

And I created ticket 2451(which you'd have seen if you had scrolled up)... haha. Damn duplicate tickets. Apparently people can't search for duplicate complaints before creating new ones... /sigh

 

[morph027]

I saw ... that's why we're using https://github.com/abahgat/redmine_didyoumean here ;)

At the moment, i switched back to normal 1Gb connection...

Also, the zfs upgrade issue mentioned before is nonexistent. 8.3 also uses zfs version 28...downgrading should be possible.

 

[Koplin]

Looks like I am hitting this same wall. Only my box crashed and restarted.
My hardware is a new system running x64 9.1.1 with Intel nic's. I did a lot of research but apparently not enough before committing to my current hardware. The 10 gig interfaces get slow then die. Like the previous individual I have reverted to 1 gig links pending resolution.

If I understand the above bug reports they are looking at 9.2 as a release for the support for this. I will keep my fingers crossed :)

 

[Durandal]

Do you think this goes for all Intel-based 10 GbE cards? I'll soon have my new FreeNAS server with a SuperMicro AOC-STG-I2 that is based of the Intel 82598EB that is much older than Intel 82599ES that the X520 is based on. Is this only a ixgbe driver problem, or just the driver with specific network controllers?

 

[Koplin]

My system is a Supermicro sandy bridge box with a 4 port intel based model onboard (reports as I350 Gigabit Network Connection)
and a single PCI card that is a 2 port Intel 10gig

the report from my system is
ix0@pci0:131:0:0: class=0x020000 card=0x00018086 chip=0x15288086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller 10 Gigabit X540-AT2'
class = network
subclass = ethernet

This is the unit that crashed with the default 9.1 config/drivers I have made no further attempts to fix this expecting that 9.2 will fix it.

If I knew how to compile the driver as a module I am sure that would fix it but I couldn't find directions that worked on how to compile FreeNAS 9.1 and gave up after several failed attempts. I don't know about other cards, my apologies however I would be very interested if you find a working 10 gig card under the current release of FreeNAS as I have the budget to acquire a replacement.

I share my ZFS volumes up via NFS to my vmware cluster. It took a few minutes of saturating the link by copying vm's around and by doing iometer benchmarking for it to crash. Thus my only advice is that IF you do buy that card you can tell fairly quickly if its going to work before relaying on it.

Best of Luck