SOLVED [BUG] IRQ and network speed problem

[BERKUT]

Hello,
NAS spec:
FreeNAS-9.10.2-U1 (86c7ef5) (this problem is on any version of FreeNAS)

CPU: Intel(R) Xeon(R) CPU L5630 @ 2.13GHz x2
Memory: 36825MB
Netowrk: Intel® Gigabit VT Quad Port Server Adapter with MTU 9000 and using MPIO for transferring
2 ports connected -> switch -> Server
2 ports connected -> Server
Storage sharing - iSCSI for Hyper-V

Tunable:


I have some problem:

When IRQ is so big all network speed down from 1 gbit to 90-200 mbit.
Where is problem and how it fix?

Thank you.

 

[m0nkey_]

First of all, turn off autotune and remove all tunables associated with it. There are very few use cases to actually enable autotune, you sir do not have one of them.

 

[BERKUT]

First of all, turn off autotune and remove all tunables associated with it. There are very few use cases to actually enable autotune, you sir do not have one of them.

But other 8 NAS work fine with the same configuration.
P.S I'll try it, but i don't sure that problem is in tunable.

 

[BERKUT]

Hm... interesting, look like it helps...
I'll monitoring it few days.

 

[bigphil]

Yeah, I'm not sure the autotune system is that great in the current 9.10 release...especially if its turning off delayed_ack on a 9.10 system when FreeBSD 10.3 fully supports the feature. I agree with M0nkey_ ...turn it off and remove the settings.

 

[BERKUT]

Hello again.
The problem has been returned.
Also reboot of NAS helping only for one week, after IRQ going up and network speed down.
Any ideas?

 

[BERKUT]

So, anybody don't know? Please write, what information need for help.

 

[BERKUT]

I'm still waiting for any help.
Or FreeNas (9.3 and 9.10) has problem with kernel?

 

[nightshade00013]

I think we need some more information about the network setups. And this very well could be a networking issue and not just the FreeNAS. If you are using LACP do you also have 802.3ad compatible switches. Are you running multiple vlan's instead of LACP. You changed the MTU as well read https://forums.freenas.org/index.php?threads/jumbo-frames-notes.26064/ Unless all the systems are using the same drivers on the exact same hardware this could be a problem in and of itself.

I would almost say backup your config and install a fresh copy of FreeNAS on a new stick and import the config and make the changes to get EVERYTHING to stock or just install FreeNAS fresh on a new stick and setup the box again leaving all the network and tuning at defaults and see what happens.

 

[BERKUT]

I think we need some more information about the network setups. And this very well could be a networking issue and not just the FreeNAS. If you are using LACP do you also have 802.3ad compatible switches. Are you running multiple vlan's instead of LACP. You changed the MTU as well read https://forums.freenas.org/index.php?threads/jumbo-frames-notes.26064/ Unless all the systems are using the same drivers on the exact same hardware this could be a problem in and of itself.

I would almost say backup your config and install a fresh copy of FreeNAS on a new stick and import the config and make the changes to get EVERYTHING to stock or just install FreeNAS fresh on a new stick and setup the box again leaving all the network and tuning at defaults and see what happens.

We using only iSCSI with MPIO and MTU 9000 (on windows side configured MTU too).
This problem also has and on just installed FreeNas after 2-10 weeks .

 

[nightshade00013]

Having it on Windows and having it on FreeNAS are two different things. Unless the hardware and drivers are 100% identical, they may not operate correctly when changing MTU and even then it is not guaranteed. You ask for a solution and to get there you have to post information and troubleshoot.

If you can not post the information and take some steps to work with the users here you can head over to https://www.ixsystems.com/freenas-commercial-support/ and someone will contact you with a quote to handle it if support is available in your area.

 

[BERKUT]

Having it on Windows and having it on FreeNAS are two different things. Unless the hardware and drivers are 100% identical, they may not operate correctly when changing MTU and even then it is not guaranteed. You ask for a solution and to get there you have to post information and troubleshoot.

If you can not post the information and take some steps to work with the users here you can head over to https://www.ixsystems.com/freenas-commercial-support/ and someone will contact you with a quote to handle it if support is available in your area.

What kind of information do you need? I'll upload all debug information here.
On windows (2012 R2) side we using Intel(R) Gigabit VT Quad Port Server Adapter

 

[BERKUT]

Having it on Windows and having it on FreeNAS are two different things. Unless the hardware and drivers are 100% identical, they may not operate correctly when changing MTU and even then it is not guaranteed. You ask for a solution and to get there you have to post information and troubleshoot.

If you can not post the information and take some steps to work with the users here you can head over to https://www.ixsystems.com/freenas-commercial-support/ and someone will contact you with a quote to handle it if support is available in your area.

So you recommended disable MTU?

 

[c32767a]

So you recommended disable MTU?

The diag log includes messages like this:

Code:

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb1: link state changed to DOWN

igb0: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb1: link state changed to DOWN

igb0: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

I would talk to the person responsible for your network switches and confirm that the cabling and switches are properly connected and error-free. It is not normal for there to be repetitive interface up/down messages in the logs.

 

[BERKUT]

The diag log includes messages like this:

Code:

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb1: link state changed to DOWN

igb0: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb1: link state changed to DOWN

igb0: link state changed to DOWN

igb1: link state changed to UP

igb0: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

igb0: link state changed to DOWN

igb1: link state changed to DOWN

igb0: link state changed to UP

igb1: link state changed to UP

I would talk to the person responsible for your network switches and confirm that the cabling and switches are properly connected and error-free. It is not normal for there to be repetitive interface up/down messages in the logs.

The cables (igb0 and igb1) have been manually reconnected to another ports, probably 1-2 weeks ago. All cables are new and well connected.

 

[BERKUT]

Every 1-2 weeks the same problem.

Any recommendation, change FreeNAS to something another? If reboot the network interface it helps, but it'll happen again.

 

[BERKUT]

We looks on other NAS too, and have some idea.
It looks like problem in FreeNAS after build 201606270534, have the IRQ problem.
We have 4 NAS at the builds 201606270534 or early (9.10), and they all work perfect, but all other NAS since 9.10.1 to last build have problem with IRQ.

We have made rollback one NAS to this old build and we will check it.

 

[BERKUT]

Well...
So now after two weeks test, NAS work perfect on stable version 201606270534
We also have some statistics.
NAS with stable version 201606270534

These three NAS as you can see working ~200 days without any problems

Now 9.10.1 to 9.10.2

Each 2-4 weeks they have problem with IRQ.

We have been rollbacked 50% of NAS where was problem from 9.10.* to stable version 201606270534, and after 2 weeks they all still working perfect. Within 30 days we'll rollback all NAS.

All NAS have the same loads per week ~2-3 TB

 

[BERKUT]

Last version FreeNAS-11.0-U2 (e417d8aa5)
Problem still here...

The same problem exists in the version 201606270534, but not so often

 

[BERKUT]

if somebody have the same problem, you can find solution here - https://bugs.freenas.org/issues/25516