Networking LAN deadlocks

[Zarovzky]

When I built the server it was around version 10.4 or so.
The stability was great, barely a restart a month.
I have the latest update of the system as per today: FreeNAS-11.1-U4
and I have to restart it daily because networking fails multiple times.

Every day since I've updated to the version 11.1 the network fails, mainly no ping, so nothing works.
What *solves* it is to reboot directly from the button.

I do have a good system : 64gb ECC DDR4, Ryzen 1600 overclocked to 3800ghz, Asrock X370 taichi mobo.
The system is used for NextCloud mainly but also for another applications I have which hammer MySQL badly.
My Apps make CPU rise at up to 20% which is mostly taken by MySQL.

Networking graph, the spikes are moments before lan stops working.
These spikes are not caused by any app (see attachments).

I've seen this problem lurking around in 11+ versions but no solution so far.

PD: ill post more information in the upcoming days.

 

[kdragon75]

Thanks for posting and welcome to the forum. I have a similar issue but thought it was a flakey NIC. I look forward to seeing what you find.

In my case the interface just stops responding. I can down and up the interface but it has no effect. I am able to reboot from the console but by then my VMs have all crashed. Mine also seems to be under load before it goes down.

 

[Zarovzky]

Thanks for posting and welcome to the forum. I have a similar issue but thought it was a flakey NIC. I look forward to seeing what you find.

In my case the interface just stops responding. I can down and up the interface but it has no effect. I am able to reboot from the console but by then my VMs have all crashed. Mine also seems to be under load before it goes down.

Did you replace the NIC and had same results? Same spike with the new NIC?

 

[kdragon75]

I ended up swapping the drive to a different server and saw the same issue. I also see a lot of iSCSI timeouts but i'm not sure if that's related. Old server was em and new is bce. It happen less on the "new" server but still happen when loaded for more than a a few minutes.

 

[Zarovzky]

I ended up swapping the drive to a different server and saw the same issue. I also see a lot of iSCSI timeouts but i'm not sure if that's related. Old server was em and new is bce. It happen less on the "new" server but still happen when loaded for more than a a few minutes.

I fail to understand why are networking deadlocks related to the drives... My main system is not on a failing drive, the jails are located on an SSD which is barely new. Neither the apps interact with the older drives.

Could you tell me which log file did you check for the iSCSI timeouts? Will check and add mine too.

 

[kdragon75]

If you enable "Show console messages in the footer" under System -> Advanced -> (about half way down) you will see the messages as they are logged to /var/log/messages.

I fail to understand why are networking deadlocks related to the drives...

Its not the drives. I think the iSCSI (a network service) timeouts are a symptom of the underlying issue.