Network Crashes but only at Night

martin481997 · Apr 23, 2020

Hi, i am in big trouble. Everytime i use my server heavily at night around 0:00 it crashes and isnt reachable ever again until i hit the Power Button to let it drive down and up again. I really dont know what is going on there because its only at night. If I dont use it heavy like copying Terrabytes of files over SMB for Backup, just with a movie over my Plex jail it stucks for 5-10 Minutes somewhere between 0:00 and 0:30 and then works normally again. The log is repeating itself until I restart the server over the Power Button. Could this be a bug of Freenas that needs to be Fixed?

Code:

Apr 24 00:00:00 freenas syslog-ng[2143]: Configuration reload request received,>
Apr 24 00:00:00 freenas syslog-ng[2143]: Configuration reload finished;
Apr 24 00:24:38 freenas em0: Watchdog timeout Queue[0]-- resetting
Apr 24 00:24:38 freenas Interface is RUNNING and ACTIVE
Apr 24 00:24:38 freenas em0: TX Queue 0 ------
Apr 24 00:24:38 freenas em0: hw tdh = 528, hw tdt = 485
Apr 24 00:24:38 freenas em0: Tx Queue Status = -2147483648
Apr 24 00:24:38 freenas em0: TX descriptors avail = 40
Apr 24 00:24:38 freenas em0: Tx Descriptors avail failure = 1117497786
Apr 24 00:24:38 freenas em0: RX Queue 0 ------
Apr 24 00:24:38 freenas em0: hw rdh = 276, hw rdt = 275
Apr 24 00:24:38 freenas em0: RX discarded packets = 0
Apr 24 00:24:38 freenas em0: RX Next to Check = 276
Apr 24 00:24:38 freenas em0: RX Next to Refresh = 275
Apr 24 00:24:38 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:38 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:42 freenas kernel: em0: link state changed to UP
Apr 24 00:24:42 freenas kernel: em0: link state changed to UP
Apr 24 00:24:44 freenas em0: Watchdog timeout Queue[0]-- resetting
Apr 24 00:24:44 freenas Interface is RUNNING and ACTIVE
Apr 24 00:24:44 freenas em0: TX Queue 0 ------
Apr 24 00:24:44 freenas em0: hw tdh = 0, hw tdt = 984
Apr 24 00:24:44 freenas em0: Tx Queue Status = -2147483648
Apr 24 00:24:44 freenas em0: TX descriptors avail = 40
Apr 24 00:24:44 freenas em0: Tx Descriptors avail failure = 1117497799
Apr 24 00:24:44 freenas em0: RX Queue 0 ------
Apr 24 00:24:44 freenas em0: hw rdh = 0, hw rdt = 1023
Apr 24 00:24:44 freenas em0: RX discarded packets = 0
Apr 24 00:24:44 freenas em0: RX Next to Check = 0
Apr 24 00:24:44 freenas em0: RX Next to Refresh = 1023
Apr 24 00:24:44 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:44 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:47 freenas kernel: em0: link state changed to UP
Apr 24 00:24:47 freenas kernel: em0: link state changed to UP
Apr 24 00:24:49 freenas em0: Watchdog timeout Queue[0]-- resetting
Apr 24 00:24:49 freenas Interface is RUNNING and ACTIVE
Apr 24 00:24:49 freenas em0: TX Queue 0 ------
Apr 24 00:24:49 freenas em0: hw tdh = 0, hw tdt = 984
Apr 24 00:24:49 freenas em0: Tx Queue Status = -2147483648
Apr 24 00:24:49 freenas em0: TX descriptors avail = 40
Apr 24 00:24:49 freenas em0: Tx Descriptors avail failure = 1117497809
Apr 24 00:24:49 freenas em0: RX Queue 0 ------
Apr 24 00:24:49 freenas em0: hw rdh = 0, hw rdt = 1023
Apr 24 00:24:49 freenas em0: RX discarded packets = 0
Apr 24 00:24:49 freenas em0: RX Next to Check = 0
Apr 24 00:24:49 freenas em0: RX Next to Refresh = 1023
Apr 24 00:24:49 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:49 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:53 freenas kernel: em0: link state changed to UP
Apr 24 00:24:53 freenas kernel: em0: link state changed to UP
Apr 24 00:25:04 freenas em0: Watchdog timeout Queue[0]-- resetting
Apr 24 00:25:04 freenas Interface is RUNNING and ACTIVE
Apr 24 00:25:04 freenas em0: TX Queue 0 ------
Apr 24 00:25:04 freenas em0: hw tdh = 0, hw tdt = 149
Apr 24 00:25:04 freenas em0: Tx Queue Status = -2147483648
Apr 24 00:25:04 freenas em0: TX descriptors avail = 875
Apr 24 00:25:04 freenas em0: Tx Descriptors avail failure = 1117497809
Apr 24 00:25:04 freenas em0: RX Queue 0 ------
Apr 24 00:25:04 freenas em0: hw rdh = 0, hw rdt = 1023
Apr 24 00:25:04 freenas em0: RX discarded packets = 0
Apr 24 00:25:04 freenas em0: RX discarded packets = 0
Apr 24 00:25:04 freenas em0: RX Next to Check = 0
Apr 24 00:25:04 freenas em0: RX Next to Refresh = 1023
Apr 24 00:25:04 freenas kernel: em0: link state changed to DOWN
Apr 24 00:25:04 freenas kernel: em0: link state changed to DOWN
Apr 24 00:25:08 freenas kernel: em0: link state changed to UP
Apr 24 00:25:08 freenas kernel: em0: link state changed to UP
Apr 24 00:25:33 freenas em0: Watchdog timeout Queue[0]-- resetting
Apr 24 00:25:33 freenas Interface is RUNNING and ACTIVE
Apr 24 00:25:33 freenas em0: TX Queue 0 ------
Apr 24 00:25:33 freenas em0: hw tdh = 0, hw tdt = 51
Apr 24 00:25:33 freenas em0: Tx Queue Status = -2147483648
Apr 24 00:25:33 freenas em0: TX descriptors avail = 973
Apr 24 00:25:33 freenas em0: Tx Descriptors avail failure = 1117497809
Apr 24 00:25:33 freenas em0: RX Queue 0 ------
Apr 24 00:25:33 freenas em0: hw rdh = 0, hw rdt = 1023
Apr 24 00:25:33 freenas em0: RX discarded packets = 0
Apr 24 00:25:33 freenas em0: RX Next to Check = 0
Apr 24 00:25:33 freenas em0: RX Next to Refresh = 1023
Apr 24 00:25:33 freenas kernel: em0: link state changed to DOWN

And here is my network config

Code:

em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: em0
        options=98<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
        ether 00:22:64:89:61:4d
        hwaddr 00:22:64:89:61:4d
        inet 192.168.212.100 netmask 0xffffff00 broadcast 192.168.212.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:35:3a:c3:0c:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: MineOs as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:22:64:40:f8:92
        hwaddr 02:b7:d0:00:04:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: Plex as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:22:64:ae:1b:75
        hwaddr 02:b7:d0:00:05:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: nextcloud as nic: epair0b
        options=8<VLAN_MTU>
        ether 00:22:64:ba:b5:81
        hwaddr 02:b7:d0:00:06:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair

Specs:

Code:

FreeNAS 11.3 U2.1
CPU: Intel Core i3-9100 4x 3.60GHz
MB: ASUS P11C-I
RAM: 2x16GB Kingston Server Premier KSM24ED8/16ME ECC
HDD: 4x 4TB Seagate Ironwolf Raidz
NIC: Intel 9301CT

SweetAndLow · Apr 23, 2020

Hardware specs? You need to provide information that might lead to someone being able to help. What setup do you have, what have you configured?

Yorick · Apr 23, 2020

Since em0 is having trouble, you're using an add-in NIC, and the on-board I210AT is known to be rock-solid, I'd swap networking over to on-board and see how things behave.

martin481997 · Apr 24, 2020

SweetAndLow said:
Hardware specs? You need to provide information that might lead to someone being able to help. What setup do you have, what have you configured?

I think all this is already provided in my signature and the network config is also there.

martin481997 · Apr 24, 2020

Yorick said:
Since em0 is having trouble, you're using an add-in NIC, and the on-board I210AT is known to be rock-solid, I'd swap networking over to on-board and see how things behave.

The I210AT doesnt work with VNET for me like i described here.
No Network in Jails if VNET is enabled
This was with Freenas 11.2 but i can check again with 11.3. And i dont think this can be hardware related because its everytime only at around 0:00.

Redcoat · Apr 24, 2020

martin481997 said:
I think all this is already provided in my signature and the network config is also there.

FYI, signature info isn't (always) available on portable devices ...

martin481997 · Apr 24, 2020

Ok i didnt know. Specs are now added in the thread.

Yorick · Apr 24, 2020

It looks like you simplified your networking since you had that VNET issue, no more VLAN setup. Is that right?

The i210-at is in most Supermicro boards, there are uncounted FreeNAS users with good success with it, and it's well-supported by FreeBSD. I haven't tried it with VLANs, as I have a simple home network (home LAN/WiFi, Guest WiFi, IoT WiFi) with no need to expose jails to Guest or IoT.

The reason I am looking at your NIC as a place to start troubleshooting is two-fold.

Indication that the NIC bounces, which would be driver related most likely:

Code:

Apr 24 00:24:44 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:44 freenas kernel: em0: link state changed to DOWN
Apr 24 00:24:47 freenas kernel: em0: link state changed to UP
Apr 24 00:24:47 freenas kernel: em0: link state changed to UP
Apr 24 00:24:49 freenas em0: Watchdog timeout Queue[0]-- resetting

And: em(4) appears to support your NIC, with a 82574L. Intel offers a driver on their download page for it, however. The i210-at, by contrast, uses the igb(4) driver. At question is the stability of em(4). See https://www.freebsd.org/releases/11.3R/hardware.html#ethernet

At the very least, this test will tell you whether the issue is NIC / driver-related. If it still occurs on igb0, then we don't need to point fingers at em(4). If the issue only occurs when you are on em0, but not igb0, then we can reasonably say the issue is somewhere within em(4).

Edit: If this issue occurs on either NIC, I'd start looking at the logs of the switch this connects to. Maybe the switch causes the link down, not FreeNAS?

martin481997 · Apr 24, 2020

Yorick said:
It looks like you simplified your networking since you had that VNET issue, no more VLAN setup. Is that right?

The i210-at is in most Supermicro boards, there are uncounted FreeNAS users with good success with it, and it's well-supported by FreeBSD. I haven't tried it with VLANs, as I have a simple home network (home LAN/WiFi, Guest WiFi, IoT WiFi) with no need to expose jails to Guest or IoT.

The reason I am looking at your NIC as a place to start troubleshooting is two-fold.

Indication that the NIC bounces, which would be driver related most likely:

Code:
Apr 24 00:24:44 freenas kernel: em0: link state changed to DOWN Apr 24 00:24:44 freenas kernel: em0: link state changed to DOWN Apr 24 00:24:47 freenas kernel: em0: link state changed to UP Apr 24 00:24:47 freenas kernel: em0: link state changed to UP Apr 24 00:24:49 freenas em0: Watchdog timeout Queue[0]-- resetting

And: em(4) appears to support your NIC, with a 82574L. Intel offers a driver on their download page for it, however. The i210-at, by contrast, uses the igb(4) driver. At question is the stability of em(4). See https://www.freebsd.org/releases/11.3R/hardware.html#ethernet

At the very least, this test will tell you whether the issue is NIC / driver-related. If it still occurs on igb0, then we don't need to point fingers at em(4). If the issue only occurs when you are on em0, but not igb0, then we can reasonably say the issue is somewhere within em(4).

Edit: If this issue occurs on either NIC, I'd start looking at the logs of the switch this connects to. Maybe the switch causes the link down, not FreeNAS?

On 11.2 I also tried igb without VLANs didnt work but now on 11.3 it seems to be working with VNET on. So i am testing it today night to watch if this is still hapenning on IGB0. The Switches shouldnt be the reason because they are all stupid unmanaged ones.

martin481997 · Apr 25, 2020

Using IGB0 fixed the issue, so this seems to be driver related or so with my eth.

Important Announcement for the TrueNAS Community.

Network Crashes but only at Night

martin481997

Dabbler

SweetAndLow

Sweet'NASty

Yorick

Wizard

martin481997

Dabbler

martin481997

Dabbler

Redcoat

MVP

martin481997

Dabbler

Yorick

Wizard

martin481997

Dabbler

martin481997

Dabbler

Similar threads

Important Announcement for the TrueNAS Community.

Network Crashes but only at Night

Dabbler

Sweet'NASty

Wizard

Dabbler

Dabbler

MVP

Dabbler

Wizard

Dabbler

Dabbler

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Network Crashes but only at Night"

Similar threads