Intermittent outages

[seanleblanc]

I'm now running TrueNAS-12.0-U8.1, but even on FreeNAS, I've had this issue that would crop up from time to time - the machine would become unresponsive - ping, web interface, and NFS would all be non-responsive. Trying to plugin the HDMI cable into a monitor and use the keyboard would not seem to get the computer to wake up, either - I would have to power cycle the machine.

However, what I've noticed is that the interface I've used had DHCP clicked on it, and so it was getting a secondary IP besides a static one I set. The last thing I saw in /var/log/messages before the machine was power cycled seem to be related to the interface going up and down:

Jun 13 16:32:31 freenas kernel: re0: link state changed to DOWN
Jun 13 16:32:34 freenas 1 2022-06-13T16:32:34.577984-06:00 freenas.local dhclient 7685 - - New IP Address (re0): 192.168.0.178
Jun 13 16:32:34 freenas 1 2022-06-13T16:32:34.579096-06:00 freenas.local dhclient 7686 - - New Subnet Mask (re0): 255.255.255.0
Jun 13 16:32:34 freenas 1 2022-06-13T16:32:34.580215-06:00 freenas.local dhclient 7687 - - New Broadcast Address (re0): 192.168.0.255
Jun 13 16:32:34 freenas 1 2022-06-13T16:32:34.581317-06:00 freenas.local dhclient 7688 - - New Routers (re0): 192.168.0.1
Jun 13 16:32:34 freenas kernel: re0: link state changed to UP
Jun 13 16:34:02 freenas kernel: re0: link state changed to DOWN
Jun 13 16:34:05 freenas kernel: re0: link state changed to UP
Jun 13 16:34:09 freenas 1 2022-06-13T16:34:09.202650-06:00 freenas.local dhclient 7740 - - New IP Address (re0): 192.168.0.178
Jun 13 16:34:09 freenas 1 2022-06-13T16:34:09.203861-06:00 freenas.local dhclient 7741 - - New Subnet Mask (re0): 255.255.255.0
Jun 13 16:34:09 freenas 1 2022-06-13T16:34:09.205025-06:00 freenas.local dhclient 7742 - - New Broadcast Address (re0): 192.168.0.255
Jun 13 16:34:09 freenas 1 2022-06-13T16:34:09.206171-06:00 freenas.local dhclient 7743 - - New Routers (re0): 192.168.0.1
Jun 13 16:43:05 freenas kernel: re0: link state changed to DOWN
Jun 13 16:43:09 freenas kernel: re0: link state changed to UP
Jun 13 16:43:10 freenas 1 2022-06-13T16:43:10.830901-06:00 freenas.local dhclient 7860 - - New IP Address (re0): 192.168.0.178
Jun 13 16:43:10 freenas 1 2022-06-13T16:43:10.832033-06:00 freenas.local dhclient 7861 - - New Subnet Mask (re0): 255.255.255.0
Jun 13 16:43:10 freenas 1 2022-06-13T16:43:10.833147-06:00 freenas.local dhclient 7862 - - New Broadcast Address (re0): 192.168.0.255
Jun 13 16:43:10 freenas 1 2022-06-13T16:43:10.834259-06:00 freenas.local dhclient 7863 - - New Routers (re0): 192.168.0.1
Jun 13 16:43:25 freenas kernel: re0: link state changed to DOWN
Jun 13 16:43:33 freenas 1 2022-06-13T16:43:33.881355-06:00 freenas.local dhclient 7884 - - New IP Address (re0): 192.168.0.178
Jun 13 16:43:33 freenas 1 2022-06-13T16:43:33.882472-06:00 freenas.local dhclient 7885 - - New Subnet Mask (re0): 255.255.255.0
Jun 13 16:43:33 freenas 1 2022-06-13T16:43:33.883569-06:00 freenas.local dhclient 7886 - - New Broadcast Address (re0): 192.168.0.255
Jun 13 16:43:33 freenas 1 2022-06-13T16:43:33.884709-06:00 freenas.local dhclient 7887 - - New Routers (re0): 192.168.0.1
Jun 13 16:43:34 freenas kernel: re0: link state changed to UP

I've turned off DHCP for this interface now. Do you think it's possible that turning on DHCP for this interface was possibly causing the network to stop responding at later times like this? If not, is there extra debugging I can turn on to try to troubleshoot this problem?

 

[Samuel Tai]

Yes, we've seen all sorts of anomalous behavior with Realtek NICs. This is just another case of Realteks ignoring a static IP in favor of DHCP. Your workaround of disabling DHCP may result in stability in your installation, but standard advice here is to switch to a better grade of NIC, Intel 1G NICs being highly favored for rock-solid stability.

 

[seanleblanc]

Intel 1G NICs

Do you happen to have any specific card model numbers I could look up on NewEgg, Amazon, etc.?

 

[Samuel Tai]

The PRO/1000 series are highly recommended, and single- and dual-port models are inexpensive and easy to find on the used market.

 

[cogprying]

I'm now running TrueNAS-12.0-U8.1, but even on FreeNAS, I've had this issue that would crop up from time to time - the machine would become unresponsive - ping, web interface, and NFS would all be non-responsive. Trying to plugin the HDMI cable into a monitor and use the keyboard would not seem to get the computer to wake up, either - I would have to power cycle the machine.

On TrueNAS 11.3-RELEASE-p14, I was able to get things stable again by following this [Guide] Resolve Realtek NIC Stability Issues on FreeBSD. I'm using version 1.96 and it seems to be stable. This also fixes smb only transferring at 11mb(!).

Related thread:

Truenas 12 and Realtek - Updating Your Driver to 1.96 to Prevent "re0 watchdog timeout"

Didn't see a new and up to date post for this once common issue so I figured I would make one. Apologies if this is the wrong forum for this tip. Upon upgrading from FreeNAS 11.3-U5 to TrueNAS Core 12, I had discovered that my NAS would randomly drop off of the network and I was paranoid I had...

www.truenas.com

(This is also for other people, especially as non-technical as me, who searching end up here looking for a solution; for me upgrading from 11.1 to 11.3 necessitated going through this again.)

An aside, but every post that says to "just put in a real NIC" doesn't seem to acknowledge the reality that these cards are present in a great quantity in the wild, often in systems where you can't just drop in a different one. In my case, my only free pcie slot is being used by a SAS card, which is essential for my device to be a functioning NAS. I don't know how the performance would be with a USB 2.0 NIC, or if BSD's support would be better for one. I know I'd love to use an Intel NIC, but it's simply not feasible, and the constant replies to to perform an untenable action aren't helpful.

So, regardless of how "crappy" Realtek is, the tl;dr seems to be "use the official driver", at least through 11.3. It seems as though 12+ may be regressive, and BSD, after 17 years, still remains unable to provide the same reliability with this hardware as other OSes(?).