Systems slow down after 24 hours

[TimTheSettler]

Hello. This is my first post.

I have three identical machines. Eventually each machine will be in different locations but for now they sit next to each other. They are meant to be basic file servers running syncthing. Triple redundancy with RAID.

Originally I intended to install Windows Server but my brother-in-law convinced me to try TrueNAS so I gave him the machines and he set up the OS. I took over and I continued the setup. I'm originally a DOS/Windows guy (30 years) so I'm completely new to Unix and FreeBSD. Since setting up the machines I realize now that my hardware is not ideal. For the past 15 years I've been using Windows Server which has been easy to use and quite reliable regardless of the hardware so my hope was that TrueNAS would be equally easy to use and reasonable considering the hardware.

Now for my problem. In general when I boot the servers they run quite nice. Syncthing synchronizes my files, the CPUs get good usage, the hard drives chug away nicely, and the network has great throughput. Unfortunately after about 24 hours everything slows down and the systems become unresponsive. I run htop and the CPUs are hardly used; maybe 50% usage for one core every couple of seconds. The hard drives become quiet with the occasional blip of usage every 5 seconds or so. The network slows down with about 15KB of data being transferred. Syncthing slows down to a crawl. If I reboot the servers they come back alive and chug away like normal. After 24 hours they all slow down again.

For the past couple of months I've been reading as much as I can to try to figure out what's going on. Reading posts here on the TrueNAS forums, on the FreeBSD forums, and on the syncthing forums. It's not the CPU. It's not the hard drives. It doesn't appear to be anything but I have some ideas of what might be happening. I just need some help to change some settings.

My guess is that it's the NIC. If I copy 2TB of data to one of the servers then syncthing kicks in and starts to synchronize those files to the other servers. All the servers start to chug away and the transfer starts to slow down but that's understandable since the servers are synchronizing as new data shows up. The transfer slows down but settles into a consistent rhythm. After about a day of copying data, the transfer slows down to a crawl and the original transfer starts to fail with network timeout errors.

The NIC is an Intel I219-V which is not supported in TrueNAS (according to the hardware document) but is supported in FreeBSD. There's actually very little difference between the two. For example, the driver is the same for both and quite generic. People have complained equally about both. I have yet to see any post where someone says that the "LM" works but the "V" does not. I've recently come across the following post from Intel and I think this might be the culprit. There are two posts by Intel moderators, one post points to the other --> "Modify the MSISupported parameter in the registry section of I219-V. The default value is "1", change it to "0". For more details, please refer to Mark Hanna's post"

My question (request for help) is, "How can I configure TrueNAS so that the NIC setting is the same as what Intel recommends?"

Links

TrueNAS Hardware Recommendations Guide	https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/
FreeBSD Support for the I219 NIC	https://redmine.ixsystems.com/issues/12399
FreeBSD Network Driver for I219	Intel® Network Adapter Gigabit Base Driver for FreeBSD* This download includes version 7.7.8 of the gigabit base driver for Intel® Network Connections. www.intel.com
MSI-X on Intel® Ethernet Connection Network Adapter Families	MSI-X on Intel® Ethernet Connection Network Adapter Families Support for Message Signaled Interrupts (MSI-X) on Intel® Ethernet Network Connections. www.intel.ca
Question about Intel Ethernet Connection I219-LM and I219-V Reset	Question about Intel Ethernet Connection I219-LM and I219-V Reset Hello, I have a question about Intel Ethernet Connection Reset. Confirmed that communication becomes unstable on a PC using the Intel NIC I219-LM or I219-V. When the phenomenon occurs, the ID 10400 is output when checking the Windows Event Log. For this reason, I think the NIC was reset by... community.intel.com
windows 8 with Intel 82566MM Network Card is not usable	windows 8 with Intel 82566MM Network Card is not usable I have HP Laptop i installed windows 8 and All my windows 7 drivers are working well on it except Intel 82566MM Network Card when i put my lan cable in it the system gone be mad alot of loss in packets and the device is laggy some times it gave me death screen i do all the trouble shooting for... community.intel.com

 

[sretalla]

My question (request for help) is, "How can I configure TrueNAS so that the NIC setting is the same as what Intel recommends?"

I would guess that it needs to be done via sysctl.

I don't know the driver and it's parameters list, so I can only guess that it would be something like sysctl hw.igb.MSISupported=0

If that's right, then you would create a tunable of sysctl type with that variable name and value to have it survive a reboot/upgrade of TrueNAS.

You could have a look at the variables already defined by looking at sysctl -a | grep hw.

 

[chris_hs]

Are you sure it's not Syncthing causing your problems?

Syncthing Jail with Folder Watcher Unusably Slow, Crashes Host
TrueNAS unusable when Syncthing plugin is enable

 

[TimTheSettler]

First of all, thanks for the great suggestions. It always amazes me how I search and search and each time there's something I didn't see before and then when I think that I've maxed out my search someone finds something I didn't find.

You could have a look at the variables already defined by looking at sysctl -a | grep hw.

I modified the command you suggested. I don't see anything useful for "hw.em" but "dev.em.0.iflib.disable_msix" is interesting but disappointing since it shows that the setting is already there.

Are you sure it's not Syncthing causing your problems?

Although I haven't seen the articles you mention there are other posts that talk about turning off the watcher.

I think I'll try some more tests for now. Thanks again.