netdata error notifications after 11.2 upgrade

[rwfitzy]

Since I upgraded to 11.2-U2.1 from 11.1-U7, I have been receiving lots of netdata udp error notifications: 1m ipv4 udp receive buffer errors = 140 errors number of UDP receive buffer errors during the last minute Alarm udp Family Escalated to CRITICAL Severity. The server was upgraded this past Sunday and I was not see any stats in the Reporting or dashboard after the upgrade. After a reboot, the reporting seems to be working now, but still getting notifications.

I saw a post on Server Fault to increase the recv buffer to a larger size to handle spikes in /etc/sysctl.conf by way of sysctl.conf and setting net.core.rmem_default and net.core.rmem_max. I understand the default is 1M, so I set the former to 2M and the latter to double (4M), but still receiving notifications.

No other noticeable issues with most services enabled and our incoming backups via NFS, SMB and FTP all working fine. This server also uses two lagg lacp interfaces for LAN/WAN with 6 ports, 2 on WAN lagg1 and 4 on LAN lagg0. This is a Supermicro server with the following specs. I couldn't find any posts regarding these messages. Is there an adjustment I can make to resolve the issue?

OS Version:
FreeNAS-11.2-U2.1
(Build Date: Feb 27, 2019 20:59)
Processor:
Intel(R) Xeon(TM) CPU 3.00GHz (4 cores)
Memory:
32 GiB

 

[dlavigne]

Were you able to resolve this or determine the cause for the messages?

 

[rwfitzy]

No, you have or seen this issue? It seems to be whenever there is a lot of activity. My system primarily handles backup from many servers each night and every morning I have a bunch of these messages. But all is working fine. I tried the sysctl tunables listed in my original post, but FreeNAS doesn't recognize. How can one increase the receive buffer?

 

[rwfitzy]

After a fresh install of 11.2-U3 and restore of backup settings, this is still happening. Anyone have an idea of the cause or how to resolve?

 

[rwfitzy]

Another thing to note on this issue is the IP address reported in the dashboard is the IP address assigned to a Jail. While my lagg has an assigned IP of x.x.x.25, a Jail on the server is assigned x.x.x.21 and shows in the dashboard as the IP for the Primary NIC for the lagg.

 

[Apollo]

Netdata is mostly reporting metrics.
Under Netdata webpage, you can set the alarm/alerts which trigger the various level or reporting.
I don't know if they are persistent or not.

 

[rwfitzy]

I don't see a Netdata webpage, where in the FreeNAS GUI? I am running 11.2-U3. Also, I am not getting any metrics. None of the graphs are reporting any activity

 

[Apollo]

It under "services".

 

[rwfitzy]

I go into Alarms at the top of the page, but don't see any way to set alarms. Some have role of silent, like ram, while others have a role such as sysadmin for the lagg1 interface. I do see the source file, do I need to make these settings in this file: /usr/local/etc/netdata/health.d/net.conf ?

Would like find out why I get all these ipv4.udperrors since upgrading to 11.2-U3 from 11.1. I can see all of them listed in the Log tab of the Alarms window of netdata as well. But it would be nice to get the notices to stop, so many. Luckily, no noticeable issues with the server.

 

[Apollo]

The "Alarms" section is what seems to be the thing to adjust.
I have never tried to configure it. I don't know if the setting are persistent once adjusted.
Prior to 11.2, netdata was not available I think, and therefore the various alarms to report errors and warnings where not possible this way.
It means prior to 11.2 we were not able to see what was happening under the hood and it is very likely those errors you are now getting notification
for were already present.

 

[rwfitzy]

Gotcha. However, I had metrics in FreeNAS before the upgrade and now no chart data except in the Netdata page. I see all the metrics and chart data in the Netdata page. On the FreeNAS dashboard, the primary IP shows the IP from my one Jail present on the server and all other Report boxes show no chart data and continuously indicate loading bar across the top of each box. No chart data present that I can find under Reporting. All min, max and avg show zeros.

 

[Apollo]

Dashboard is not very reliable in the sense it doesn't work with everything.
You can still monitor RRD charts with the Legacy GUI.
Ideally, Netdata metrics needs to be sent over to a syslog server of some kind so that it can be collected for storage and post-processing. Maximum netdata life expectancy is showing about the last 24hours.

Iocage, Docker, VM's are not hardened well enough for Freenas to provide a reliable network interface. As a side effect you can loose Web GUI access.
But it is getting a bit better on every new updates.

 

[rwfitzy]

Yes, all much better over the years. I first used 9.3 and the more important stability issues are all but gone. Some experience has helped too though.