Disabling packet storm alarms on netdata

[phlmike18]

I'm being driven a bit batty with the email storms of packet storms on my ixl1. I ssh'd in and in netdata.conf I only see net_packets.vtnet0 but I do not see one for ixl1. I searched with nano. I also looked in health_alarms_notify.conf.

I did set enabled = no for the vtnet0 and I set DEFAULT_RECIPIENT_EMAIL to root|critical. Not sure yet if it will have an effect, I don't get them every day but when I do I get dozens in one fell swoop.

I'm on FreeNAS-11.2-RELEASE, this is production so updates are done quarterly only.

I think the ability to have an on/off switch for alarms on the gui were feature requests made to both netdata and freenas.

 

[Meyers]

This was frustrating for me too. I just went through all this and discovered that you have to change two different configs. One is for the running instance of netdata and one is to make the config persist between reboots (in case that ever happens outside of an upgrade). You'll have to change this again on every upgrade because the config gets overwritten.

Both of these have to be changed:

/usr/local/etc/netdata/netdata.conf
/conf/base/etc/local/netdata.conf

Code:

[health]
        enabled = no

This works for me and persists between reboots.

 

[phlmike18]

*sigh* C'mon FreeNAS! Between this sort of thing and that atrocious modern-UI that has 1/3 the functionality that literally nobody uses except nerdy lumberjacks with manbuns and skinny jeans.

 

[Meyers]

That's a bit harsh and unfair, but I get where you're coming from. I think the UI was dramatically improved in U2 but the reporting and other things aren't quite there yet for me. Reporting is completely broken right now on one box. Netdata feels tacked on.

I've had other issues that are ongoing with 11.2 enough to where my patience is spread pretty thin right now. I'm seriously considering just using vanilla FreeBSD or ZFS on Linux which appears to be stable enough for production use. To be fair, a lot of this is because I can configure all this stuff myself and my needs are minimal (just need NFS really). I'd prefer something less like an appliance and more like all of our other systems I configure with Ansible. Monitoring FreeNAS has been a real pain too.

 

[phlmike18]

I was being facetious, or well, half facetious. The old UI had its issues and the new UI had it's advantages going all the way back to the pulled FreeNAS 10.
See the issue is, FreeNAS is growing and aspiring to be so much more than a Free NAS software. Think XCP-NG with ZFS tacked on. My issue is, ZFS is WAYYY too memory hungry to also be the hypervisor.

As far as I know, ZFS on Debian is not too terribly broken for mild production use, but I would not use it on Mission Critical. Vanilla FreeBSD would be the way to go, just have good hardware. I'm building my dad a NAS for his house, and I'm using debian 9 with a simple raid Z2.

I use NFS right now, but I am on vmware as a vhost. I have development environments on vanilla freebsd with ZFS and it works, but you do quickly miss some of the features of FreeNAS.

I wish their was a FreeNAS light - that has basic alerts, little to no UI that did storage and iscsi/nfs/cifs, where you can install basic freebsd and just do pkg install freenas-light or something of the like. No jails, no vms, no apple crap.

It isn't that Netdata is just tacked on, but netdata is not complete. The whole editing it in two places is just the way FreeNAS operates.

 

[Chris Moore]

I am not sure what you are looking for.

I wish their was a FreeNAS light - that has basic alerts, little to no UI that did storage and iscsi/nfs/cifs, where you can install basic freebsd and just do pkg install freenas-light or something of the like. No jails, no vms, no apple crap.

I suppose you could submit a feature request on the Redmine site:
https://redmine.ixsystems.com/projects/freenas/

 

[Meyers]

As far as I know, ZFS on Debian is not too terribly broken for mild production use, but I would not use it on Mission Critical.

Looks like ZFS on Linux is ready for mission critical use. I'm going to test it out on my backup server that FreeNAS 11.2 won't properly run on.

https://www.reddit.com/r/zfs/comments/86x8um/zfs_on_linux_ready_for_prime_time/

https://www.reddit.com/r/linuxadmin/comments/a9owpd/zol_experience_on_production/ecmjkw3

 

[Chris Moore]

Looks like ZFS on Linux is ready for mission critical use. I'm going to test it out on my backup server that FreeNAS 11.2 won't properly run on.

I have some large systems, over 300 TB and 60 drives, at work that are running reliably with RedHat (RHEL) and ZFS on Linux with a few other things tacked on to make management a little easier.
Like this for example:
https://github.com/zfsonlinux/zfs-auto-snapshot

I still find FreeNAS easier to manage and I am looking to convert the Linux systems over to FreeNAS in the near future.

 

[Meyers]

I still find FreeNAS easier to manage

Oh no doubt about that.

I am testing FreeBSD 12 now because I wanted ZFS on root on install. It's easier to incorporate something like this into our environment than it is FreeNAS for a variety of reasons. Our FreeNAS servers aren't really as integrated with our environment as every other server is which is annoying.

Plus I'd prefer to configure almost everything with Ansible if I can. I know I could do that with FreeNAS as well but I try to treat that as more of an appliance as I believe the docs suggested.

 

[Chris Moore]

Plus I'd prefer to configure almost everything with Ansible if I can. I know I could do that with FreeNAS as well but I try to treat that as more of an appliance as I believe the docs suggested.

Have you looked into the API that FreeNAS has?
There was a discussion here that might be helpful:
https://www.ixsystems.com/community/threads/will-it-openstack.64073/page-2#post-489720

The rewrite from the old API to the new middlewared continues. Once the API stabilizes and the rewrite is complete, api.freenas.org will be deprecated and replaced by the new API documentation. In the mean time, to see the API documentation for the new middleware, log into the new UI, click on the URL for the FreeNAS system in your browser’s location bar, and add /api/docs to the end of that URL.

 

[Beemer533]

This was frustrating for me too. I just went through all this and discovered that you have to change two different configs. One is for the running instance of netdata and one is to make the config persist between reboots (in case that ever happens outside of an upgrade). You'll have to change this again on every upgrade because the config gets overwritten.

Both of these have to be changed:

/usr/local/etc/netdata/netdata.conf
/conf/base/etc/local/netdata.conf

Code:

[health]
        enabled = no

This works for me and persists between reboots.

Thanks, I've been dealing with this with my build. This seems to be working for me... Hopefully cut down on the massive amount of emails.

Only difference is that the second file I had to edit was here:

/conf/base/etc/local/netdata/netdata.conf

 

[Mikk36_]

Better option is to edit the netdata/health.d/net.conf files directly and only set the 10s packet storm alert to silent.