11.2 Beta1 Netdata ram swapped out - system.swapio

[Amsoil_Jim]

Beta 3 does not make this go away... I started getting alerts about Ram after installing Beta 2 and I thought they had enabled some new alerts or something not realizing I had Netdata turned on for a while and never received any of these alerts... I installed Beta 3 yesterday and today started getting alerts again relating to Ram. I have received a couple other alerts when a scrub is taking place. When I look at my reports in the old UI i can see way higher CPU usage and ram usage since installing 11.2 Beta 2

 

[Wolfeman0101]

Beta 3 does not make this go away... I started getting alerts about Ram after installing Beta 2 and I thought they had enabled some new alerts or something not realizing I had Netdata turned on for a while and never received any of these alerts... I installed Beta 3 yesterday and today started getting alerts again relating to Ram. I have received a couple other alerts when a scrub is taking place. When I look at my reports in the old UI i can see way higher CPU usage and ram usage since installing 11.2 Beta 2

Exact same issues with RAM and even after going from 16 to 32 GB still getting spammed errors and swapping RAM.

 

[Amsoil_Jim]

Exact same issues with RAM and even after going from 16 to 32 GB still getting spammed errors and swapping RAM.

I have 192GB, so... i have plenty of RAM

 

[anodos]

Beta 3 does not make this go away... I started getting alerts about Ram after installing Beta 2 and I thought they had enabled some new alerts or something not realizing I had Netdata turned on for a while and never received any of these alerts... I installed Beta 3 yesterday and today started getting alerts again relating to Ram. I have received a couple other alerts when a scrub is taking place. When I look at my reports in the old UI i can see way higher CPU usage and ram usage since installing 11.2 Beta 2

Can you post contents of /etc/local/netdata/health.d/ram.conf ?
There was a version bump for netdata between 11.1 and 11.2, and the upstream added more support for freebsd. Unfortunately, the probably needs to be tuned on a case-by-case basis, which means that work needs to be done to expose it in the UI.

 

[Amsoil_Jim]

ram.conf contents

Code:

# you can disable an alarm notification by setting the 'to' line to: silent

   alarm: used_ram_to_ignore
	  on: system.ram
	  os: linux freebsd
   hosts: *
	calc: ($zfs.arc_size.arcsz = nan)?(0):($zfs.arc_size.arcsz)
   every: 10s
	info: the amount of memory that is reported as used, but it is actually capable for resizing itself based on the system needs (eg. ZFS ARC)

   alarm: ram_in_use
	  on: system.ram
	  os: linux
   hosts: *
#   calc: $used * 100 / ($used + $cached + $free)
	calc: ($used - $used_ram_to_ignore) * 100 / ($used - $used_ram_to_ignore + $cached + $free)
   units: %
   every: 10s
	warn: $this > (($status >= $WARNING)  ? (80) : (90))
	crit: $this > (($status == $CRITICAL) ? (90) : (98))
   delay: down 15m multiplier 1.5 max 1h
	info: system RAM used
	  to: sysadmin

   alarm: ram_available
	  on: mem.available
	  os: linux
   hosts: *
	calc: ($avail + $used_ram_to_ignore) * 100 / ($system.ram.used + $system.ram.cached + $system.ram.free + $system.ram.buffers)
   units: %
   every: 10s
	warn: $this < (($status >= $WARNING)  ? ( 5) : (10))
	crit: $this < (($status == $CRITICAL) ? (10) : ( 5))
   delay: down 15m multiplier 1.5 max 1h
	info: estimated amount of RAM available for userspace processes, without causing swapping
	  to: sysadmin

## FreeBSD
alarm: ram_in_use
   on: system.ram
   os: freebsd
hosts: *
 calc: (($active + $wired) - $used_ram_to_ignore) * 100 / (($active + $wired) - $used_ram_to_ignore + $cached + $free)
units: %
every: 10s
 warn: $this > (($status >= $WARNING)  ? (80) : (90))
 crit: $this > (($status == $CRITICAL) ? (90) : (98))
delay: down 15m multiplier 1.5 max 1h
 info: system RAM usage
   to: sysadmin

 alarm: ram_available
	on: system.ram
	os: freebsd
 hosts: *
  calc: ($free + $inactive + $used_ram_to_ignore) * 100 / ($free + $active + $inactive + $wired + $cache + $buffers)
 units: %
 every: 10s
  warn: $this < (($status >= $WARNING)  ? ( 5) : (10))
  crit: $this < (($status == $CRITICAL) ? (10) : ( 5))
 delay: down 15m multiplier 1.5 max 1h
  info: estimated amount of RAM available for userspace processes, without causing swapping
	to: sysadmin

I am getting email alerts at above 90% usage


I get these when a scrub is running


The good thing is that since installing Beta 3 my CPU usage seems to have returned to normal.

Before 11.2 Beta 2


After 11.2 Beta 2 and Beta 3

 

[gurdalal]

I just changed my update train version to 11.2 STABLE but I start getting the warnings below:
ram in use = 90.3%
30min ram swapped out = 23.6% of RAM

And wired RAM usage is above %90

This version is NOT stable at all. Before I was at 11.0STABLE and wasn't getting any of these warnings.

 

[Wolfeman0101]

Yeah it's still happening to me on 11.2-RC1 still. I had to manually set the vfs.zfs.arc_max to something reasonable or I'd be swapping and getting spammed by warnings. I have 32GB of RAM and 17GB of storage.

 

[anodos]

Yeah it's still happening to me on 11.2-RC1 still. I had to manually set the vfs.zfs.arc_max to something reasonable or I'd be swapping and getting spammed by warnings. I have 32GB of RAM and 17GB of storage.

To be fair, these warning weren't being generated on 11.1. 11.2 bumped the version of netdata which caused it to actually start alerting when you were swapping hard.

 

[Wolfeman0101]

To be fair, these warning weren't being generated on 11.1. 11.2 bumped the version of netdata which caused it to actually start alerting when you were swapping hard.

I guess it could've been happening but it's weird that RAM is being managed so poorly to cause swapping when I have double the recommended amount.