Freenas memory problem

[z3r0bu6]

Hi there,


i have some problem with my freenas server. iam running 32 gb of ram, but the zfs cache is using nearly nothing of it. And as far i could read online it should be the other way.


i run on the same server 2 plugins (syncthing and emby) and 2 vms with (1 x 2gb ram and 1 x 4 gb ram)


and the amount of ram is weired. Any ideas here ?

 

[spiceygas]

ZFS Cache fills overtime as you use the system. How recently did you reboot? Is the usage slowly growing over time as you access files?

 

[HoneyBadger]

What revision of FreeNAS are you running? There were some revisions that had memory leaks in certain middleware components, so upgrading to the latest 11.3-U5 would be prudent if you aren't on there now.

You're also running 6GB of VMs and two plugins - I can see syncthing presently using less than 1GB, but that might change under activity - and how much is Emby using? This is memory that won't be available for ARC use.

The system is swapping to disk (482MB used) so that means you've got memory pressure somewhere.

 

[z3r0bu6]

yeah tam already at the latest version 11.3-U5. and now only a vm for container are running. Emby ist just running doing nothing. Syncthing is just running after creating the plugin so no syncs configured yet. Emby unconfigured aswell.
nothing else is running..

 

[HoneyBadger]

"Services" just keeps growing - can you dump the contents of ps -duxm here inside CODE tags?

 

[z3r0bu6]

Code:

USER   PID  %CPU %MEM     VSZ    RSS TT  STAT STARTED       TIME COMMAND
root     0   0.0  0.0       0  11936  -  DLs  Thu19     42:07.79 [kernel]
root     1   0.0  0.0    5392    192  -  ILs  Thu19      0:00.06 - /sbin/init --
root  1765   0.0  0.0   12480  12576  -  Ss   Thu19      0:03.81 |-- /usr/sbin/ntpd -g -c /etc/ntp.conf -p /var/run/n
root 54393   0.0  0.0   28868   6988  -  Is   06:34      0:00.00 |-- nginx: master process /usr/local/sbin/nginx
root 72050   0.0  0.0   11684   2236  -  IsJ  17:54      0:00.00 |-- nginx: master process /usr/local/sbin/nginx
root 60163   0.0  0.0   38172  11628  -  Ss   11:40      0:00.37 |-- /usr/local/sbin/winbindd --daemon
root 60164   0.0  0.0   38996  11612  -  I    11:40      0:00.07 | |-- winbindd: domain child [ALARIS] (winbindd)
root 61230   0.0  0.0   39460  11640  -  I    12:21      0:00.05 | `-- winbindd: idmap child (winbindd)
root  1417   0.0  0.1  129456  24204  -  S    Thu19      0:04.67 |-- /usr/local/bin/uwsgi --chdir=/usr/local/www --mo
root  1545   0.0  0.2  181740  65552  -  I    Thu19      0:12.22 | `-- /usr/local/bin/uwsgi --chdir=/usr/local/www --
root    89   0.0  0.0    9160   1012  -  Ss   Thu19      0:00.21 |-- /sbin/devd -q
root 63675   0.0  0.0   12608   5856  -  Ss   14:21      0:00.01 |-- /usr/local/sbin/sshd
root 80711   0.0  0.0   12920   7324  -  Ss   21:15      0:00.01 | `-- sshd: root@pts/2 (sshd)
root 80713   0.0  0.0    8152   4280  2  Ss   21:15      0:00.01 |   `-- -zsh (zsh)
root 80715   0.0  0.0    6964   2964  2  R+   21:15      0:00.00 |     `-- ps -duxm
root  2370   0.0  0.0    8496   2612  -  Is   Thu19      0:00.15 |-- /usr/local/sbin/smartd -n -i 1800 -c /usr/local/
root 60179   0.0  0.0   31184   9872  -  Ss   11:40      0:00.30 |-- /usr/local/sbin/nmbd --daemon
root  1508   0.0  0.0    9008   2884  -  Ss   Thu19      0:00.03 |-- /usr/sbin/zfsd
root   968   0.0  0.0    6508   1976  -  Is   Thu19      0:00.01 |-- dhclient: igb0 [priv] (dhclient)
root 71869   0.0  0.0    6544   1428  -  IsJ  17:54      0:00.00 |-- dhclient: epair0b [priv] (dhclient)
root 78709   0.0  0.0    6544   2096  -  IsJ  20:51      0:00.00 |-- dhclient: epair0b [priv] (dhclient)
root 60142   0.0  0.4  165304 136732  -  Is   11:40      0:00.55 |-- /usr/local/sbin/smbd --daemon
root 60154   0.0  0.3  122836  94984  -  I    11:40      0:00.02 | |-- /usr/local/sbin/smbd --daemon
root 60155   0.0  0.3  122884  95028  -  I    11:40      0:00.02 | |-- /usr/local/sbin/smbd --daemon
root 66962   0.0  0.5  176568 141660  -  S    16:59      0:03.06 | `-- /usr/local/sbin/smbd --daemon
root  2558   0.0  0.0   42336  11080  -  Ss   Thu19      0:38.27 |-- /usr/local/bin/rrdcached -s www -l /var/run/rrdc
root 72449   0.0  0.0   18652   5680  -  Ss   18:04      0:00.01 |-- /usr/sbin/rpc.lockd
root 71983   0.0  0.0    6428   1536  -  IsJ  17:54      0:00.02 |-- /usr/sbin/syslogd -c -ss
root 78828   0.0  0.0    6420   2208  -  IsJ  20:51      0:00.00 |-- /usr/sbin/syslogd -c -ss
root 72385   0.0  0.0    6332   2328  -  Ss   18:04      0:00.01 |-- /usr/sbin/rpcbind
root 72406   0.0  0.0   18664   5916  -  Is   18:04      0:00.05 |-- /usr/sbin/mountd -rS /etc/exports /etc/zfs/expor
root  1518   0.0  0.0    6424   2136  -  Ss   Thu19      0:00.18 |-- /usr/sbin/cron -s
root 72058   0.0  0.0    6460   1364  -  SsJ  17:54      0:00.03 |-- /usr/sbin/cron -J 15 -s
root 80707   0.0  0.0    6460   1388  -  DJ   21:15      0:00.00 | `-- cron: running job (cron)
root 80709   0.0  0.0    6460   1388  -  SVJ  21:15      0:00.00 |   `-- cron: running job (cron)
root 78895   0.0  0.0    6460   2088  -  SsJ  20:51      0:00.01 |-- /usr/sbin/cron -J 15 -s
root 80706   0.0  0.0    6460   2088  -  DJ   21:15      0:00.00 | `-- cron: running job (cron)
root 80708   0.0  0.0    6460   2088  -  SVJ  21:15      0:00.00 |   `-- cron: running job (cron)
root  1582   0.0  0.0    6368   1792 v1  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv1
root  1583   0.0  0.0    6368   1792 v2  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv2
root  1584   0.0  0.0    6368   1792 v3  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv3
root  1585   0.0  0.0    6368   1792 v4  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv4
root  1586   0.0  0.0    6368   1792 v5  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv5
root  1587   0.0  0.0    6368   1792 v6  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv6
root  1588   0.0  0.0    6368   1792 v7  Is+  Thu19      0:00.00 |-- /usr/libexec/getty Pc ttyv7
root 72417   0.0  0.0   18552   5640  -  Is   18:04      0:00.01 |-- nfsd: master (nfsd)
root 72419   0.0  0.0    6264   2124  -  S    18:04      0:00.01 | `-- nfsd: server (nfsd)
root    99   0.0  0.0    6348   1728  -  Is   Thu19      0:00.00 |-- daemon: /usr/local/bin/middlewared[100] (daemon)
root   100   0.3  0.8  317388 237744  -  S    Thu19      9:29.95 | `-- python3.7: middlewared (python3.7)
root 74474   4.3  2.6 2149240 824544  -  SC   19:55      4:14.20 |   |-- bhyve: 1_dockerhost (bhyve)
root   106   0.0  0.0   15940   4576  -  I    Thu19      0:00.04 |   |-- /usr/local/bin/python3.7 -c from multiproces
root   164   0.0  0.3  180016  78536  -  I    Thu19      0:16.23 |   |-- python3.7: middlewared (worker) (python3.7)
root   529   0.0  0.0   15940   4512  -  I    Thu19      0:00.04 |   | `-- /usr/local/bin/python3.7 -c from multiproc
root   165   0.0  0.3  179248  80836  -  I    Thu19      0:16.51 |   |-- python3.7: middlewared (worker) (python3.7)
root   528   0.0  0.0   15940   4512  -  I    Thu19      0:00.04 |   | `-- /usr/local/bin/python3.7 -c from multiproc
root   166   0.0  0.3  179760  84812  -  I    Thu19      0:16.56 |   |-- python3.7: middlewared (worker) (python3.7)
root   533   0.0  0.0   15940   4512  -  I    Thu19      0:00.04 |   | `-- /usr/local/bin/python3.7 -c from multiproc
root   167   0.0  0.3  180272  79020  -  I    Thu19      0:17.24 |   |-- python3.7: middlewared (worker) (python3.7)
root   527   0.0  0.0   15940   4512  -  I    Thu19      0:00.04 |   | `-- /usr/local/bin/python3.7 -c from multiproc
root   168   0.0  0.4  180272 121188  -  I    Thu19      0:16.70 |   |-- python3.7: middlewared (worker) (python3.7)
root   522   0.0  0.0   15940   4512  -  I    Thu19      0:00.04 |   | `-- /usr/local/bin/python3.7 -c from multiproc
root 38860   0.0  0.2   79380  57504  -  S    22:18      0:04.44 |   |-- python3.7: middlewared (zettarepl) (python3.
root 74475   0.0  0.1   22408  18580  -  S    19:55      0:00.53 |   `-- /usr/local/bin/python3.7 /usr/local/libexec/
root 80693   0.0  0.1   22408  18736  -  I    21:14      0:00.00 |     `-- /usr/local/bin/python3.7 /usr/local/libexe
root  2582   0.0  0.0    6348   1792  -  Is   Thu19      0:00.00 |-- daemon: /usr/local/sbin/collectd[2583] (daemon)
root  2583   0.0  0.1   86056  31388  -  S    Thu19      5:57.36 | `-- /usr/local/sbin/collectd -f
root 60192   0.0  0.0    6348   2116  -  Is   11:40      0:00.00 |-- daemon: /usr/local/bin/wsdd.py[60193] (daemon)
root 72431   0.0  0.0  280744   5724  -  Ss   18:04      0:00.01 |-- /usr/sbin/rpc.statd
root  2551   0.0  0.0   13324   5056  -  I    Thu19      0:00.00 |-- /usr/local/sbin/syslog-ng -p /var/run/syslog.pid
root  2552   0.0  0.0   18712   6108  -  Ss   Thu19      0:00.50 | `-- /usr/local/sbin/syslog-ng -p /var/run/syslog.p
root  1581   0.0  0.0   71140   9652 v0  Is+  Thu19      0:00.80 `-- /usr/local/bin/python /etc/netcli (python3.7)
root     2   0.0  0.0       0     16  -  DL   Thu19      0:00.00 - [crypto]
root     3   0.0  0.0       0     16  -  DL   Thu19      0:00.00 - [crypto returns]
root     4   0.0  0.0       0     32  -  DL   Thu19      0:46.54 - [cam]
root     5   0.0  0.0       0     32  -  DL   Thu19      0:00.03 - [ctl]
root     6   0.0  0.0       0     16  -  DL   Thu19      0:00.01 - [soaiod1]
root     7   0.0  0.0       0     16  -  DL   Thu19      0:00.01 - [soaiod2]
root     8   0.0  0.0       0     16  -  DL   Thu19      0:00.01 - [soaiod3]
root     9   0.0  0.0       0     16  -  DL   Thu19      0:00.01 - [soaiod4]
root    10   0.0  0.0       0     16  -  DL   Thu19      0:00.00 - [audit]
root    11 400.0  0.0       0     64  -  RNL  Thu19   5822:33.31 - [idle]
root    12   0.0  0.0       0    304  -  WL   Thu19     32:08.52 - [intr]
root    13   0.0  0.0       0     48  -  DL   Thu19      4:15.43 - [geom]
root    14   0.0  0.0       0    240  -  DL   Thu19      0:02.66 - [usb]
root    15   0.0  0.0       0    272  -  DL   Thu19      4:35.08 - [zfskern]
root    16   0.0  0.0       0     16  -  DL   Thu19     11:11.90 - [rand_harvestq]
root    17   0.0  0.0       0     48  -  DL   Thu19      2:10.27 - [pagedaemon]
root    18   0.0  0.0       0     16  -  DL   Thu19      0:00.00 - [vmdaemon]
root    19   0.0  0.0       0     16  -  DNL  Thu19      0:00.00 - [pagezero]
root    20   0.0  0.0       0     16  -  DL   Thu19      0:00.32 - [bufdaemon]
root    21   0.0  0.0       0     16  -  DL   Thu19      2:07.94 - [bufspacedaemon]
root    22   0.0  0.0       0     16  -  DL   Thu19      0:46.03 - [syncer]
root    23   0.0  0.0       0     16  -  DL   Thu19      0:00.35 - [vnlru]
root   637   0.0  0.0       0     16  -  DL   Thu19      0:00.00 - [ftcleanup]
root   641   0.0  0.0       0     16  -  DL   Thu19      0:00.00 - [g_mp_kt]
root  2262   0.0  0.0       0     16  -  DL   Thu19      1:30.75 - [g_eli[0] gptid/9809]
root  2263   0.0  0.0       0     16  -  DL   Thu19      1:33.80 - [g_eli[1] gptid/9809]
root  2264   0.0  0.0       0     16  -  DL   Thu19      1:28.30 - [g_eli[2] gptid/9809]
root  2265   0.0  0.0       0     16  -  DL   Thu19      1:33.66 - [g_eli[3] gptid/9809]
root  2267   0.0  0.0       0     16  -  DL   Thu19      1:29.40 - [g_eli[0] gptid/982b]
root  2268   0.0  0.0       0     16  -  DL   Thu19      1:34.22 - [g_eli[1] gptid/982b]
root  2269   0.0  0.0       0     16  -  DL   Thu19      1:25.52 - [g_eli[2] gptid/982b]
root  2270   0.0  0.0       0     16  -  DL   Thu19      1:37.26 - [g_eli[3] gptid/982b]
root  2271   0.0  0.0       0     16  -  DL   Thu19      1:28.66 - [g_eli[0] gptid/97f9]
root  2272   0.0  0.0       0     16  -  DL   Thu19      1:31.47 - [g_eli[1] gptid/97f9]
root  2273   0.0  0.0       0     16  -  DL   Thu19      1:25.63 - [g_eli[2] gptid/97f9]
root  2274   0.0  0.0       0     16  -  DL   Thu19      1:36.48 - [g_eli[3] gptid/97f9]
root  2275   0.0  0.0       0     16  -  DL   Thu19      1:29.21 - [g_eli[0] gptid/981d]
root  2276   0.0  0.0       0     16  -  DL   Thu19      1:30.04 - [g_eli[1] gptid/981d]
root  2277   0.0  0.0       0     16  -  DL   Thu19      1:25.81 - [g_eli[2] gptid/981d]
root  2278   0.0  0.0       0     16  -  DL   Thu19      1:29.32 - [g_eli[3] gptid/981d]
root  2398   0.0  0.0       0     16  -  DL   Thu19      0:00.09 - [g_mirror swap0]
root  2401   0.0  0.0       0     16  -  DL   Thu19      0:00.09 - [g_mirror swap1]
root  2403   0.0  0.0       0     16  -  DL   Thu19      0:00.52 - [g_eli[0] mirror/swa]
root  2404   0.0  0.0       0     16  -  DL   Thu19      0:00.49 - [g_eli[1] mirror/swa]
root  2405   0.0  0.0       0     16  -  DL   Thu19      0:00.51 - [g_eli[2] mirror/swa]
root  2406   0.0  0.0       0     16  -  DL   Thu19      0:00.50 - [g_eli[3] mirror/swa]
root  2427   0.0  0.0       0     16  -  DL   Thu19      0:00.51 - [g_eli[0] mirror/swa]
root  2428   0.0  0.0       0     16  -  DL   Thu19      0:00.50 - [g_eli[1] mirror/swa]
root  2429   0.0  0.0       0     16  -  DL   Thu19      0:00.48 - [g_eli[2] mirror/swa]
root  2430   0.0  0.0       0     16  -  DL   Thu19      0:00.50 - [g_eli[3] mirror/swa]

 

[HoneyBadger]

You've got a relatively huge amount of memory in the "inactive" state (19G) which isn't being cleaned up by the laundry queue.

Memory - FreeBSD Wiki

wiki.freebsd.org

It looks like your Services and even your Swap amounts have increased between initial and second post - which makes little sense since you stopped the 4G VM and it almost looks like whatever other process gobbled it up.

Do you have this FreeNAS machine mounting other servers/exports as a client? Trying to think if it's somehow trying to cache a different filesystem as a client.

If you try stopping your plugins and the other VMs, and check top over time to see if your INACT memory memory starts to drop (it will go through "laundry") if it's dirty memory, and then get cleaned - this might help. But based on the trends here I feel like you might be headed for a reboot.