11.1u7 Webdav swap_pager_getswapspace failed

[fr0stbyte]

I have 11.1u7 setup as a Webdav server. Almost every other service is turned off. Raidz2 64TB with 64GB ECC Ram and no tuning. I have some Windows server backing up to this system overnight via Webdav with a Backup software. A few weeks ago, the Freenas system started locking up every night with swap_pager_getswapspace failed. Upon investigating, the system was using 100% swap space. I then tried enabling auto tune. The system became unresponsive much quicker with auto tune turned on the following night. I disabled auto tune, but left the vfs.zfs.arc_max=61776000000. The next night the web interface became unresponsive and dmesg showed sonewconn: pcb 0xfffff80152eaeae0: Listen queue overflow: 193 already in queue a waiting acceptance (16 occurrences). I have attached the netstat -Lan and swap output. Any suggestions?

 

[SweetAndLow]

Hardware specs?

Also it would have been interesting to see what was using all your memory. Try looking at top next time.

 

[fr0stbyte]

Build FreeNAS-11.1-U7
Platform Intel(R) Xeon(R) CPU E5-1660 v2 @ 3.70GHz
Memory 65462MB
Storage 10 x 8TB HGST Deskstar NAS in Raidz2

Most of the time when it locks up with the swap_pager_getswapspace failed, i cannot do a top. I can get connected to SSH, but it won't produce output from any command. I can only reboot the machine.

 

[SweetAndLow]

Build FreeNAS-11.1-U7
Platform Intel(R) Xeon(R) CPU E5-1660 v2 @ 3.70GHz
Memory 65462MB
Storage 10 x 8TB HGST Deskstar NAS in Raidz2

Most of the time when it locks up with the swap_pager_getswapspace failed, i cannot do a top. I can get connected to SSH, but it won't produce output from any command. I can only reboot the machine.

Yeah that probably true it's it's just thrashing on swap. It's just strange that you have everything turned off and you see this problem. Most people see it when using jails or vm's and have over committed memory.

Also note that you didn't include you hardware correctly. It seems like you don't have a motherboard, PSU or case? You might want to get those for a working system.


What does top look like during a backup from your servers?

 

[fr0stbyte]

I don't know what PSU, but it should be a 1200 watt gold variety and a Chinese 4u rack-mount case with lots of hot swap bays. Mobo is ASRock EP2C602. I attached a top from a few minutes ago for reference.

 

[SweetAndLow]

Well if it was my system I would try to reproduce the issue so that I could narrow it down to a single thing. Does the backup process cause the swap issues? This could be a bug in webdav.

 

[fr0stbyte]

The issue is reproduced every night and yes the backup process causes the swap. Webdav is all we are doing on this Freenas box.

 

[anodos]

Can you get top and ps output from when a backup job is in progress?

 

[fr0stbyte]

I started the backup job on 3 of the remote server and then took a ps and top:

USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 11 1193.8 0.0 0 192 - RL 08:19 2276:37.00 [idle]
root 12 1.3 0.0 0 592 - WL 08:19 3:02.13 [intr]
webdav 17426 1.2 0.0 13948 9080 - S 11:30 0:00.40 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17394 0.6 0.0 13952 9032 - D 11:30 0:00.51 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17390 0.5 0.0 13948 9052 - S 11:29 0:00.59 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17427 0.4 0.0 13948 9032 - S 11:30 0:00.30 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17380 0.1 0.0 13948 9028 - S 11:29 0:00.44 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17381 0.1 0.0 13948 9028 - S 11:29 0:00.42 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17389 0.1 0.0 13948 9052 - S 11:29 0:00.61 /usr/local/sbin/httpd -DNOHTTPACCEPT
webdav 17396 0.1 0.0 13948 9036 - S 11:30 0:00.51 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 0 0.0 0.0 0 18000 - DLs 08:19 2:39.34 [kernel]
root 1 0.0 0.0 5388 600 - SLs 08:19 0:00.06 /sbin/init --
root 2 0.0 0.0 0 16 - DL 08:19 0:00.00 [crypto]
root 3 0.0 0.0 0 16 - DL 08:19 0:00.00 [crypto returns]
root 4 0.0 0.0 0 64 - DL 08:19 0:18.39 [cam]
root 5 0.0 0.0 0 80 - DL 08:19 0:00.00 [ctl]
root 6 0.0 0.0 0 192 - DL 08:19 0:06.03 [zfskern]
root 7 0.0 0.0 0 16 - DL 08:19 0:00.00 [soaiod1]
root 8 0.0 0.0 0 16 - DL 08:19 0:00.00 [soaiod2]
root 9 0.0 0.0 0 16 - DL 08:19 0:00.00 [soaiod3]
root 10 0.0 0.0 0 16 - DL 08:19 0:00.00 [audit]
root 13 0.0 0.0 0 192 - DL 08:19 0:00.00 [ng_queue]
root 14 0.0 0.0 0 48 - DL 08:19 0:00.97 [geom]
root 15 0.0 0.0 0 240 - DL 08:19 0:00.76 [usb]
root 16 0.0 0.0 0 16 - DL 08:19 0:00.00 [soaiod4]
root 17 0.0 0.0 0 16 - DL 08:19 0:03.63 [rand_harvestq]
root 18 0.0 0.0 0 48 - DL 08:19 0:00.88 [pagedaemon]
root 19 0.0 0.0 0 16 - DL 08:19 0:00.00 [vmdaemon]
root 20 0.0 0.0 0 16 - DL 08:19 0:00.00 [pagezero]
root 21 0.0 0.0 0 16 - DL 08:19 0:00.03 [bufspacedaemon]
root 22 0.0 0.0 0 16 - DL 08:19 0:00.03 [bufdaemon]
root 23 0.0 0.0 0 16 - DL 08:19 0:00.03 [vnlru]
root 24 0.0 0.0 0 16 - DL 08:19 0:00.56 [syncer]
root 221 0.0 0.0 6344 1348 - Is 08:19 0:00.00 daemon: /usr/local/bin/middlewared[223] (daemon)
root 223 0.0 0.2 188472 135388 - S 08:19 0:18.22 python3.6: middlewared (python3.6)
root 312 0.0 0.0 0 16 - DL 08:20 0:00.47 [ipmi0: kcs]
root 352 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_mp_kt]
root 395 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_mirror swap0]
root 397 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_mirror swap1]
root 399 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_mirror swap2]
root 401 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_mirror swap3]
root 403 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_mirror swap4]
root 405 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[0] mirror/swa]
root 406 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[1] mirror/swa]
root 407 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[2] mirror/swa]
root 408 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[3] mirror/swa]
root 409 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[4] mirror/swa]
root 410 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[5] mirror/swa]
root 411 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[6] mirror/swa]
root 412 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[7] mirror/swa]
root 413 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[8] mirror/swa]
root 414 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[9] mirror/swa]
root 415 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[10] mirror/sw]
root 416 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[11] mirror/sw]
root 419 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[0] mirror/swa]
root 420 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[1] mirror/swa]
root 421 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[2] mirror/swa]
root 422 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[3] mirror/swa]
root 423 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[4] mirror/swa]
root 424 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[5] mirror/swa]
root 425 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[6] mirror/swa]
root 426 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[7] mirror/swa]
root 427 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[8] mirror/swa]
root 428 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[9] mirror/swa]
root 429 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[10] mirror/sw]
root 430 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[11] mirror/sw]
root 433 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[0] mirror/swa]
root 434 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[1] mirror/swa]
root 435 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[2] mirror/swa]
root 436 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[3] mirror/swa]
root 437 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[4] mirror/swa]
root 438 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[5] mirror/swa]
root 439 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[6] mirror/swa]
root 440 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[7] mirror/swa]
root 441 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[8] mirror/swa]
root 442 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[9] mirror/swa]
root 443 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[10] mirror/sw]
root 444 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[11] mirror/sw]
root 447 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[0] mirror/swa]
root 448 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[1] mirror/swa]
root 449 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[2] mirror/swa]
root 450 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[3] mirror/swa]
root 451 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[4] mirror/swa]
root 452 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[5] mirror/swa]
root 453 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[6] mirror/swa]
root 454 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[7] mirror/swa]
root 455 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[8] mirror/swa]
root 456 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[9] mirror/swa]
root 457 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[10] mirror/sw]
root 458 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[11] mirror/sw]
root 461 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[0] mirror/swa]
root 462 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[1] mirror/swa]
root 463 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[2] mirror/swa]
root 464 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[3] mirror/swa]
root 465 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[4] mirror/swa]
root 466 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[5] mirror/swa]
root 467 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[6] mirror/swa]
root 468 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[7] mirror/swa]
root 469 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[8] mirror/swa]
root 470 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[9] mirror/swa]
root 471 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[10] mirror/sw]
root 472 0.0 0.0 0 16 - DL 08:20 0:00.00 [g_eli[11] mirror/sw]
root 1036 0.0 0.0 0 16 - DL 08:21 0:00.00 [ftcleanup]
root 1369 0.0 0.0 6564 1956 - Is 08:21 0:00.00 dhclient: em0 [priv] (dhclient)
_dhcp 1427 0.0 0.0 6564 1980 - Is 08:21 0:00.00 dhclient: em0 (dhclient)
nobody 1545 0.0 0.0 6924 2236 - Ss 08:22 0:00.09 /usr/local/sbin/mdnsd
nobody 1546 0.0 0.0 6924 2332 - Ss 08:22 0:00.11 /usr/local/sbin/mdnsd
root 1771 0.0 0.0 6556 1724 - Is 08:22 0:00.00 /usr/sbin/moused -p /dev/ums0 -t auto -I /var/run/moused.ums0.pid
root 1795 0.0 0.0 6556 1724 - Is 08:22 0:00.00 /usr/sbin/moused -p /dev/ums1 -t auto -I /var/run/moused.ums1.pid
root 1812 0.0 0.0 9172 5216 - Is 08:22 0:00.00 /sbin/devd -q
root 2009 0.0 0.0 12864 4112 - I 08:22 0:00.00 /usr/local/sbin/syslog-ng -p /var/run/syslog.pid
root 2010 0.0 0.0 20280 6360 - Ss 08:22 0:00.56 /usr/local/sbin/syslog-ng -p /var/run/syslog.pid
root 2138 0.0 0.0 3584 3648 - Ss 08:22 0:36.50 /usr/sbin/watchdogd --pretimeout 5 --pretimeout-action log,printf
root 2178 0.0 0.0 7068 2660 - Is 08:22 0:00.00 /usr/sbin/ctld
root 2433 0.0 0.0 12500 12608 - Ss 08:22 0:00.47 /usr/sbin/ntpd -g -c /etc/ntp.conf -p /var/run/ntpd.pid -f /var/db/ntpd.drift
root 2779 0.0 0.0 12568 6480 - Is 08:22 0:00.00 /usr/local/sbin/sshd
root 3014 0.0 0.0 10476 4624 - Is 08:22 0:00.28 /usr/local/sbin/smartd -n -i 1800 -c /usr/local/etc/smartd.conf -p /var/run/smartd-daemon.pid
root 3039 0.0 0.0 7732 2572 - Is 08:22 0:00.00 /usr/local/bin/rsync --daemon --config /usr/local/etc/rsync/rsyncd.conf
root 3043 0.0 0.0 6344 2152 - Is 08:22 0:00.00 daemon: /usr/local/sbin/pbid[3044] (daemon)
root 3044 0.0 0.0 7096 3668 - I 08:22 0:00.05 /bin/sh /usr/local/sbin/pbid
root 3185 0.0 0.0 12272 6900 - Is 08:22 0:00.00 nginx: master process /usr/local/sbin/nginx
www 3186 0.0 0.0 12272 7680 - I 08:22 0:00.60 nginx: worker process (nginx)
messagebus 3196 0.0 0.0 7344 3036 - Is 08:22 0:00.00 /usr/local/bin/dbus-daemon --system
nobody 3199 0.0 0.0 6924 2816 - Ss 08:22 0:00.09 /usr/local/sbin/mdnsd
root 3286 0.0 0.1 93840 79472 - I 08:22 0:09.82 python: alertd (python3.6)
root 3298 0.0 0.2 136668 112728 - S 08:22 0:03.17 /usr/local/bin/uwsgi --chdir=/usr/local/www --module=freenasUI.wsgi:application --master --pidfile=/var/run/django.pid --fastcgi-socket=127.0.0.1:9042 -T --threads=1
root 3306 0.0 0.0 6344 2152 - Is 08:22 0:00.00 daemon: /usr/bin/env[3307] (daemon)
root 3307 0.0 0.0 49104 12000 - S 08:22 0:00.65 /usr/local/bin/consul-alerts start --alert-addr=localhost:8542 --watch-events --watch-checks --log-level=warn
root 3313 0.0 0.0 6344 2252 - Is 08:22 0:00.00 daemon: /usr/bin/env[3314] (daemon)
root 3314 0.0 0.0 37936 19828 - I 08:22 0:15.40 /usr/local/bin/consul agent -data-dir=/var/tmp/consul -config-dir=/usr/local/etc/consul.d -server -bootstrap-expect=1 -ui -bind=127.0.0.1 -enable-script-checks=true
root 3618 0.0 0.0 110428 29268 - Ss 08:23 0:21.25 /usr/local/sbin/collectd
root 3687 0.0 0.0 13948 8492 - Ss 08:23 0:00.16 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 4424 0.0 0.0 33164 13724 - I 08:23 0:00.05 consul watch -http-addr localhost:8500 -datacenter dc1 -type checks /usr/local/bin/consul-alerts watch checks --alert-addr localhost:8542 --log-level warn
root 4425 0.0 0.0 30984 12348 - I 08:23 0:00.02 consul watch -http-addr localhost:8500 -datacenter dc1 -type event /usr/local/bin/consul-alerts watch event --alert-addr localhost:8542 --log-level warn
root 4511 0.0 0.0 9000 4696 - Is 08:23 0:00.01 /usr/sbin/zfsd
root 4521 0.0 0.0 6484 2520 - Ss 08:23 0:00.02 /usr/sbin/cron -s
root 4739 0.0 0.2 202008 139852 - I 08:23 0:42.91 /usr/local/bin/uwsgi --chdir=/usr/local/www --module=freenasUI.wsgi:application --master --pidfile=/var/run/django.pid --fastcgi-socket=127.0.0.1:9042 -T --threads=1
root 4886 0.0 0.0 13948 8576 - I 08:26 0:00.04 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 8741 0.0 0.0 37552 30232 - S 09:19 0:02.24 python: webshelld (python3.6)
root 14997 0.0 0.0 13948 8576 - I 11:11 0:00.01 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15045 0.0 0.0 13948 8576 - S 11:12 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15055 0.0 0.0 13948 8576 - I 11:13 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15078 0.0 0.0 13948 8576 - I 11:13 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15083 0.0 0.0 13948 8576 - I 11:13 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15095 0.0 0.0 13948 8576 - I 11:13 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15096 0.0 0.0 13948 8576 - I 11:13 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 15098 0.0 0.0 13948 8576 - I 11:13 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16955 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16963 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16964 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16972 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16973 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16974 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 16975 0.0 0.0 13948 8576 - I 11:26 0:00.00 /usr/local/sbin/httpd -DNOHTTPACCEPT
root 17134 0.0 0.0 4192 1996 - I 11:27 0:00.00 sleep 300
root 17285 0.0 0.0 12880 7372 - Ss 11:28 0:00.02 sshd: root@pts/0 (sshd)
root 4729 0.0 0.1 69596 62488 v0 Is+ 08:23 0:01.11 /usr/local/bin/python /etc/netcli (python3.6)
root 4730 0.0 0.0 6364 2200 v1 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv1
root 4731 0.0 0.0 6364 2200 v2 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv2
root 4732 0.0 0.0 6364 2200 v3 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv3
root 4733 0.0 0.0 6364 2200 v4 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv4
root 4734 0.0 0.0 6364 2200 v5 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv5
root 4735 0.0 0.0 6364 2200 v6 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv6
root 4736 0.0 0.0 6364 2200 v7 Is+ 08:23 0:00.00 /usr/libexec/getty Pc ttyv7
root 17312 0.0 0.0 7468 3856 0 Ss 11:28 0:00.01 -csh (csh)
root 17435 0.0 0.0 6948 3008 0 R+ 11:31 0:00.00 ps aux

 

[anodos]

Keep it running and check on it periodically to see if the httpd process has RAM usage growing without bounds.

 

[SweetAndLow]

Well httpd is already using 8GB of memory. Seems like that could be the problem.

 

[anodos]

Well httpd is already using 8GB of memory. Seems like that could be the problem.

Yeah, my brain was fried when I saw that. @frostbyte please file a bug ticket if you haven't yet.

 

[fr0stbyte]

Will do. Thanks for the assistance.

 

[fr0stbyte]

Well, I have held off on the bug report because shortly after I replied yesterday, the httpd memory usage had went back down. What it looks like is happening is when the backup job begins, it is doing a delta comparison. That comparison causes httpd to spike memory usage on the NAS. Once that delta comparison is complete, the backup job is transferring data. Long story short the NAS didn't become unresponsive last night. The only thing that I have done different was that I hadn't rebooted the NAS after the tuning changes previous nights. I did reboot it yesterday before creating this post. I have attached what the memory and ARC looked like last night. Is this normal for the scenario I explained above? My only tuning variable is vfs.zfs.arc_max=61776000000. Also the NAS dmesg was full of swap_pager_getswapspace failed this morning again, so I am guessing that means the NAS was killing off processes to try and regain swap?

 

[SweetAndLow]

Well, I have held off on the bug report because shortly after I replied yesterday, the httpd memory usage had went back down. What it looks like is happening is when the backup job begins, it is doing a delta comparison. That comparison causes httpd to spike memory usage on the NAS. Once that delta comparison is complete, the backup job is transferring data. Long story short the NAS didn't become unresponsive last night. The only thing that I have done different was that I hadn't rebooted the NAS after the tuning changes previous nights. I did reboot it yesterday before creating this post. I have attached what the memory and ARC looked like last night. Is this normal for the scenario I explained above? My only tuning variable is vfs.zfs.arc_max=61776000000. Also the NAS dmesg was full of swap_pager_getswapspace failed this morning again, so I am guessing that means the NAS was killing off processes to try and regain swap?

The fact that memory increased and decreases shows it's not a memory leak. But that's almost worse because that means it's working the way it's designed. You should still file a bug and include your graphs. What backup application are you using? Can you do it over something other than webdav?

 

[fr0stbyte]

I am using Backup Assist. It also has an rsync options, but rsync with BA doesn't support versioning, just a mirror copy. We really need versioning. I can't seem to report a bug, when I click the link it tells me the project is closed and is read only.

 

[anodos]

I am using Backup Assist. It also has an rsync options, but rsync with BA doesn't support versioning, just a mirror copy. We really need versioning. I can't seem to report a bug, when I click the link it tells me the project is closed and is read only.

We're in the process of migrating the bugtracker off of redmine.