getswapspace failure

[nanodec]

Hey guys - been a while since I've posted, but i've been getting weekly log emails with freenas needing to kill uid-0 to fix this. I didn't see this until I went to v9.10, so I'm wondering if any of you have been seeing this or if you know what is causing it. I'm rolling no jails at all.

Thoughts?

Craig

> swap_pager_getswapspace(10): failed

> swap_pager_getswapspace(8): failed

> swap_pager_getswapspace(8): failed

> swap_pager_getswapspace(13): failed

> swap_pager_getswapspace(7): failed

> pid 1320 (devd), uid 0, was killed: out of swap space

quick post of swapinfo:

[root@freenasbox ~]# swapinfo
Device 512-blocks Used Avail Capacity
/dev/ada0p1.eli 4194304 80696 4113608 2%
/dev/ada1p1.eli 4194304 80096 4114208 2%
/dev/ada2p1.eli 4194304 82336 4111968 2%
/dev/ada3p1.eli 4194304 85144 4109160 2%
Total 16777216 328272 16448944 2%
[root@freenasbox ~]#

 

[jgreco]

There was a bug (memory leak) in devd that supposedly has been fixed. Please try updating to the latest version of FreeNAS and see if the problem persists.

 

[toadman]

There was a bug (memory leak) in devd that supposedly has been fixed. Please try updating to the latest version of FreeNAS and see if the problem persists.

Do you know in what version it was fixed?

Asking as I had a weird issue with active memory (and the arc) slowly dropping over time. Not in an of itself an issue (as arc hit rates were fine), but the system was left with a large amount of inactive memory (like 5GB in a 12GB system). Then the system started to swap, which seems odd when there are multiple GB of inactive memory that could be allocated. (Described in the 9.10 swap thread.)

 

[jgreco]

No, I'm sorry I don't, but if you check the bug tracker or the forum for devd in the last ~month? or two? ... that'll probably get you what you need to know.

 

[nanodec]

It's odd... I'll recheck but when I looked for updates thinking the same thing it wasn't populating any. I'll check again tonight and see. It seemed weird...

thanks @jgreco

 

[scriptmonkey]

Having the same issue as well:

>swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> swap_pager_getswapspace(16): failed
> pid 1198 (devd), uid 0, was killed: out
of swap space

I am running 9.10 as well so I will take a look at the bug reports as well. Thanks

 

[nanodec]

I checked and i'm on current 9.10, stating i'm up to date...

thoughts?

 

[cyberjock]

You should have multiple GB of RAM free as a normal process. That's one of many reasons why I keep saying that ZFS doesn't scale down as well as it scales up.

You need to have RAM free for the middleware to work, other processes to take RAM as necessary, etc. You'll get nasty things like processes terminating inappropriately and in really nasty cases, kernel panics, from having insufficient free RAM.

 

[scriptmonkey]

You should have multiple GB of RAM free as a normal process. That's one of many reasons why I keep saying that ZFS doesn't scale down as well as it scales up.

You need to have RAM free for the middleware to work, other processes to take RAM as necessary, etc. You'll get nasty things like processes terminating inappropriately and in really nasty cases, kernel panics, from having insufficient free RAM.

I am running 12 GB RAM in an i7 box only running 1 500GB iSCSI volume and a 10TB CIFS share, with a uptime of a little over 4 days, I would hope I have enough ram for that. I am running No jails, no plugins, just a plain vanilla install.

 

[cyberjock]

I am running 12 GB RAM in an i7 box only running 1 500GB iSCSI volume and a 10TB CIFS share, with a uptime of a little over 4 days, I would hope I have enough ram for that. I am running No jails, no plugins, just a plain vanilla install.

You are correct. I was responding to your prior comment about there being "a large amount of RAM inactive". Totally normal.

The fact you had to use swap (and then later ran out of swap) is a bad thing. For your situation I'd not expect to have your issues with your hardware, so there's almost certainly a software bug at work. Upgrading to the latest 9.10 is my recommendation. If the issue is still occurring, I'd open a bug ticket at bugs.freenas.org and see what the devs have to say. There is almost certainly a bug in the software somewhere.

You could also try shutting off all of the services in the WebGUI and see if one of them is responsible for the swap usage, and by turning it off and seeing swap usage drop significantly that could be a clue as to the problem.

 

[nanodec]

yeah it's weird. The box I'm using is not running any jails, and my swap is never used. I only saw this error pop up once I upgraded to 9.10

I haven't seen it in the past week and a half, and the system takes care of automatically killing the thread... just seems kinda odd...

 

[firestorm99]

Today i got a mail with lots of similar messages:

Code:

kernel log messages:
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
swap_pager_getswapspace(16): failed
....

Screenshot from the reporting tab confirms the error: Seems something startet using more and more RAM over the weeks:

My System:
- No jails
- AFP and SMB shares

Code:

Build: FreeNAS-9.10-STABLE-201606270534 (dd17351)
Platform: Intel(R) Core(TM) i5-2400S CPU @ 2.50GHz
Memory: 10GB
Uptime: 41 days
ZPool: 5x 2TB Raid Z1

Can anybody give a hint how to finde the responsible process?

 

[firestorm99]

This night same thing happend. FreeNAS even sent me a "the pool has degraded" email. After a reboot everything was fine.

Code:

(da5:mpt1:0:4:0): WRITE(10). CDB: 2a 00 ae 2e 86 88 00 00 08 00
(da5:mpt1:0:4:0): CAM status: CCB request terminated by the host
(da5:mpt1:0:4:0): Retrying command
da5 at mpt1 bus 0 scbus3 target 4 lun 0
da5: <ATA WDC WD20EARS-00M AB51> s/n  WD-WCAZAA203208 detached
GEOM_ELI: g_eli_read_done() failed (error=6) da5p1.eli[READ(offset=8097792, length=4096)]
swap_pager: I/O error - pagein failed; blkno 2099133,size 4096, error 6
vm_fault: pager read error, pid 3614 (zfsd)
GEOM_ELI: g_eli_read_done() failed (error=6) da5p1.eli[READ(offset=4788224, length=28672)]
swap_pager: I/O error - pagein failed; blkno 2098325,size 28672, error 6
vm_fault: pager read error, pid 1112 (devd)
swap_pager: I/O error - pagein failed; blkno 2099540,size 32768, error 6
vm_fault: pager read error, pid 3614 (zfsd)
Failed to write core file for process zfsd (error 14)
pid 3614 (zfsd), uid 0: exited on signal 11
GEOM_ELI: Device da5p1.eli destroyed.
GEOM_ELI: Detached da5p1.eli on last close.
(da5:mpt1:0:4:0): Periph destroyed
swap_pager: I/O error - pagein failed; blkno 2098332,size 16384, error 6
vm_fault: pager read error, pid 1112 (devd)
Failed to write core file for process devd (error 14)
pid 1112 (devd), uid 0: exited on signal 11
da5 at mpt1 bus 0 scbus3 target 4 lun 0
da5: <ATA WDC WD20EARS-00M AB51> Fixed Direct Access SPC-3 SCSI device
da5: Serial Number  WD-WCAZAA203208
da5: 300.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da5: quirks=0x8<4K>
swap_pager: I/O error - pagein failed; blkno 2097618,size 4096, error 6
vm_fault: pager read error, pid 3051 (collectd)
swap_pager: I/O error - pagein failed; blkno 2097796,size 32768, error 6
vm_fault: pager read error, pid 3051 (collectd)
Failed to write core file for process collectd (error 14)
pid 3051 (collectd), uid 0: exited on signal 11
swap_pager: I/O error - pagein failed; blkno 2459473,size 45056, error 6
vm_fault: pager read error, pid 2780 (python2.7)
pid 2780 (python2.7), uid 0: exited on signal 11