Transmission/qBittorrent services killed "out of swap space" since 12.x version jails

[baskabas]

I have been using FreeNAS 10/11 for years with the Transmission plugin always running stable. But since I upgraded the TrueNAS and the jails to version 12.x the service of specifically my torrent plugins randomly stops after an hour or so. I also have a MiniDLNA version 12.x jail running just fine. I kept my Plex and Ubiquiti jails at 11.2 because they don't have this issue. I don't know how to debug te problem, get the right logging, etc. so I installed 2 new qBittorrent plugins from scratch (version 12.2) and they got the same issue?! After a couple of hours the service crashes and it becomes unavailable. I don't have any error to start with and a general search for "jail/service stopping randomly" did not get me to anything tangible. Any help would be great. I am not too shy to dive in the terminal. Thanks!

 

[Grinas]

I had the same problem a few months ago with qbittorrent when i upgraded. I spent a while trying to figure out why but couldnt determine the cause.

I just created a new jail and copied over the config and havent had a problem since. I have 2 qbittorrent jails 1 on 12.1 and 1 on 12.2. Do you have a snapshots of your jails you can revert too?

have a look here how to create a new jail and copy your old config.

 

[Alecmascot]

I vaguely remember an issue with transmission fixed by disabling "uTP".

 

[baskabas]

I had the same problem a few months ago with qbittorrent when i upgraded. I spent a while trying to figure out why but couldnt determine the cause.

I just created a new jail and copied over the config and havent had a problem since. I have 2 qbittorrent jails 1 on 12.1 and 1 on 12.2. Do you have a snapshots of your jails you can revert too?

have a look here how to create a new jail and copy your old config.

Hi Grinas, I created the qBittorrent jails from scratch in 12.2. That's what's weird. The Transmission jails upgraded from 11.2 to 12.2 and started having issues... so I though it would change with a newly created jail with a different program. I removed the Transmission jails, but I do have some snapshots from 11.2 yes. I would really like to solve the issue actually.

 

[baskabas]

I was reading up in the "Plex service stopping inside jail" thread and I noticed a few comments about memory usage so I went and looked for that... and I didn't notice earlier. but the qbittorrent jails are using all the 16GB of ram after a while nog leaving anything for the rest of the system. After reviewing the connection setting I found a typo in the maximum connections per torrent setting... it was set to 100 instead of 10. It's stable now. Maybe my transmission jails caused memory issues as well.

 

[baskabas]

Unfortunately the services are still crashing. How can I get more info about the issue?

 

[Grinas]

I was reading up in the "Plex service stopping inside jail" thread and I noticed a few comments about memory usage so I went and looked for that... and I didn't notice earlier. but the qbittorrent jails are using all the 16GB of ram after a while nog leaving anything for the rest of the system. After reviewing the connection setting I found a typo in the maximum connections per torrent setting... it was set to 100 instead of 10. It's stable now. Maybe my transmission jails caused memory issues as well.

I think you have something else going on is your Truenas over provisioned? I have 500+ torrents running on my qbittorrent and a tonne of other jails without any issues.

Is it all or just some jails that are crashing. Have you checked the logs for the jail and for the services running in the jail?

 

[baskabas]

I think you have something else going on is your Truenas over provisioned? I have 500+ torrents running on my qbittorrent and a tonne of other jails without any issues.

Is it all or just some jails that are crashing. Have you checked the logs for the jail and for the services running in the jail?

That sounds promising. Where can I find these logs?

 

[Grinas]

That sounds promising. Where can I find these logs?

For
Truenas
/var/log/


For iocage
/var/log/iocage.log

For service in jail this will depend on what the service is and how it's installed and configured. For my qbittorrent logs I go to.
/var/db/qbittorrent/conf/qBittorrent/data/logs/qbittorrent.log

 

[baskabas]

Bingo! In the /var/log directory I found a couple of messages:

Code:

Nov 23 02:09:47 HOST kernel: pid 3691 (qbittorrent-nox), jid 4, uid 0, was killed: out of swap space
Nov 23 02:09:47 HOST kernel: pid 3032 (qbittorrent-nox), jid 3, uid 0, was killed: out of swap space

Now to figure out where I can manage my swap space...

 

[baskabas]

So I upgraded the RAM to 32GB to see if that solves the issue. Nope!
Again just a couple of minutes after 02:00 first get a couple of (with different indexnumbers):

Code:

swap_pager_getswapspace(0): failed

And after that the was killed: out of swap space error for Plex and the two qBittorrent services.

I looked at the memory reports and swap space usage for the past month was max 500MB of 2GB?

 

[Grinas]

now that you know whats wrong i would say open a new thread or change the name of this one and provide more details on your specs. CPU, RAM, # of Vms, Jails, version etc..

did a search for the error and found a few threads.
https://www.truenas.com/community/threads/out-of-swap-space-error.74794/

minidlnad was killed: out of swap space

So i bought myself a HP ProLiant MicroServer G7 N54L 4 GB-U SATA-Server and installed Freenas 9.2 on a 16GB USB2-stick added 4 * 1TB western digital SATA drives then i did set up a (non-optimal) ZFS created userspecific directory a public directory eneabled (local) FTP and SSH access...

www.truenas.com

 

[baskabas]

Yes, I found these also. Memory usage of all services maxes out at about 7GB when everything is running. Swap space usage max 520MB.
I removed all old snapshots (read that in another thread) and rebooted. The processes are always killed around 02:05 at night.

My specs are in my signature.

 

[Christopher_P]

This seems to be becoming common behavior for folks with home media setups. When you exhaust swap space, the kernel is going to kill off whatever it can preserve the integrity of the system. For me, sometimes that's Plex, Sonarr or Radarr others today it was LazyLibrarian and Plex twice.

Here's a thread I jumped onto earlier this year when I first noticed this was an issue.

Sonarr Crashes Truecore 12.0-U2.1

Hello everyone, I have an interesting problem with my sever. When I use Sonarr and go into a TV show and click automatic search it will search for the tv show but the UI will time out. The system would come back within 20-30sec no problem but then I search again and it times out. I wanted to...

www.truenas.com

I have been running on basically the same hardware for the last 3-4 years (specs below). This only started becoming an issue earlier this year. It was definitely after I upgraded to Truenas 12.x but I can't recall exactly when. Let me know if anyone has suggestions, parts for my new rig are here so more memory is not too far away!

Code:

Dec 27 17:54:32 freenas swap_pager_getswapspace(32)[29248]: Last message 'failed' repeated 8 times, suppressed by syslog-ng on freenas.homenet.lan
Dec 27 17:54:32 freenas swap_pager_getswapspace(20): failed
Dec 27 17:54:32 freenas swap_pager_getswapspace(18): failed
Dec 27 17:54:37 freenas kernel: pid 62524 (python3.6), jid 9, uid 111, was killed: out of swap space
Dec 27 17:55:36 freenas kernel[29248]: Last message 'pid 62524 (python3.6' repeated 1 times, suppressed by syslog-ng on freenas.homenet.lan

Fractal Design Node 804
X10SL7-F w/ 32G ECC RAM
Xeon E3-1231 v3 CPU
Two pools, single vdev in each with 6x16TB and 6x10TB disks in a RAIDZ2 configuration

 

[Christopher_P]

I should have closed the loop on this sooner. For some reason, I found that my system had some "autotune" settings enabled. It's possible I toggled something a while back and just don't remember doing it. Anyway, after disabling the autotune settings, this behavior has mostly disappeared. I've had it happen once or twice since disabling autotune but it's no longer a regular occurrence. I think I was just pushing the system a little too hard those other times. I simply turned off some unneeded jails and have been running mostly ok since.