HOWTO: Set-up TMPFS in a Jail for transcoding or Zoneminder

Klontje

Dabbler
Joined
Feb 7, 2016
Messages
47
I just had to RMA two SSD's because they died after 10 months of Zoneminder video data processing taking place on them. This resulted in more than 400TiB of write activity. First thing that came to mind was to move the memory map file used for this to my HDD pool but to be honest, no storage media is made for 20MB/s continuous write 24x7. Digging some deeper I found that the best practice is to move the SWAP and MAP data to a ramdisk. There was no cut-and-paste solution anywhere so it took some digging and testing to get it working and thought it might be good to document this.

Step 1: FreeNAS jail host configuration

Make sure the following entries are added to the sysctl's section of your Jail. You can find this under 'Advanced Mode' when you open your jails settings in the GUI of FreeNAS. These settings will allow FreeNAS/FreeBSD to mount a tmpfs filesystem which is a dynamic ramdisk on FreeBSD.

allow.raw_sockets=true,allow.mount.tmpfs,enforce_statfs=0,allow.mount

Restart your jail to make these settings effective.

Step 2: FreeNAS jail configuration
Go into your jails command line and edit the file /etc/fstab (which probably is empty) and add the following line:

tmpfs /tmp tmpfs rw,mode=777 0 0

Next step is to edit/create /etc/rc.local and add the following line:

mount /tmp

This will automatically mount the tmpfs ramdisk we defined in fstab at each start of the jail. For some reason fstab entries are not automatically mounted inside a jail.

Restart your jail again to activate the tmpfs ramdisk. You can check if it is active and mounted with the command mount | grep tmpfs. This should show an entry for the /tmp mount inside your jail. You can issue this command in or outside of the jail.

Step 3: Configure your application
Last step is to make sure that your application uses /tmp for any transcoding or streaming data processing. For Zoneminder this means go into the web console and edit the values for PATH_SWAP and PATH_MAP on the Paths tab to /tmp.

Restart your application to activate the changes.
 

Frank Collins

Explorer
Joined
Apr 10, 2019
Messages
53
Does this work for ZM 1.34.9 in a 11.3 jail, please?
 

abishai

Cadet
Joined
Apr 12, 2020
Messages
8
Sure thing. Actually, this is mandatory for ZM - it kills SSD rather fast or puts rather harsh load on spindle drives. I strongly suggest to include this to FreeNAS plugin.

I recommend to limit tmpfs size with size=xxx option (check zm.mmap file size and set it accordingly) or you can ran out of memory if something bad happens.

For example, tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0 (for fstab)
 

abishai

Cadet
Joined
Apr 12, 2020
Messages
8
Another good candidate for tmpfs is /var/tmp - ZM uses this directory for images it sends to front end (web or ZMNinja).
 

Frank Collins

Explorer
Joined
Apr 10, 2019
Messages
53
I tried to do all this but there have been changes to both ZM and FN which render it unusable.
Step 1: FreeNAS jail host configuration
Make sure the following entries are added to the sysctl's section of your Jail. You can find this under 'Advanced Mode' when you open your jails settings in the GUI of FreeNAS. These settings will allow FreeNAS/FreeBSD to mount a tmpfs filesystem which is a dynamic ramdisk on FreeBSD.

allow.raw_sockets=true,allow.mount.tmpfs,enforce_statfs=0,allow.mount

Restart your jail to make these settings effective.
FN 11.3 jails do not have this under the Advanced Mode, so I edited the sysctl.conf file to add the line.

Step 2: FreeNAS jail configuration
Go into your jails command line and edit the file /etc/fstab (which probably is empty) and add the following line:

tmpfs /tmp tmpfs rw,mode=777 0 0
I created a file fstab.conf and added this line. I suspect fstab isn't a .cnf file though. :(

Next step is to edit/create /etc/rc.local and add the following line:

mount /tmp
Did this.

This will automatically mount the tmpfs ramdisk we defined in fstab at each start of the jail. For some reason fstab entries are not automatically mounted inside a jail.

Restart your jail again to activate the tmpfs ramdisk. You can check if it is active and mounted with the command mount | grep tmpfs. This should show an entry for the /tmp mount inside your jail. You can issue this command in or outside of the jail.
Did this at root of FreeNAS and it showed tmps files at its root but no response at root of ZM.

Step 3: Configure your application
Last step is to make sure that your application uses /tmp for any transcoding or streaming data processing. For Zoneminder this means go into the web console and edit the values for PATH_SWAP and PATH_MAP on the Paths tab to /tmp.

Restart your application to activate the changes.
PATH_SWAP and PATH_MAP do not appear in the options of ZM 1.34.9.

I take the point about the constant writes wearing out the discs and I would like to implement this properly. Thanks for your help so far. I am about to post a thread (a poll) asking whether a thread on best practice installation and setup for ZM 1.34.9 to see if people are interested.
 

abishai

Cadet
Joined
Apr 12, 2020
Messages
8
This OP guide describes how to mount tmpfs from the jails itself. Another option is to mount it from the host. Actually, this is recommended way - the less priveleges you give to jail, the better.

You need 2 things for this (revert all the OP guide suggests)
1. Full path to jail from the host. (check zfs mount for mount points if this information is not available in web interface)
2. Find where jail fstab lies.
Code:
mount.fstab
             An fstab(5) format file containing filesystems to mount before
             creating a jail.


Usually, this file located in /etc/fstab.<jail-name>, but it's totally under control of admin and/or software used to manage jails. After the file is found (or created) write usual fstab entry and host will mount it during jail start automatically.

tmpfs <path to jail root>/tmp tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0

Random search in the forum shows that FreeNAS uses iocage.

Probably, I can't tell your more, as I don't have FreeNAS, just plain FreeBSD.

PATH_SWAP and PATH_MAP do not appear in the options of ZM 1.34.9.
This is a compile option now. See https://svnweb.freebsd.org/ports/head/multimedia/zoneminder/Makefile?view=markup#l87

I am about to post a thread (a poll) asking whether a thread on best practice installation and setup for ZM 1.34.9 to see if people are interested.
FreeBSD port has rather good guide: https://svnweb.freebsd.org/ports/head/multimedia/zoneminder/files/README.FreeBSD?view=markup
 

Frank Collins

Explorer
Joined
Apr 10, 2019
Messages
53
@abishai,
Thank you for your post, however, I am trying to do this in an already installed package in a FreeNAS 11.3 jail and your suggestions not applicable to this situation (plus a bit of it is over my head).
 

abishai

Cadet
Joined
Apr 12, 2020
Messages
8
My suggestions are applicable for existing installations. It just 1 command (iocage fstab) applied to stopped jail.
For example, https://www.ixsystems.com/community/threads/set-up-storage-location-for-zoneminder.71686/post-496367 here they add additional storage. You can just add /tmp as tmpfs backed directory.

Theoretically, something like this
1. Stop the jail
2. Delete jail /tmp contents, leaving it empty
3.
iocage fstab <jail name> -e
editor should open.
Code:
tmpfs <full path to ZM jail root from the host>/tmp tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0

4. start the jail.
5. check mount command to verify that something is mounted to jail /tmp
 

Frank Collins

Explorer
Joined
Apr 10, 2019
Messages
53
@abishai,
Thanks, but I don't follow how this works.
1. Stop the jail
Easy enough, or do you mean stop the ZoneMinder service?
2. Delete jail /tmp contents, leaving it empty
If the tmp is in the jail, how can I empty it without having access to the jail shell?
3. iocage fstab <jail name> -e
editor should open.
Code:
tmpfs <full path to ZM jail root from the host>/tmp tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0
I assume this is at the root of FreeNAS and the full path is something like "/mnt/Vol2/iocage/jails/ZoneMinder/"?
4. start the jail.
5. check mount command to verify that something is mounted to jail /tmp
How do I do this, please?
 

abishai

Cadet
Joined
Apr 12, 2020
Messages
8
Easy enough, or do you mean stop the ZoneMinder service?
Jail.

If the tmp is in the jail, how can I empty it without having access to the jail shell?
You can access root of your jail from the host.

I assume this is at the root of FreeNAS and the full path is something like "/mnt/Vol2/iocage/jails/ZoneMinder/"?
Honestly, I have no idea how things organized in FreeNSD, but if you are unsure, you can try zfs mount and look for mounts with jail name or jls command (you need to start your ZM jail for this), it prints jail names and mount points. The latter is probably the fastest way to get jail full path and it's error prone.

How do I do this, please?
Start ZM jail and issue mount |grep tmpfs from the host. You should see your tmpfs mount.
 

rvassar

Guru
Joined
May 2, 2018
Messages
971
I've been slowly working on moving my Zoneminder setup to FreeNAS. This tmpfs procedure is really key, I'll stick my neck out and suggest the plugin maintainer make it the default somehow.

I have 4 jails: Plex, MySQL, Gogs, and Zoneminder, and only 16Gb of memory. It all fits, but I'm still not convinced I'm stable. I have a couple open source repo's mirrored in Gogs, and git can really go nuts and burn memory & CPU if I start drilling down on a large project tree. I know I need a better CPU, but now isn't the time.

Some observations:

zma & zmc in my setup are running roughly 400 - 600mb of memory per camera. Since I'm worried about memory, and only have a few cameras, I set my tmpfs size to 1Gb, not the 4Gb in the example above. So far its working, but I haven't done a three camera motion test to see if I can swamp it, and I don't have a lot of soak time. I may have to bump it up.

But just to give you some idea how much of a difference this tmpfs procedure makes, see attached pic. My iocage root is on a 2-vdev mirror pool, and my camera event storage is on a separate raidz pool. I grabbed a screen shot of one disk's I/O chart across the change from tmp on the mirror pool to on tmpfs. All four disks in the pool had similar charts.
 

Attachments

  • notmpfs.PNG
    notmpfs.PNG
    86.6 KB · Views: 473

ere109

Contributor
Joined
Aug 22, 2017
Messages
190
Can someone clarify which of the above instructions to follow, to enable this feature in 11.2? I've done my best with both, but had no luck. Perhaps I'm missing something. For the original post instructions, I could't find an "advanced" section under the jail settings, but did find and click enable tmpfs, allow raw sockets and enable mount. I then added the first referenced line to the sysctl.conf, updated fstab, etc. No dice.
I tried abishai's instructions, but felt there was something missing. Either way, I'm three hours in, at CLI, and no luck.
 

Frank Collins

Explorer
Joined
Apr 10, 2019
Messages
53
I have followed the instructions above in ZM 1.34.9, as listed below;

Step 1: FreeNAS jail host configuration

allow.raw_sockets=true
allow.mount.tmpfs
enforce_statfs=0
allow.mount
Checked all the boxes and edited enforce_statfs to equal zero

Restarted the jail to make these settings effective.

Step 2: FreeNAS jail configuration
Go into your jails command line and edit the file /etc/fstab (which probably is empty) and add the following line:

tmpfs /tmp tmpfs rw,mode=777 0 0

but used the suggestion by @abishai instead;
tmpfs <full path to ZM jail root from the host>/tmp tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0

1593049931452.png



Next step is to edit/create /etc/rc.local and add the following line:

mount /tmp

This is the file;

1593049364636.png


This will automatically mount the tmpfs ramdisk we defined in fstab at each start of the jail. For some reason fstab entries are not automatically mounted inside a jail.

Restart your jail again to activate the tmpfs ramdisk. You can check if it is active and mounted with the command mount | grep tmpfs. This should show an entry for the /tmp mount inside your jail. You can issue this command in or outside of the jail.

1593049972974.png


Step 3: Configure your application
Last step is to make sure that your application uses /tmp for any transcoding or streaming data processing. For Zoneminder this means go into the web console and edit the values for PATH_SWAP and PATH_MAP on the Paths tab to /tmp.

Restart your application to activate the changes.

For ZM 1.34.9, I don't believe Step 3 is necessary because the options are not available and they now default to /tmp.

My question is; is the output from mount | grep tmpfs correct, please?
 

Attachments

  • fstab file.JPG
    fstab file.JPG
    20.8 KB · Views: 505

ere109

Contributor
Joined
Aug 22, 2017
Messages
190
I have followed the instructions above in ZM 1.34.9, as listed below;
My question is; is the output from mount | grep tmpfs correct, please?
Thank you. I thought the latter instructions negated the first. Combining them did the trick. Yes, I can see the tmpfs mount now, but not specifically on /tmp. In fact, I have three:

Code:
root@zoneminder:~ # mount | grep tmpfs
tmpfs on /etc (tmpfs, local)
tmpfs on /mnt (tmpfs, local)
tmpfs on /var (tmpfs, local)
 

ere109

Contributor
Joined
Aug 22, 2017
Messages
190
Does my tmpfs allocation seem right? I ran the code that should have given 4gb of tmpfs, but it shows 269.5gb available - on a system with 128gb total RAM. The other is a 4tb spin disc, so that makes sense. Best way to test?
 

Attachments

  • tmpfs.jpg
    tmpfs.jpg
    167.3 KB · Views: 461

ere109

Contributor
Joined
Aug 22, 2017
Messages
190
Does my tmpfs allocation seem right?

I figured it out. abishai and Frank Collins had both said to link "<full path to ZM jail root from the host>" in tmpfs, but then it wasn't mounting. This command looked like this:
Code:
tmpfs /mnt/SECURITY/iocage/jails/zoneminder/root/tmp tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0

However, when Frank shared his pictures, I saw that his fstab only said
Code:
tmpfs /tmp tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0

The light clicked on. So I changed my fstab to the latter entry, restarted, and voila, tmpfs is 4gb.

Thanks all.
 

ngalfas

Dabbler
Joined
Jul 19, 2020
Messages
17
Sure thing. Actually, this is mandatory for ZM - it kills SSD rather fast or puts rather harsh load on spindle drives. I strongly suggest to include this to FreeNAS plugin.

I recommend to limit tmpfs size with size=xxx option (check zm.mmap file size and set it accordingly) or you can ran out of memory if something bad happens.

For example, tmpfs rw,nosuid,noexec,size=4G,mode=01777 0 0 (for fstab)

as a matter of fact on ZM documentation says
It is important that you do not use a disk based filesystem for your memory mapped files as this will cause memory access to be extremely slow. ZoneMinder creates files called .zm.mmap.<monitor id> in the mapped memory filesystem.

I really think we should add this to the plugin and set a low initial value. And explain its usage and then memory size modification should be easier for everyone who is willing to adjust.
 

Roy360

Dabbler
Joined
Nov 27, 2017
Messages
28
Navigated to
Code:
\Plugins\iocage\jails\Zoneminder
in windows. (Plugins is my share for jails)

Added this line to the end of my fstab.
Code:
tmpfs /mnt/Plugins/iocage/jails/Zoneminder/root/tmp tmpfs rw,nosuid,noexec,size=3G,mode=01777 0 0


I am using 3 cameras. 5MP@15fps using ~1GB, 4MP@20fps using 720MB and 3MP@25fps using 608MB
Haven't decided what fps I'll use going forward, but 3GB should hopefully be enough space.

Next up is getting Ninja to work. Is it really neccessary to setup a second ramdisk? I'm already running pretty close to the edge. (16GB total system memory for a 12GB RAID5 + single drive for jails)

1601182708914.png
 

Zorlack01

Cadet
Joined
Nov 27, 2020
Messages
4
Hello,

I was having some issues with Zoneminder after upgrading FreeNAS to TrueNAS 12. Removed the old jail and installed Zoneminder Plugin 1.34.21_1. This is how I got tmpfs working.


Under Jail Properties I ticked allow_raw_sockets and allow_mount

Under the allow_mount_* drop down I selected allow_mount_tmpfs

Finished the install and once Zomeminder was up, went to /root/etc in the jail and created fstab with the following entry.

tmpfs /tmp tmpfs rw,nosuid,noexec,size=2G,mode=01777 0 0

In /root/etc/rc.local I added

mount /tmp

Rebooted the jail and checked tmpfs

mount | grep tmpfs

Only saw

tmpfs on /etc (tmpfs, local)
tmpfs on /mnt (tmpfs, local)
tmpfs on /var (tmpfs, local)

Checked iocage enforce_statfs Level.

iocage get -r enforce_statfs

Saw that it was set to 2. I tried adding enforce_statfs=0 to sysctl.conf and rebooting the jail. The enforce_statfs value did not change.

From the TrueNAS shell ran

iocage set enforce_statfs=0 zoneminder

Rebooted the jail and tmpfs was listed under Storage in Zonemminder.



Weird issue I saw while troubleshooting. If the jail was rebooted multiple entries for tmpfs would appear. I do not know if this is a problem or expected behavior but thought I would pass it along.

mount | grep tmpfs

tmpfs on /etc (tmpfs, local)
tmpfs on /mnt (tmpfs, local)
tmpfs on /var (tmpfs, local)
tmpfs on /mnt/Pool/iocage/jails/zoneminder/root/tmp (tmpfs, local, noexec, nosuid)
tmpfs on /mnt/Pool/iocage/jails/zoneminder/root/tmp (tmpfs, local, noexec, nosuid)
tmpfs on /mnt/Pool/iocage/jails/zoneminder/root/tmp (tmpfs, local, noexec, nosuid)

df -h

tmpfs 2.0G 8.0K 2.0G 0% /mnt/Pool/iocage/jails/zoneminder/root/tmp
tmpfs 2.0G 4.0K 2.0G 0% /mnt/Pool/iocage/jails/zoneminder/root/tmp
tmpfs 2.0G 8.0K 2.0G 0% /mnt/Pool/iocage/jails/zoneminder/root/tmp

tmpfs 2.0G 367M 1.6G 18% /mnt/Pool/iocage/jails/zoneminder/root/tmp
 

Zorlack01

Cadet
Joined
Nov 27, 2020
Messages
4
<Updated POST>

I was having some issues with ZoneMinder after upgrading FreeNAS to TrueNAS 12. Removed the old jail and installed ZoneMinder Plugin 1.34.21_1. This is how I got tmpfs working from fresh install.

Under Jail Properties I ticked allow_raw_sockets and allow_mount
Under the allow_mount_* drop down I selected allow_mount_tmpfs

There is no need to use rc.local for TMPFS under FreeBSD. Please check out- http://www.defcon1.org/html/rc-local.html

Finished the install and once ZoneMinder was ready, went to /iocage/jails/zoneminder This is where the fstab is located for the TrueNAS ZoneMinder jail 1.34.21_1NOT under /jails/zoneminder/root/etc Modify fstab with the following entry

tmpfs /iocage/jails/zoneminder/root/tmp tmpfs rw,nosuid,noexec,mode=01777 0 0

Let FreeBSD handle the tmpfs sizing first. Let me stress that again DO NOT USE THE "size=" setting without testing it both ways. I found that FreeBSD handled the allocation better on its own. ZoneMinder can use a lot of memory and if you force a hard limit it may cause issues.. Be sure to check the usage on the host when you are making these changes.

The enforce_statfs setting is next. Be careful when modifying this setting it exposes the hosts mount points depending on value.
If the value is "2" tmpfs will not mount under the jail. If you change it to "1" the jail can see only from /root down. If you change it to "0" the jail can see everything on the host. DANGER!!!!

From the TrueNAS shell run.

iocage get -r enforce_statfs

This will list the setting for all the jails. Next you will need to modify the value of enforce_statfs for the jail.

iocage set enforce_statfs=1 zoneminder

Reboot the jail and check tmpfs from the jail shell.

mount | grep tmpfs

You should see tmpfs mounted to /tmp

tmpfs on /tmp (tmpfs, local, noexec, nosuid)

ZoneMinder should now be using system memory. Check from the host shell.

top -b -o res
df -h


I found a lot of info about creating/modifying rc.local, sysctl.conf and using /root/etc for the fstab. This is what worked for me under FreeBSD 12.2-RC3, TrueNAS-12.0-RELEASE I hope it helps someone.
 
Top