Put Jail /tmp and /var/tmp in memory?

[Milhouse]

Is it possible, within an 8.3.0-RC1 Jail, to have /tmp (and /var/tmp) mounted in memory (ie. tmpfs)?

MySQL will, by default, create temporary files in /var/tmp- even when performing read-only queries (ie. SELECT). This is causing excessive disk writes to ZFS (which is where /var/tmp lives inside a Jail), and prevents the disks from spinning down (assuming ataidle worked, which isn't actually the case in 8.3.0-RC1, but I digress).

Unfortunately, the Jail seems to have no access to volatile storage such as tmpfs, so activity that wouldn't normally trigger disk activity outside of a jail, do so inside of a jail. The only obvious solution appears to be to lump the entire Jail on to a USB memory stick, but if possible I'd rather shift /var/tmp and /tmp into RAM.

Anyone got any ideas? :)

 

[paleoN]

Installing the jail itself to the Flash drive will cause writes every 5 minutes. See ticket #1681.

AFAICT, the jail has a 'single' mount for / set to the plugins jail path. There are also a couple of nullfs mounts, in the base system, for the plugins & pbi directories in the jail path to the archive path.

Inside the jail /var is a normal directory under / which is on whatever drive the jail path is on.

Given that nullfs mounts work with the jail, you could always mount something else, e.g. another md drive, over the jails /var directory among other things.

I haven't tried it or even bothered to think about it any further.

 

[Milhouse]

Hi, yes - I identified several of the cron jobs as part of the problem, the default cron setup doesn't seem particularly appropriate for the jail.

I've just created a memory-backed disk, and mounted it so that it can be written to within the jail, and that's solved the MySQL excessive write problem.

Next step is to work out how I can create the memory backed disk and mount it automatically with each boot...

 

[Milhouse]

Right, here's a patch to add support for a memory backed /var/tmp in 8.3.0-RC1. Its rather quick & dirty, but it works for me.

Apply the attached patch to /conf/base/etc/rc.d/ix-jail:

Code:

mount -uw /
cd /conf/base/etc/rc.d/
patch ix-jail /path/to/ix-jail.patch
mount -ur /

Enable the memory backed modification by specifying a value for gaol_tmp_fs_size in /conf/base/etc/rc.conf (or /etc/rc.conf, if only testing).

Example: gaol_tmp_fs_size="64m".

The modified ix-jail will create a memory backed device (always unit 3, ie. /dev/md3) of the specified size, that will be mounted on /var/tmp for the relevant plugin/jail. This device will be unmounted and then destroyed when the plugin is stopped. Obviously, this patch assumes there will only ever be one plugin/jail per server... not sure if multiple plugins are currently supported, but this patch will need additional work to support multiple memory backed devices.

Note that I've had to use "gaol_" as an identifier because ix-jail strips out from rc.conf every entry that contains the prefix "jail_". As an aside, ix-jail should probably be changed to use the regex "^jail_" so that it removes only the entries that begin with "jail_".

 

[paleoN]

Right, here's a patch to add support for a memory backed /var/tmp in 8.3.0-RC1. Its rather quick & dirty, but it works for me.

Glad the idea seems to pan out. :) I will have to play with this myself some. Though I won't likely get a chance to do so all that soon.

 

[Milhouse]

Patch updated (see post #4) with support for dynamic memory disk unit allocation and multiple jails per server.

Changelog:

• Jail-specific gaol_tmp_fs_size added: Optionally specify "global" gaol_tmp_fs_size (as before), or gaol_<name>_tmp_fs_size (if present, latter supersedes the former)
• Dynamically determines memory disk unit rather than assume md3 (so potentially each jail can have it's own memory backed device)
• Changed memory back disk to use swap instead of malloc

Eg., in rc.conf:

gaol_tmp_fs_size=64M
gaol_myjail1_tmp_fs_size=32M
gaol_myjail2_tmp_fs_size=48M

If you had 3 jails, myjail1 would be allocated 32M, myjail2 48M and jail #3 (and any other jails) 64M.

If you remove the global gaol_tmp_fs_size setting, then jail #3 would not be allocated a memory backed device at all.

 

[Milhouse]

Patch updated (see post #4), now with support for /var/log and additional directories. Tested with 8.3.0-RELEASE.

Due to continued logging activity within the Jail preventing disk spin-down, I've decided to cut my losses and include /var/log within the memory backed storage area.

To support multiple directories (tmp, log etc.), the memory backed storage will have its own mount point /var/mdisk, and the relevant directories automatically created within by searching for and locating symbolic links in /var that have their content defined as being within /var/mdisk.

If you want to relocate any other directories from /var to memory backed disk, simply create a symbolic link to /var/mdisk. If you don't want log or tmp to be in memory backed storage, keep them as physical directories rather than symbolic links.

The easiest way to create the required symbolic links is to stop the plugin, cd directly to the /var directory in the jail (ie. cd /mnt/<tank>/jail/software/var) and:

Code:

rm -fr log tmp
mkdir mdisk
ln -s ./mdisk/log log
ln -s ./mdisk/tmp tmp

As you start the plugin, the memory backed storage will be mounted on /var/mdisk, and within /var/mdisk the tmp and log directories will be created.

If, you also wanted to relocate /var/run into memory backed storage: stop the plugin, remove the /var/run physical directory, create a symbolic link for /var/run to ./mdisk/run (or if you prefer, /var/mdisk/run or even just mdisk/run), and start the plugin.

Changelog:

• Use own mount point, /var/mdisk
• Support multiple directories based on symbolic links to /var/mdisk that are located in /var

 

[joeschmuck]

Good posting. I'll give it a shot tomorrow.

EDIT: I got this to work fine in a VM but not on the real machine. I will try it again when my head is clear and the Christmas lights are on the house.

 

[vvanderm]

Any luck with testing this joeschmuck? Thoughts as to which version of freenas this would make it into?

Thanks!

 

[joeschmuck]

Any luck with testing this joeschmuck? Thoughts as to which version of freenas this would make it into?

Thanks!

I tested it out and it worked from what I recalled but I didn't leave it installed, only because I don't have any issue with my NAS drives spinning down.

As for it becoming part of FreeNAS, I haven't heard that before so I don't know if it ever will be part or not. Also keep in mind that the patch is only for the FreeNAS version it was built for, you could probably create one for a future version as well. But if this is something you would like to see implemented into FreeNAS, you should submit a ticket requesting the modification but it should be an optional modification.

 

[Milhouse]

Patch still works with 8.3.0-RELEASE-p1.

Very unlikely this would be accepted officially, and frankly I don't even bother asking for such stuff now, I've decided the only way to fix stuff is do it yourself (hence patches like mine).

 

[joeschmuck]

And your patches are welcomed.

 

[fracai]

Could this also be implemented as a startup script? In other words, is there anything that can't be run from within the jail? That would seem more stable than patching each FreeNAS release, if this isn't added to a proper release that is.

 

[Milhouse]

No, you need to patch FreeNAS.

 

[fracai]

Could you elaborate on that?

I've seen people getting tmpfs running in FreeBSD jails. Are the tools not available in the FreeNAS jail? Or are the actions (mounting a filesystem) simply not permitted in the jail?

 

[Milhouse]

Could you elaborate on that?

I've seen people getting tmpfs running in FreeBSD jails. Are the tools not available in the FreeNAS jail? Or are the actions (mounting a filesystem) simply not permitted in the jail?

Look at the patch to see what it does. The way in which the jail is created has to be modified to support the memory backed storage, it's not about creating tempfs mounts within the jail as tempfs is created outside of the jail so that the jail will use it automatically without modifying any addons running within the jail. Possibly you could modify addons to use tempfs on an add-hoc basis but you'll be modifying individual addons each time rather than doing it once at the "system" level.

I will backtrack slightly and say that yes, you could possibly apply the functionality discussed in this thread by not patching the system but instead making the changes at startup, but you would need a startup script that shuts down all jails, replaces the jail creation scripts with the memory-backed version and then restarts the jails. If you're going to do that then you might as well patch the system...

 

[fracai]

I did look at the script and it didn't look like anything that wouldn't be possible without patching. We're probably talking past each other though. I'd be fine with one tmpfs mount in the only jail. I don't need one per plugin. So I'll probably play around with it tonight if I get the chance to see if I can just put together an rc script that would run when the jail starts up.

 

[Milhouse]

OK, I see what you are getting at, let us know how you get on.

 

[smufu]

I tested the fix with the following version: FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty), and seems like the fix isn't working.
I can see the following message in the boot-log:

Code:

Apr 17 21:41:17 FreeNAS ix-jail: Created memory backed disk /dev/md3 with size 64M

But the disk don't spin down if the plugin-service is running.
Can anybody confirm this?
Can I check by myself if the fix is working as expected?

Regfards,
smufu

 

[Milhouse]

Can I check by myself if the fix is working as expected?

Sure, write to the directories you think are mounted from mdisk (/tmp, /var/tmp, /var/log etc.) and see if the act of writing to these directories causes your disks to spin up - if not, then the "fix" is working as planned.

If however your disks continue to spin up even with this fix in place, then you need to track down what process is reading/writing to your disks, and where, then see if there is a way to prevent it.