Great question! Your analysis is correct; I'll attempt to highlight the differences in detail in the section below.
TLDR;
The choice of jail type determines: the size on disk, the ease of updating/upgrading, and the suitability to keep it around across releases.
The typical approach of most tutorials and scripts in this forum is to keep your "data" in a dataset outside of the iocage jail dataset. With your data separate from the jail, it's trivial to:
- stop existing jail
- create a new jail based on the latest release and/or patches
- (re)mount the data in new jail
- nuke existing jail if everything works as desired in new jail
If you stick to this recommendation, you'll avoid a lot of heartburn and, unless you manage a very large number of jails, the jail type won't have a huge long term impact.
The different jail types
The following is based on FreeNAS 11.3-U1. Jails/plugins created with older releases of FreeNAS (iocage) may behave differently.
There are 4 types of jails in iocage, the FreeNAS UI only exposes "Clone jails" and "Base jails". Note that there are also "template" - and "empty" jails, but these aren't considered here.
- Clone jails
- cloned from a snapshot of the "release"
- this is the default when you create a new jail (FreeNAS 11.3-U1)
- is the fastest to create
- take up smallest amount of space - initially
- Due to the clone process, it keeps dependency on the release. As a result, you won't be be to delete the release, unless you delete all jails that were cloned from it
- Is patched (read
iocage update
) individually
- Intended for jails with relative short lifespan. Instead of trying to "upgrade", rather plan on nuking and rebuild. Once the clone diverts too far from the release, the space advantage gets lost.
- Thick jails
- created as a copy of the "release"
- Takes up most space initially
- Completely independent, so use this type for jails you want to keep around as pets and upgrade across FreeNAS releases.
- Base jails
- base jails are created as thick jails
- removes certain userland directories and nullfs mounts them back from the "release"
- this is the default when you create a plugin (FreeNAS 11.3-U1)
- smaller then Thick jail but larger then Clone jail
- easy to patch in bulk; when you
iocage update
the jail (or iocage fetch
the "release" again), it will patch all base jails, based on that release. If you manage a large amount of jails and you want to patch them as on, this would be a good choice
- Clone_base jails
- This is what it sounds like: Initially cloned from a snapshot of the "release"
- removes same base directories as Base jail and nullfs mounts them back from the "release"
- this used to be default for plugins in very early iocage versions
- Is like a Clone jail that can be patched in bulk.
See how deep the rabbit hole goes
Let's create a few jails and compare the differences. I ran this in a clean FreeNAS 11.3-U1 install, in case anybody want to reproduce this exercise.
Start with downloading the release, 11.3-RELEASE. I'm using the
--noupdate
option, because I want update to the latest patch level in a separate step.
Code:
root@freenas-113U1[~]# iocage activate tank
ZFS pool 'tank' successfully activated.
root@freenas-113U1[~]# iocage fetch -r 11.3-RELEASE --noupdate
Creating tank/iocage
Creating tank/iocage/download
Creating tank/iocage/images
Creating tank/iocage/jails
Creating tank/iocage/log
Creating tank/iocage/releases
Creating tank/iocage/templates
Default configuration missing, creating one
Fetching: 11.3-RELEASE
Downloading: MANIFEST [####################] 100%
Downloading: base.txz [####################] 100%
Downloading: lib32.txz [####################] 100%
Downloading: src.txz [####################] 100%
Downloading: doc.txz [####################] 100%
Extracting: base.txz...
Extracting: lib32.txz...
Extracting: src.txz...
Extracting: doc.txz...
Create some jails
Code:
root@freenas-113U1[~]# iocage create help
Usage: [OPTIONS] [PROPS]...
Create a jail.
Options:
-c, --count TEXT Designate a number of jails to create. Jails are
numbered sequentially.
-C, --thickconfig Do not use inheritable configuration with jails
-r, --release TEXT Specify the RELEASE to use for the new jail.
-t, --template TEXT Specify the template to use for the new jail instead
of a RELEASE.
-p, --pkglist TEXT Specify a JSON file which manages the installation of
each package in the newly created jail.
-n, --name TEXT Provide a specific name instead of an UUID for this
jail.
-u, --uuid TEXT Provide a specific UUID for this jail.
-S, --proxy TEXT Provide proxy to use for creating jail
-b, --basejail Set the new jail type to a basejail. Basejails are
thick jails (unless specified) that mount the
specified RELEASE directories as nullfs mounts over
the jail's directories.
-B, --clone_basejail Set the new jail type to a clonetype basejail.
Basejails mount the specified RELEASE directories as
nullfs mounts over the jail's directories.
-T, --thickjail Set the new jail type to a thickjail. Thickjails are
copied (not cloned) from specified RELEASE.
-e, --empty Create an empty jail used for unsupported or custom
jails.
-s, --short Use a short UUID of 8 characters instead of the
default 36.
--help Show this message and exit.
root@freenas-113U1[~]# iocage create -n clone -r 11.3-RELEASE
clone successfully created!
root@freenas-113U1[~]# iocage create -n thick -T -r 11.3-RELEASE
thick successfully created!
root@freenas-113U1[~]# iocage create -n base-1 -b -r 11.3-RELEASE
base-1 successfully created!
root@freenas-113U1[~]# iocage create -n base-2 -b -r 11.3-RELEASE
base-2 successfully created!
root@freenas-113U1[~]# iocage create -n clonebase-1 -B -r 11.3-RELEASE
clonebase-1 successfully created!
root@freenas-113U1[~]# iocage create -n clonebase-2 -B -r 11.3-RELEASE
clonebase-2 successfully created!
This is the summary of the jails we've just created. Both the Base jails and Base-clone jails are flagged as BASEJAIL.
Code:
root@freenas-113U1[~]# iocage list -l
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| JID | NAME | BOOT | STATE | TYPE | RELEASE | IP4 | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+==============+=====+=====+==========+==========+
| - | base-1 | off | down | jail | 11.3-RELEASE | - | - | - | yes |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| - | base-2 | off | down | jail | 11.3-RELEASE | - | - | - | yes |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| - | clone | off | down | jail | 11.3-RELEASE | - | - | - | no |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| - | clonebase-1 | off | down | jail | 11.3-RELEASE | - | - | - | yes |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| - | clonebase-2 | off | down | jail | 11.3-RELEASE | - | - | - | yes |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| - | thick | off | down | jail | 11.3-RELEASE | - | - | - | no |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
A quick look at the initial space shows that Thick jails > Base jail > Clone-base > Clone jail. The size of a Clone-Base > Clone jails, is because deleting the base directories from the snapshot still takes up space.
Code:
root@freenas-113U1[~]# iocage df
+-------------+-------+------+------+-------+-------+
| NAME | CRT | RES | QTA | USE | AVA |
+=============+=======+======+======+=======+=======+
| base-1 | 2.51x | none | none | 676M | 9.48G |
+-------------+-------+------+------+-------+-------+
| base-2 | 2.51x | none | none | 676M | 9.48G |
+-------------+-------+------+------+-------+-------+
| clone | 1.00x | none | none | 292K | 9.48G |
+-------------+-------+------+------+-------+-------+
| clonebase-1 | 1.00x | none | none | 432K | 9.48G |
+-------------+-------+------+------+-------+-------+
| clonebase-2 | 1.00x | none | none | 432K | 9.48G |
+-------------+-------+------+------+-------+-------+
| thick | 2.38x | none | none | 1012M | 9.48G |
+-------------+-------+------+------+-------+-------+
The underlying datasets show that the Clone jails and and the Clone-base jails originate from a snapshot of the "release".
Code:
root@freenas-113U1[~]# zfs list -t all -o name,origin,used,refer,mountpoint -r tank/iocage
NAME ORIGIN USED REFER MOUNTPOINT
tank/iocage - 3.59G 5.52M /mnt/tank/iocage
tank/iocage/download - 289M 88K /mnt/tank/iocage/download
tank/iocage/download/11.3-RELEASE - 289M 289M /mnt/tank/iocage/download/11.3-RELEASE
tank/iocage/images - 88K 88K /mnt/tank/iocage/images
tank/iocage/jails - 2.31G 96K /mnt/tank/iocage/jails
tank/iocage/jails/base-1 - 676M 96K /mnt/tank/iocage/jails/base-1
tank/iocage/jails/base-1/root - 675M 675M /mnt/tank/iocage/jails/base-1/root
tank/iocage/jails/base-2 - 676M 96K /mnt/tank/iocage/jails/base-2
tank/iocage/jails/base-2/root - 675M 675M /mnt/tank/iocage/jails/base-2/root
tank/iocage/jails/clone - 292K 92K /mnt/tank/iocage/jails/clone
tank/iocage/jails/clone/root tank/iocage/releases/11.3-RELEASE/root@clone 200K 1012M /mnt/tank/iocage/jails/clone/root
tank/iocage/jails/clonebase-1 - 432K 96K /mnt/tank/iocage/jails/clonebase-1
tank/iocage/jails/clonebase-1/root tank/iocage/releases/11.3-RELEASE/root@clonebase-1 336K 675M /mnt/tank/iocage/jails/clonebase-1/root
tank/iocage/jails/clonebase-2 - 432K 96K /mnt/tank/iocage/jails/clonebase-2
tank/iocage/jails/clonebase-2/root tank/iocage/releases/11.3-RELEASE/root@clonebase-2 336K 675M /mnt/tank/iocage/jails/clonebase-2/root
tank/iocage/jails/thick - 1012M 92K /mnt/tank/iocage/jails/thick
tank/iocage/jails/thick/root - 1012M 1012M /mnt/tank/iocage/jails/thick/root
tank/iocage/log - 88K 88K /mnt/tank/iocage/log
tank/iocage/releases - 1013M 88K /mnt/tank/iocage/releases
tank/iocage/releases/11.3-RELEASE - 1013M 88K /mnt/tank/iocage/releases/11.3-RELEASE
tank/iocage/releases/11.3-RELEASE/root - 1012M 1012M /mnt/tank/iocage/releases/11.3-RELEASE/root
tank/iocage/releases/11.3-RELEASE/root@clone - 152K 1012M -
tank/iocage/releases/11.3-RELEASE/root@clonebase-1 - 152K 1012M -
tank/iocage/releases/11.3-RELEASE/root@clonebase-2 - 144K 1012M -
tank/iocage/templates
Finally, let's update one of the Base Jails and see what happens
Code:
root@freenas-113U1[~]# iocage update base-1
Snapshot: tank/iocage/jails/base-1@ioc_update_11.3-RELEASE_2020-02-27_23-39-19 created.
Updating jail...
* Updating 11.3-RELEASE to the latest patch level...
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 11.3-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 115 patches.....10....20....30....40....50....60....70....80....90....100....110.. done.
Applying patches... done.
The following files will be updated as part of updating to
11.3-RELEASE-p6:
/bin/freebsd-version
/boot/loader
**** snip *****
/usr/src/usr.sbin/bhyve/pci_xhci.c
Installing updates... done.
release: 11.3-RELEASE -> 11.3-RELEASE-p6
release: 11.3-RELEASE -> 11.3-RELEASE-p6
release: 11.3-RELEASE -> 11.3-RELEASE-p6
release: 11.3-RELEASE -> 11.3-RELEASE-p6
Updated jail successfully.
base-1 updates have been applied successfully.
As a result all Base jails and Base-clone jails that are created of 11.3-RELEASE have been updated to the latest patch-release 11.3-RELEASE-p6, without consuming more space.
Code:
root@freenas-113U1[~]# iocage list -l
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| JID | NAME | BOOT | STATE | TYPE | RELEASE | IP4 | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+=================+=====+=====+==========+==========+
| - | base-1 | off | down | jail | 11.3-RELEASE-p6 | - | - | - | yes |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| - | base-2 | off | down | jail | 11.3-RELEASE-p6 | - | - | - | yes |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| - | clone | off | down | jail | 11.3-RELEASE | - | - | - | no |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| - | clonebase-1 | off | down | jail | 11.3-RELEASE-p6 | - | - | - | yes |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| - | clonebase-2 | off | down | jail | 11.3-RELEASE-p6 | - | - | - | yes |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| - | thick | off | down | jail | 11.3-RELEASE | - | - | - | no |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
root@freenas-113U1[~]# iocage df
+-------------+-------+------+------+-------+-------+
| NAME | CRT | RES | QTA | USE | AVA |
+=============+=======+======+======+=======+=======+
| base-1 | 2.51x | none | none | 676M | 9.41G |
+-------------+-------+------+------+-------+-------+
| base-2 | 2.51x | none | none | 676M | 9.41G |
+-------------+-------+------+------+-------+-------+
| clone | 1.00x | none | none | 292K | 9.41G |
+-------------+-------+------+------+-------+-------+
| clonebase-1 | 1.00x | none | none | 432K | 9.41G |
+-------------+-------+------+------+-------+-------+
| clonebase-2 | 1.00x | none | none | 432K | 9.41G |
+-------------+-------+------+------+-------+-------+
| thick | 2.38x | none | none | 1012M | 9.41G |
+-------------+-------+------+------+-------+-------+
root@freenas-113U1[~]# zfs list -t all -o name,origin,used,refer,mountpoint -r tank/iocage
NAME ORIGIN USED REFER MOUNTPOINT
tank/iocage - 3.66G 5.52M /mnt/tank/iocage
tank/iocage/download - 289M 88K /mnt/tank/iocage/download
tank/iocage/download/11.3-RELEASE - 289M 289M /mnt/tank/iocage/download/11.3-RELEASE
tank/iocage/images - 88K 88K /mnt/tank/iocage/images
tank/iocage/jails - 2.31G 96K /mnt/tank/iocage/jails
tank/iocage/jails/base-1 - 676M 96K /mnt/tank/iocage/jails/base-1
tank/iocage/jails/base-1@ioc_update_11.3-RELEASE_2020-02-27_23-39-19 - 60K 96K -
tank/iocage/jails/base-1/root - 676M 675M /mnt/tank/iocage/jails/base-1/root
tank/iocage/jails/base-1/root@ioc_update_11.3-RELEASE_2020-02-27_23-39-19 - 252K 675M -
tank/iocage/jails/base-2 - 676M 96K /mnt/tank/iocage/jails/base-2
tank/iocage/jails/base-2/root - 675M 675M /mnt/tank/iocage/jails/base-2/root
tank/iocage/jails/clone - 292K 92K /mnt/tank/iocage/jails/clone
tank/iocage/jails/clone/root tank/iocage/releases/11.3-RELEASE/root@clone 200K 1012M /mnt/tank/iocage/jails/clone/root
tank/iocage/jails/clonebase-1 - 432K 96K /mnt/tank/iocage/jails/clonebase-1
tank/iocage/jails/clonebase-1/root tank/iocage/releases/11.3-RELEASE/root@clonebase-1 336K 675M /mnt/tank/iocage/jails/clonebase-1/root
tank/iocage/jails/clonebase-2 - 432K 96K /mnt/tank/iocage/jails/clonebase-2
tank/iocage/jails/clonebase-2/root tank/iocage/releases/11.3-RELEASE/root@clonebase-2 336K 675M /mnt/tank/iocage/jails/clonebase-2/root
tank/iocage/jails/thick - 1012M 92K /mnt/tank/iocage/jails/thick
tank/iocage/jails/thick/root - 1012M 1012M /mnt/tank/iocage/jails/thick/root
tank/iocage/log - 92K 92K /mnt/tank/iocage/log
tank/iocage/releases - 1.06G 88K /mnt/tank/iocage/releases
tank/iocage/releases/11.3-RELEASE - 1.06G 88K /mnt/tank/iocage/releases/11.3-RELEASE
tank/iocage/releases/11.3-RELEASE/root - 1.06G 1.03G /mnt/tank/iocage/releases/11.3-RELEASE/root
tank/iocage/releases/11.3-RELEASE/root@clone - 152K 1012M -
tank/iocage/releases/11.3-RELEASE/root@clonebase-1 - 152K 1012M -
tank/iocage/releases/11.3-RELEASE/root@clonebase-2 - 152K 1012M -
tank/iocage/templates - 88K 88K /mnt/tank/iocage/templates
Long post, but I hope this was helpful and helps people decide what type of jail to use.