iocage jail type: base jail vs clone. Which to choose?

GeekGoneOld

Dabbler
Joined
Aug 1, 2015
Messages
48
Trying to figure out when I would use which kind of jail (base vs clone). It all started when I was trying to clean up my snapshots and found one named for my one and only jail.

I read the manual which explains each. It seems a base jail has a mount point of the base OS whereas clone is a clone of (a snapshot of) the base jail. The default is clone. I'm trying to figure out when I would use each. Below I have tried to list what I think are true about each:

Basejail:
-is the "mount" of the base persistent or is the base mounted once then copied into the jail? The manual implies persistent.
-if it is persistent, is it r/o. If it is r/w, altering it in one jail (e.g. Update) would affect all other base jails of the same base?
-Again, if it is permanent, how does upgrade work? Does it change the mount to the new base?
-if it is not permanent (i.e. copies the base to the jail) it seems that each base jail is an independent clean slate.

Clone:
-it seems it creates a snapshot of the base and names the snapshot specific to the jail name then creates a clone. So is there one snapshot per clone jail? Why would it not use a single snapshot (multiple clones) for all jails of that base?
-Also (snapshot question), if there are multiple snapshots of a single base, does each snapshot take the same amount of space given that the base isn't changing? If there is only one snapshot, would it take the same amount of space as the base?
-the manual says it is linked to the base even if upgraded. This makes sense since it is a clone of the original base.
-is there any need to keep the snapshot for any given jail or can I simply promote the clone then delete the snapshot

I guess I see this as "why is there a choice"? Actually, to me, it would seem that basejail would be the preferred in most cases. My only thought about clonejail would be less space if they were all clones of the same snapshot of the base. If not, I'm stumped to know the advantage of a clone jail!

Finally, can I promote the clone then delete the snapshot? Would this effectively make it a basejail?

Keith
 
Joined
Jul 10, 2016
Messages
521
Great question! Your analysis is correct; I'll attempt to highlight the differences in detail in the section below.

TLDR;
The choice of jail type determines: the size on disk, the ease of updating/upgrading, and the suitability to keep it around across releases.

The typical approach of most tutorials and scripts in this forum is to keep your "data" in a dataset outside of the iocage jail dataset. With your data separate from the jail, it's trivial to:
  1. stop existing jail
  2. create a new jail based on the latest release and/or patches
  3. (re)mount the data in new jail
  4. nuke existing jail if everything works as desired in new jail
If you stick to this recommendation, you'll avoid a lot of heartburn and, unless you manage a very large number of jails, the jail type won't have a huge long term impact.

The different jail types
The following is based on FreeNAS 11.3-U1. Jails/plugins created with older releases of FreeNAS (iocage) may behave differently.

There are 4 types of jails in iocage, the FreeNAS UI only exposes "Clone jails" and "Base jails". Note that there are also "template" - and "empty" jails, but these aren't considered here.

  1. Clone jails
    • cloned from a snapshot of the "release"
    • this is the default when you create a new jail (FreeNAS 11.3-U1)
    • is the fastest to create
    • take up smallest amount of space - initially
    • Due to the clone process, it keeps dependency on the release. As a result, you won't be be to delete the release, unless you delete all jails that were cloned from it
    • Is patched (read iocage update) individually
    • Intended for jails with relative short lifespan. Instead of trying to "upgrade", rather plan on nuking and rebuild. Once the clone diverts too far from the release, the space advantage gets lost.
  2. Thick jails
    • created as a copy of the "release"
    • Takes up most space initially
    • Completely independent, so use this type for jails you want to keep around as pets and upgrade across FreeNAS releases.
  3. Base jails
    • base jails are created as thick jails
    • removes certain userland directories and nullfs mounts them back from the "release"
    • this is the default when you create a plugin (FreeNAS 11.3-U1)
    • smaller then Thick jail but larger then Clone jail
    • easy to patch in bulk; when you iocage update the jail (or iocage fetch the "release" again), it will patch all base jails, based on that release. If you manage a large amount of jails and you want to patch them as on, this would be a good choice
  4. Clone_base jails
    • This is what it sounds like: Initially cloned from a snapshot of the "release"
    • removes same base directories as Base jail and nullfs mounts them back from the "release"
    • this used to be default for plugins in very early iocage versions
    • Is like a Clone jail that can be patched in bulk.

See how deep the rabbit hole goes
Let's create a few jails and compare the differences. I ran this in a clean FreeNAS 11.3-U1 install, in case anybody want to reproduce this exercise.

Start with downloading the release, 11.3-RELEASE. I'm using the --noupdate option, because I want update to the latest patch level in a separate step.
Code:
root@freenas-113U1[~]# iocage activate tank
ZFS pool 'tank' successfully activated.
root@freenas-113U1[~]# iocage fetch -r 11.3-RELEASE --noupdate
Creating tank/iocage
Creating tank/iocage/download
Creating tank/iocage/images
Creating tank/iocage/jails
Creating tank/iocage/log
Creating tank/iocage/releases
Creating tank/iocage/templates
Default configuration missing, creating one
Fetching: 11.3-RELEASE

Downloading: MANIFEST [####################] 100%
Downloading: base.txz [####################] 100%
Downloading: lib32.txz [####################] 100%
Downloading: src.txz [####################] 100%
Downloading: doc.txz [####################] 100%
Extracting: base.txz...
Extracting: lib32.txz...
Extracting: src.txz...
Extracting: doc.txz...

Create some jails
Code:
root@freenas-113U1[~]# iocage create help
Usage:  [OPTIONS] [PROPS]...

  Create a jail.

Options:
  -c, --count TEXT      Designate a number of jails to create. Jails are
                        numbered sequentially.
  -C, --thickconfig     Do not use inheritable configuration with jails
  -r, --release TEXT    Specify the RELEASE to use for the new jail.
  -t, --template TEXT   Specify the template to use for the new jail instead
                        of a RELEASE.
  -p, --pkglist TEXT    Specify a JSON file which manages the installation of
                        each package in the newly created jail.
  -n, --name TEXT       Provide a specific name instead of an UUID for this
                        jail.
  -u, --uuid TEXT       Provide a specific UUID for this jail.
  -S, --proxy TEXT      Provide proxy to use for creating jail
  -b, --basejail        Set the new jail type to a basejail. Basejails are
                        thick jails (unless specified) that mount the
                        specified RELEASE directories as nullfs mounts over
                        the jail's directories.
  -B, --clone_basejail  Set the new jail type to a clonetype basejail.
                        Basejails mount the specified RELEASE directories as
                        nullfs mounts over the jail's directories.
  -T, --thickjail       Set the new jail type to a thickjail. Thickjails are
                        copied (not cloned) from specified RELEASE.
  -e, --empty           Create an empty jail used for unsupported or custom
                        jails.
  -s, --short           Use a short UUID of 8 characters instead of the
                        default 36.
  --help                Show this message and exit.

root@freenas-113U1[~]# iocage create -n clone -r 11.3-RELEASE
clone successfully created!
root@freenas-113U1[~]# iocage create -n thick -T -r 11.3-RELEASE
thick successfully created!
root@freenas-113U1[~]# iocage create -n base-1 -b -r 11.3-RELEASE
base-1 successfully created!
root@freenas-113U1[~]# iocage create -n base-2 -b -r 11.3-RELEASE
base-2 successfully created!
root@freenas-113U1[~]# iocage create -n clonebase-1 -B -r 11.3-RELEASE
clonebase-1 successfully created!
root@freenas-113U1[~]# iocage create -n clonebase-2 -B -r 11.3-RELEASE
clonebase-2 successfully created!

This is the summary of the jails we've just created. Both the Base jails and Base-clone jails are flagged as BASEJAIL.
Code:
root@freenas-113U1[~]# iocage list -l
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| JID |    NAME     | BOOT | STATE | TYPE |   RELEASE    | IP4 | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+==============+=====+=====+==========+==========+
| -   | base-1      | off  | down  | jail | 11.3-RELEASE | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| -   | base-2      | off  | down  | jail | 11.3-RELEASE | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| -   | clone       | off  | down  | jail | 11.3-RELEASE | -   | -   | -        | no       |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| -   | clonebase-1 | off  | down  | jail | 11.3-RELEASE | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| -   | clonebase-2 | off  | down  | jail | 11.3-RELEASE | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+
| -   | thick       | off  | down  | jail | 11.3-RELEASE | -   | -   | -        | no       |
+-----+-------------+------+-------+------+--------------+-----+-----+----------+----------+

A quick look at the initial space shows that Thick jails > Base jail > Clone-base > Clone jail. The size of a Clone-Base > Clone jails, is because deleting the base directories from the snapshot still takes up space.
Code:
root@freenas-113U1[~]# iocage df
+-------------+-------+------+------+-------+-------+
|    NAME     |  CRT  | RES  | QTA  |  USE  |  AVA  |
+=============+=======+======+======+=======+=======+
| base-1      | 2.51x | none | none | 676M  | 9.48G |
+-------------+-------+------+------+-------+-------+
| base-2      | 2.51x | none | none | 676M  | 9.48G |
+-------------+-------+------+------+-------+-------+
| clone       | 1.00x | none | none | 292K  | 9.48G |
+-------------+-------+------+------+-------+-------+
| clonebase-1 | 1.00x | none | none | 432K  | 9.48G |
+-------------+-------+------+------+-------+-------+
| clonebase-2 | 1.00x | none | none | 432K  | 9.48G |
+-------------+-------+------+------+-------+-------+
| thick       | 2.38x | none | none | 1012M | 9.48G |
+-------------+-------+------+------+-------+-------+

The underlying datasets show that the Clone jails and and the Clone-base jails originate from a snapshot of the "release".
Code:
root@freenas-113U1[~]# zfs list -t all -o name,origin,used,refer,mountpoint -r tank/iocage
NAME                                                ORIGIN                                               USED  REFER  MOUNTPOINT
tank/iocage                                         -                                                   3.59G  5.52M  /mnt/tank/iocage
tank/iocage/download                                -                                                    289M    88K  /mnt/tank/iocage/download
tank/iocage/download/11.3-RELEASE                   -                                                    289M   289M  /mnt/tank/iocage/download/11.3-RELEASE
tank/iocage/images                                  -                                                     88K    88K  /mnt/tank/iocage/images
tank/iocage/jails                                   -                                                   2.31G    96K  /mnt/tank/iocage/jails
tank/iocage/jails/base-1                            -                                                    676M    96K  /mnt/tank/iocage/jails/base-1
tank/iocage/jails/base-1/root                       -                                                    675M   675M  /mnt/tank/iocage/jails/base-1/root
tank/iocage/jails/base-2                            -                                                    676M    96K  /mnt/tank/iocage/jails/base-2
tank/iocage/jails/base-2/root                       -                                                    675M   675M  /mnt/tank/iocage/jails/base-2/root
tank/iocage/jails/clone                             -                                                    292K    92K  /mnt/tank/iocage/jails/clone
tank/iocage/jails/clone/root                        tank/iocage/releases/11.3-RELEASE/root@clone         200K  1012M  /mnt/tank/iocage/jails/clone/root
tank/iocage/jails/clonebase-1                       -                                                    432K    96K  /mnt/tank/iocage/jails/clonebase-1
tank/iocage/jails/clonebase-1/root                  tank/iocage/releases/11.3-RELEASE/root@clonebase-1   336K   675M  /mnt/tank/iocage/jails/clonebase-1/root
tank/iocage/jails/clonebase-2                       -                                                    432K    96K  /mnt/tank/iocage/jails/clonebase-2
tank/iocage/jails/clonebase-2/root                  tank/iocage/releases/11.3-RELEASE/root@clonebase-2   336K   675M  /mnt/tank/iocage/jails/clonebase-2/root
tank/iocage/jails/thick                             -                                                   1012M    92K  /mnt/tank/iocage/jails/thick
tank/iocage/jails/thick/root                        -                                                   1012M  1012M  /mnt/tank/iocage/jails/thick/root
tank/iocage/log                                     -                                                     88K    88K  /mnt/tank/iocage/log
tank/iocage/releases                                -                                                   1013M    88K  /mnt/tank/iocage/releases
tank/iocage/releases/11.3-RELEASE                   -                                                   1013M    88K  /mnt/tank/iocage/releases/11.3-RELEASE
tank/iocage/releases/11.3-RELEASE/root              -                                                   1012M  1012M  /mnt/tank/iocage/releases/11.3-RELEASE/root
tank/iocage/releases/11.3-RELEASE/root@clone        -                                                    152K  1012M  -
tank/iocage/releases/11.3-RELEASE/root@clonebase-1  -                                                    152K  1012M  -
tank/iocage/releases/11.3-RELEASE/root@clonebase-2  -                                                    144K  1012M  -
tank/iocage/templates 

Finally, let's update one of the Base Jails and see what happens
Code:
root@freenas-113U1[~]# iocage update base-1
Snapshot: tank/iocage/jails/base-1@ioc_update_11.3-RELEASE_2020-02-27_23-39-19 created.
Updating jail...

* Updating 11.3-RELEASE to the latest patch level...
Looking up update.FreeBSD.org mirrors... 3 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 11.3-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 115 patches.....10....20....30....40....50....60....70....80....90....100....110.. done.
Applying patches... done.
The following files will be updated as part of updating to
11.3-RELEASE-p6:
/bin/freebsd-version
/boot/loader
**** snip *****
/usr/src/usr.sbin/bhyve/pci_xhci.c
Installing updates... done.
release: 11.3-RELEASE -> 11.3-RELEASE-p6
release: 11.3-RELEASE -> 11.3-RELEASE-p6
release: 11.3-RELEASE -> 11.3-RELEASE-p6
release: 11.3-RELEASE -> 11.3-RELEASE-p6
Updated jail successfully.

base-1 updates have been applied successfully.

As a result all Base jails and Base-clone jails that are created of 11.3-RELEASE have been updated to the latest patch-release 11.3-RELEASE-p6, without consuming more space.
Code:
root@freenas-113U1[~]# iocage list -l
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| JID |    NAME     | BOOT | STATE | TYPE |     RELEASE     | IP4 | IP6 | TEMPLATE | BASEJAIL |
+=====+=============+======+=======+======+=================+=====+=====+==========+==========+
| -   | base-1      | off  | down  | jail | 11.3-RELEASE-p6 | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| -   | base-2      | off  | down  | jail | 11.3-RELEASE-p6 | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| -   | clone       | off  | down  | jail | 11.3-RELEASE    | -   | -   | -        | no       |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| -   | clonebase-1 | off  | down  | jail | 11.3-RELEASE-p6 | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| -   | clonebase-2 | off  | down  | jail | 11.3-RELEASE-p6 | -   | -   | -        | yes      |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
| -   | thick       | off  | down  | jail | 11.3-RELEASE    | -   | -   | -        | no       |
+-----+-------------+------+-------+------+-----------------+-----+-----+----------+----------+
root@freenas-113U1[~]# iocage df
+-------------+-------+------+------+-------+-------+
|    NAME     |  CRT  | RES  | QTA  |  USE  |  AVA  |
+=============+=======+======+======+=======+=======+
| base-1      | 2.51x | none | none | 676M  | 9.41G |
+-------------+-------+------+------+-------+-------+
| base-2      | 2.51x | none | none | 676M  | 9.41G |
+-------------+-------+------+------+-------+-------+
| clone       | 1.00x | none | none | 292K  | 9.41G |
+-------------+-------+------+------+-------+-------+
| clonebase-1 | 1.00x | none | none | 432K  | 9.41G |
+-------------+-------+------+------+-------+-------+
| clonebase-2 | 1.00x | none | none | 432K  | 9.41G |
+-------------+-------+------+------+-------+-------+
| thick       | 2.38x | none | none | 1012M | 9.41G |
+-------------+-------+------+------+-------+-------+
root@freenas-113U1[~]# zfs list -t all -o name,origin,used,refer,mountpoint -r tank/iocage
NAME                                                                       ORIGIN                                               USED  REFER  MOUNTPOINT
tank/iocage                                                                -                                                   3.66G  5.52M  /mnt/tank/iocage
tank/iocage/download                                                       -                                                    289M    88K  /mnt/tank/iocage/download
tank/iocage/download/11.3-RELEASE                                          -                                                    289M   289M  /mnt/tank/iocage/download/11.3-RELEASE
tank/iocage/images                                                         -                                                     88K    88K  /mnt/tank/iocage/images
tank/iocage/jails                                                          -                                                   2.31G    96K  /mnt/tank/iocage/jails
tank/iocage/jails/base-1                                                   -                                                    676M    96K  /mnt/tank/iocage/jails/base-1
tank/iocage/jails/base-1@ioc_update_11.3-RELEASE_2020-02-27_23-39-19       -                                                     60K    96K  -
tank/iocage/jails/base-1/root                                              -                                                    676M   675M  /mnt/tank/iocage/jails/base-1/root
tank/iocage/jails/base-1/root@ioc_update_11.3-RELEASE_2020-02-27_23-39-19  -                                                    252K   675M  -
tank/iocage/jails/base-2                                                   -                                                    676M    96K  /mnt/tank/iocage/jails/base-2
tank/iocage/jails/base-2/root                                              -                                                    675M   675M  /mnt/tank/iocage/jails/base-2/root
tank/iocage/jails/clone                                                    -                                                    292K    92K  /mnt/tank/iocage/jails/clone
tank/iocage/jails/clone/root                                               tank/iocage/releases/11.3-RELEASE/root@clone         200K  1012M  /mnt/tank/iocage/jails/clone/root
tank/iocage/jails/clonebase-1                                              -                                                    432K    96K  /mnt/tank/iocage/jails/clonebase-1
tank/iocage/jails/clonebase-1/root                                         tank/iocage/releases/11.3-RELEASE/root@clonebase-1   336K   675M  /mnt/tank/iocage/jails/clonebase-1/root
tank/iocage/jails/clonebase-2                                              -                                                    432K    96K  /mnt/tank/iocage/jails/clonebase-2
tank/iocage/jails/clonebase-2/root                                         tank/iocage/releases/11.3-RELEASE/root@clonebase-2   336K   675M  /mnt/tank/iocage/jails/clonebase-2/root
tank/iocage/jails/thick                                                    -                                                   1012M    92K  /mnt/tank/iocage/jails/thick
tank/iocage/jails/thick/root                                               -                                                   1012M  1012M  /mnt/tank/iocage/jails/thick/root
tank/iocage/log                                                            -                                                     92K    92K  /mnt/tank/iocage/log
tank/iocage/releases                                                       -                                                   1.06G    88K  /mnt/tank/iocage/releases
tank/iocage/releases/11.3-RELEASE                                          -                                                   1.06G    88K  /mnt/tank/iocage/releases/11.3-RELEASE
tank/iocage/releases/11.3-RELEASE/root                                     -                                                   1.06G  1.03G  /mnt/tank/iocage/releases/11.3-RELEASE/root
tank/iocage/releases/11.3-RELEASE/root@clone                               -                                                    152K  1012M  -
tank/iocage/releases/11.3-RELEASE/root@clonebase-1                         -                                                    152K  1012M  -
tank/iocage/releases/11.3-RELEASE/root@clonebase-2                         -                                                    152K  1012M  -
tank/iocage/templates                                                      -                                                     88K    88K  /mnt/tank/iocage/templates


Long post, but I hope this was helpful and helps people decide what type of jail to use.
 

GeekGoneOld

Dabbler
Joined
Aug 1, 2015
Messages
48
Wow! What incredible detail. Exactly what I was looking for. I don't like a simple "do this" answer, I like the details behind it because when you come up with a special use-case, you can make an informed choice if you understand the details.

Your answer was amazing and very easy to understand. Many, many thanks.
 

GeekGoneOld

Dabbler
Joined
Aug 1, 2015
Messages
48
I assume, therefore, that if I do promote the clone-jail I can delete the snapshot and it is effectively now a thick jail. Big price (in storage) to eliminate a single snapshot!
 

rudds

Dabbler
Joined
Apr 17, 2018
Messages
34
Great question! Your analysis is correct; I'll attempt to highlight the differences in detail in the section below.

Thanks for the thorough explanation. I'm still not sure I understand all the ins and outs of basejails yet, e.g. why is it necessary to update basejails individually? Couldn't you just 'iocage fetch' the latest patch version of your release and that would trickle the updates down to all the basejails? But this is helpful for getting started.

At any rate, my setup in 11.2 was:

1 clone jail customized to my liking, then converted to template
6 clone jails created from that template, each handling a different service

I'm rebuilding everything fresh for 11.3, and wondering: Would it make more sense to use basejails for this setup? Is it possible to convert a basejail to a template, and if so, should the jails created from that template also be basejails? What would the update process look like for such a setup? And would it be possible to make changes to the template basejail (even something simple like, say, a new prompt) that affect the jails made from that template?

Sorry for so many questions -- casting a pretty wide net since I still don't fully have my head around the basejail concept.

EDIT: I did a bit of experimenting on a fresh dataset by making a basejail, customizing a bit, stopping and converting it to a template, then trying to create some more basejails from that template, but all failed to start due to various errors. So it seems like making basejails from templated basejails is a bad idea? Should I be starting with an initial basejail for my template and then making clone jails from that? Or vice versa?
 
Last edited:
Joined
Jul 10, 2016
Messages
521
Couldn't you just 'iocage fetch' the latest patch version of your release and that would trickle the updates down to all the basejails?
Correct, that's how it works. The idea is that base jails do not need to be patched individually.

Is it possible to convert a basejail to a template, and if so, should the jails created from that template also be basejails?
I tried the iocage create from a template in a VM and noted the following:
Code:
+-----+-------------+-------+--------------+------+
| JID |    NAME     | STATE |   RELEASE    | IP4  |
+=====+=============+=======+==============+======+
| -   | base-1      | down  | 11.3-RELEASE | DHCP |
+-----+-------------+-------+--------------+------+
| -   | base-2      | down  | 11.3-RELEASE | -    |
+-----+-------------+-------+--------------+------+
| 4   | clone       | up    | 11.3-RELEASE | DHCP |
+-----+-------------+-------+--------------+------+
| -   | clonebase-1 | down  | 11.3-RELEASE | -    |
+-----+-------------+-------+--------------+------+
| -   | clonebase-2 | down  | 11.3-RELEASE | -    |
+-----+-------------+-------+--------------+------+
| -   | thick       | down  | 11.3-RELEASE | -    |
+-----+-------------+-------+--------------+------+
root@freenas-113U1[~]# iocage set template=yes base-1
base-1 converted to a template.
root@freenas-113U1[~]# iocage create -t base-1 -n from-base-1   
from-base-1 successfully created!
root@freenas-113U1[~]# iocage create -t base-1 -n base-from-base-1 -b
base-from-base-1 successfully created!
root@freenas-113U1[~]# iocage list -l
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| JID |       NAME       | BOOT | STATE | TYPE |     RELEASE     |          IP4          | IP6 | TEMPLATE | BASEJAIL |
+=====+==================+======+=======+======+=================+=======================+=====+==========+==========+
| -   | base-2           | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | base-from-base-1 | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| 4   | clone            | off  | up    | jail | 11.3-RELEASE    | epair0b|192.168.0.148 | -   | -        | no       |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | clonebase-1      | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | clonebase-2      | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | from-base-1      | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | base-1   | no       |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | thick            | off  | down  | jail | 11.3-RELEASE    | -                     | -   | -        | no       |
+-----+------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
root@freenas-113U1[~]# iocage fstab -l base-from-base-1
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| INDEX |                                                                    FSTAB ENTRY                                                                    |
+=======+===================================================================================================================================================+
| 0     | /mnt/tank/iocage/templates/base-1/root/bin      /mnt/tank/iocage/jails/base-from-base-1/root/bin        nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 1     | /mnt/tank/iocage/templates/base-1/root/boot     /mnt/tank/iocage/jails/base-from-base-1/root/boot       nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 2     | /mnt/tank/iocage/templates/base-1/root/lib      /mnt/tank/iocage/jails/base-from-base-1/root/lib        nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 3     | /mnt/tank/iocage/templates/base-1/root/libexec  /mnt/tank/iocage/jails/base-from-base-1/root/libexec    nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 4     | /mnt/tank/iocage/templates/base-1/root/rescue   /mnt/tank/iocage/jails/base-from-base-1/root/rescue     nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 5     | /mnt/tank/iocage/templates/base-1/root/sbin     /mnt/tank/iocage/jails/base-from-base-1/root/sbin       nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 6     | /mnt/tank/iocage/templates/base-1/root/usr/bin  /mnt/tank/iocage/jails/base-from-base-1/root/usr/bin    nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 7     | /mnt/tank/iocage/templates/base-1/root/usr/include      /mnt/tank/iocage/jails/base-from-base-1/root/usr/include        nullfs  ro      0       0 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 8     | /mnt/tank/iocage/templates/base-1/root/usr/lib  /mnt/tank/iocage/jails/base-from-base-1/root/usr/lib    nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 9     | /mnt/tank/iocage/templates/base-1/root/usr/libexec      /mnt/tank/iocage/jails/base-from-base-1/root/usr/libexec        nullfs  ro      0       0 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 10    | /mnt/tank/iocage/templates/base-1/root/usr/sbin /mnt/tank/iocage/jails/base-from-base-1/root/usr/sbin   nullfs  ro      0       0                 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 11    | /mnt/tank/iocage/templates/base-1/root/usr/share        /mnt/tank/iocage/jails/base-from-base-1/root/usr/share  nullfs  ro      0       0         |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 12    | /mnt/tank/iocage/templates/base-1/root/usr/libdata      /mnt/tank/iocage/jails/base-from-base-1/root/usr/libdata        nullfs  ro      0       0 |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+
| 13    | /mnt/tank/iocage/templates/base-1/root/usr/lib32        /mnt/tank/iocage/jails/base-from-base-1/root/usr/lib32  nullfs  ro      0       0         |
+-------+---------------------------------------------------------------------------------------------------------------------------------------------------+

If you create from a template with the -b flag, it will create a basejail, based on the template. Templates become read only; you can't readily update these.

You can also use iocage clone to clone a (base) jail configured to your liking instead of converting it to a template. By default it creates a clone-base jail, but there is a -T option to make it a thick base jail.
Code:
root@freenas-113U1[~]# iocage clone base-2 -n clone-from-base-2     
clone-from-base-2 successfully cloned!
root@freenas-113U1[~]# iocage clone -T base-2 -n thick-from-base-2               
thick-from-base-2 successfully cloned!
root@freenas-113U1[~]# iocage list -l
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| JID |       NAME        | BOOT | STATE | TYPE |     RELEASE     |          IP4          | IP6 | TEMPLATE | BASEJAIL |
+=====+===================+======+=======+======+=================+=======================+=====+==========+==========+
| -   | base-2            | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | base-from-base-1  | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| 4   | clone             | off  | up    | jail | 11.3-RELEASE    | epair0b|192.168.0.148 | -   | -        | no       |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | clone-from-base-2 | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | base-2   | yes      |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | clonebase-1       | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | clonebase-2       | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | from-base-1       | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | base-1   | no       |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | thick             | off  | down  | jail | 11.3-RELEASE    | -                     | -   | -        | no       |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+
| -   | thick-from-base-2 | off  | down  | jail | 11.3-RELEASE-p6 | -                     | -   | -        | yes      |
+-----+-------------------+------+-------+------+-----------------+-----------------------+-----+----------+----------+

If your goal is to be able to patch all 6 cloned jails at once, that would be a better option than using templates.
 

rudds

Dabbler
Joined
Apr 17, 2018
Messages
34
Thanks, I think it's all starting to sink in!

So to break down the steps of what (I think) you're recommending, just to make sure I follow:

-create a basejail and customize it (let's call it firstjail)
-clone my working jails from firstjail (to clarify: the clones will all be basejails as well, right?)
-when it's time to update the FreeBSD release, simply iocage update firstjail and all child jails will automatically be brought up to the same patch level as well?

I think the real reason for my confusion is not being clear which types of subsequent changes you can make to the initial jail that will also trickle down to the child jails. If I understand, it's just updates to the base FreeBSD system, not user-level changes, right? (That seems to be what the nullfs mounts in a basejail imply, but I'm not too experienced with FreeBSD.)

To put the question another way, once you've created a bunch of child jails (either as clones or from a template), there's no viable method of then later making user changes to the initial jail -- such as, say, installing new packages or adding users or changing your shell -- that will be reflected in all the child jails, correct? The user state of the child jails is basically set in stone at the time of creation?
 
Joined
Jul 10, 2016
Messages
521
So to break down the steps of what (I think) you're recommending, just to make sure I follow:

-create a basejail and customize it (let's call it firstjail)
-clone my working jails from firstjail (to clarify: the clones will all be basejails as well, right?)
-when it's time to update the FreeBSD release, simply iocage update firstjail and all child jails will automatically be brought up to the same patch level as well?
Yep, that's it.

If I understand, it's just updates to the base FreeBSD system, not user-level changes, right?
Correct, it will only update the FreeBSD system itself, e.g. from 11.3-RELEASE-p6 to 11.3-RELEASE-p7. It will not update any of the 3rd party software packages that were installed via pkg install or the ports tree.

The user state of the child jails is basically set in stone at the time of creation?
Yes. Only the directories that make up the base FreeBSD system will "trickle down" from the "release". You can check what is mounted from the release into the base jail via iocage fstab -l <your_base_jail>
It may be easiest to just create users and install packages via a script, and run the scrip in all jails that need it. :)
 

rudds

Dabbler
Joined
Apr 17, 2018
Messages
34
Thanks for all the info, this has been really helpful. I've been playing around with this new setup and it's been working out well overall.

The one problem I'm running into is that all of my cloned basejails (configured to use VNET and DHCP) are being assigned the same IP address by my router, where the clone jails in 11.2 had no problem all pulling different IPs.

Looking at this output, it seems like each jail is binding itself to the same virtual network interface? With my old setup, that epair0b interface was unique to each jail, which is to say each one got a unique alphabetical suffix in addition to a unique IP. (base is the original jail in this list and all other jails were cloned directly from it via iocage clone base -n jailname)

Code:
root@freenas[~]# iocage list -l
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| JID |  NAME  | BOOT | STATE | TYPE |     RELEASE     |         IP4          | IP6 | TEMPLATE | BASEJAIL |
+=====+========+======+=======+======+=================+======================+=====+==========+==========+
| 9   | base   | off  | up    | jail | 11.3-RELEASE-p6 | epair0b|192.168.7.53 | -   | -        | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| 5   | cups   | on   | up    | jail | 11.3-RELEASE-p6 | epair0b|192.168.7.53 | -   | base     | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| 7   | forked | off  | up    | jail | 11.3-RELEASE-p6 | epair0b|192.168.7.53 | -   | base     | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| -   | home   | off  | down  | jail | 11.3-RELEASE-p6 | DHCP (not running)   | -   | base     | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| 8   | plex   | off  | up    | jail | 11.3-RELEASE-p6 | epair0b|192.168.7.53 | -   | base     | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| -   | quake  | off  | down  | jail | 11.3-RELEASE-p6 | DHCP (not running)   | -   | base     | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+
| -   | wifi   | off  | down  | jail | 11.3-RELEASE-p6 | DHCP (not running)   | -   | base     | yes      |
+-----+--------+------+-------+------+-----------------+----------------------+-----+----------+----------+


Any thoughts on what's going on here? I wonder, if I had removed the networking config from base before cloning it, and then reset the network options manually for each clone, whether it would have avoided this issue. Although since I've already done some work in the clones, if it's possible to fix it without having to destroy all the clones that would be better.

EDIT: Doing iocage get all jailname on several of the jails reveals that they all have the same settings for hostid, mac_prefix, and vnet0_mac, which I'm guessing is what's causing the problem. I'm not nearly well versed enough in either iocage or BSD networking to know where to go from here, though.

EDIT2: Well, this should teach me to google the problem before running back to the forums -- this issue was brought to the iocage devs' attention a while back.



Looks like they added --newmac (or -N) to the list of iocage clone flags to address the problem. So it's important to note that flag should be used if you want your clones' networking to work out of the box. Per that thread, it looks like manually setting a clone's vnet0_mac to "none" and then restarting the clone will also accomplish the same thing after the fact.
 
Last edited:
Joined
Jul 10, 2016
Messages
521
Good note on that --newmac option!
 

alexr

Explorer
Joined
Apr 14, 2016
Messages
59
What I'm less clear on is how this interacts with package installation. For example, if I build the following tree:

Code:
RELEASE -> clonebase +--> clone-a
                      \-> clone-b


Can I update packages/ports in clonebase and have those updates apply to clone-a and clone-b without breaking the installation of the packages specific to clone-a or clone-b? Will pkg or any port management in clone-a and clone-b see the updates in clonebase or have a stale idea of things?
 
Joined
Jul 10, 2016
Messages
521
Any packages installed in the jails will be independent. The packages will be identical in the "clone" and the "clonebase" at the time of creating the clone, but any subsequent changes (install, update, etc...) in one jail will not be refected in any of the other jails.
 

alexr

Explorer
Joined
Apr 14, 2016
Messages
59
Bummer. I was hoping for more of a file system overlay behavior.
 

NasKar

Guru
Joined
Jan 8, 2016
Messages
739
Can you upgrade your exisiting jails to base jails or do you have to destroy and recreate with the -b option?
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Bummer. I was hoping for more of a file system overlay behavior.
If you want that you can build it with ezjail on stock FreeBSD or iocage, even on FreeNAS, but you won't get it ready-to-use with a nice UI.

We do it like this with Ansible and iocage on FreeBSD:

I even go to explain why clone jails are not "overlay" or "union" FS, so you might enjoy the talk.
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
Can you upgrade your exisiting jails to base jails or do you have to destroy and recreate with the -b option?

I did some testing with zfs promote. The answer is: While you can promote a clone jail, that doesn't make it a base jail. You'd need to destroy and recreate.
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,737
Hmmm ... I am still curious if you could not at least turn in into a thick jail. Create a thick one, then zfs send|zfs receive ...
 

Yorick

Wizard
Joined
Nov 4, 2018
Messages
1,912
At that point, would it not be easier to just recreate the jail, and mount the jail data back in?
 

sankayop

Cadet
Joined
Sep 12, 2020
Messages
3
Yep, that's it.


Correct, it will only update the FreeBSD system itself, e.g. from 11.3-RELEASE-p6 to 11.3-RELEASE-p7. It will not update any of the 3rd party software packages that were installed via pkg install or the ports tree.


Yes. Only the directories that make up the base FreeBSD system will "trickle down" from the "release". You can check what is mounted from the release into the base jail via iocage fstab -l <your_base_jail>
It may be easiest to just create users and install packages via a script, and run the scrip in all jails that need it. :)
Good evening @Jurgen Segaert and all,
I'm new to the use of jails, I just learned how to use ezjail and was happy with it until I got some networking/permission/firewall issues and thought, perhaps I could try iocage since I see a lot of people are using it and that the github is also quite active.
Thus, after reading the iocage's manual, looking some video's (e.g. https://www.youtube.com/watch?v=kAJ7RzfPaLA) and reading your post, I feel nearly ready to use iocage.
Although, there's still something that's not clear to me.

If I take the setup proposed by @rudds where the idea is to be able to update in one shot, 6 jails that all have the same basic configuration.
What is the difference between:
- creating a basejail, tune it, make a template of it ; create 6 basejails following the template.
```
iocage create -r -n base -b
(tune it)
iocage set base template=yes
iocage create -t base -c 6 -b
```
- creating a basejail, tune it ; create 6 clones of the previous basejail.
```
iocage create -r -n base -b
(tune it)
iocage clone base -c 6
```
(sorry if there are mistakes in the commandlines, I'm just starting with this :) The important is the point).

My understanding is that for the first case, the template will be read-only and can not be deleted unless the 6 basejails first deleted. But the 6 basejails can be update with one `iocage update`
For the second case, the 6 clones are copies of the basejail's snapshot (thus relying on the existence of the basejail). Nevertheless, the basejail is still writable. Moreover, here also, one `iocage update` will update all jails.

Am I right or completely wrong?
Thanks for enlighting me.
 

sankayop

Cadet
Joined
Sep 12, 2020
Messages
3
Good evening @Jurgen Segaert and all,
I'm new to the use of jails, I just learned how to use ezjail and was happy with it until I got some networking/permission/firewall issues and thought, perhaps I could try iocage since I see a lot of people are using it and that the github is also quite active.
Thus, after reading the iocage's manual, looking some video's (e.g. https://www.youtube.com/watch?v=kAJ7RzfPaLA) and reading your post, I feel nearly ready to use iocage.
Although, there's still something that's not clear to me.

If I take the setup proposed by @rudds where the idea is to be able to update in one shot, 6 jails that all have the same basic configuration.
What is the difference between:
- creating a basejail, tune it, make a template of it ; create 6 basejails following the template.
```
iocage create -r -n base -b
(tune it)
iocage set base template=yes
iocage create -t base -c 6 -b
```
- creating a basejail, tune it ; create 6 clones of the previous basejail.
```
iocage create -r -n base -b
(tune it)
iocage clone base -c 6
```
(sorry if there are mistakes in the commandlines, I'm just starting with this :) The important is the point).

My understanding is that for the first case, the template will be read-only and can not be deleted unless the 6 basejails first deleted. But the 6 basejails can be update with one `iocage update`
For the second case, the 6 clones are copies of the basejail's snapshot (thus relying on the existence of the basejail). Nevertheless, the basejail is still writable. Moreover, here also, one `iocage update` will update all jails.

Am I right or completely wrong?
Thanks for enlighting me.
Update: I forgot the diskspace... base: 676M, clonebase: 432K. Is it the only difference?
 
Top