Is it possible to allow a jail to see all host pools and datasets?

[scott2500uk]

As the title asks: Is it possible to allow a jail to see all host pools and datasets?

I'd like to use zrepl in a jail to be able to snapshot manage and backup our pools and datasets to our offsite server. Zrepl has a few features that I think are making it a more preferable solution for us compared to the built-in replication system of TrueNAS.

I can see there is a zrepl community plugin but I can only see a way to pass in a single dataset that has to be given full control to the jail which wont allow us to use zrepl to backup all our datasets.

 

[Heracles]

Any reason why you are not using the built-in snapshots / replication tasks ???

 

[scott2500uk]

Any reason why you are not using the built-in snapshots / replication tasks ???

Yes. Not going to explain all my reason as they are not relevant to my question but to give you an idea https://zrepl.github.io/ many of the advanced replication features in zrepl are not available in the built-in replication system.

 

[Heracles]

So next question is any reason for multiple pools ? The reason to have more than 1 pool is when you need different pool-level options for different data. But considering how rare that is, to have multiple pools is often more problems and risks than benefits. Considering how many times we see this in the forum. it is often a sign of misunderstanding some fundamentals about ZFS and TrueNAS...

 

[scott2500uk]

So next question is any reason for multiple pools ? The reason to have more than 1 pool is when you need different pool-level options for different data. But considering how rare that is, to have multiple pools is often more problems and risks than benefits. Considering how many times we see this in the forum. it is often a sign of misunderstanding some fundamentals about ZFS and TrueNAS...

Thank you for your input, I'm not sure how your question gets you any closer to telling me if it is possible to expose all my pools and datasets to a jail? I think you are jumping to conclusions because I don't want to use the built-in system that I must be doing something strange or wrong?

I have two pools just like 99.9% of other TrueNAS installs. A boot pool and a data pool. I'm not doing anything out of the ordinary with my setup. I just want better control over snapshots and replication than what TrueNAS offers. zrepl offers that and is obviously aimed at the more advanced user.

Respectfully, I'm just looking for advice to know if exposing zfs pools and datasets in a jail is possible under TrueNAS.

I appreciate you are a longstanding forum member and offer your help for free here, so thank you again. I also understand that many questions that land here are from novice users doing strange things who usually end up in tears because they have lost all their data. I can assure you I'm no novice, granted I'm no expert either, but I have been managing many ZFS systems for over 10 years, most in small businesses with pools in excess of 80TB.

Cheers

 

[Patrick M. Hausen]

Yes, you can access all datasets on all pools from inside a jail but there is no global switch to enable that. You need to mount all of them one by one to a suitable mountpoint inside the jail.

 

[scott2500uk]

Yes, you can access all datasets on all pools from inside a jail but there is no global switch to enable that. You need to mount all of them one by one to a suitable mountpoint inside the jail.

Thank you Patrick. I hadn't tried mounting as I thought this would just present the filesystem and not expose them as zfs datasets that zrepl could then create snapshots on. Am I wrong in this assumption? I hope I am wrong as I could live with creating all the mount points manually. Would the same apply to the receiving end jail, as in creating the same datasets and then mounting them into the jail the same way?

 

[Patrick M. Hausen]

Ah ... now I get what you are intending. No, you cannot do that in a jail to my knowledge. There is the "jail_zfs" property to iocage and friends for a jail, but that only allows you to create and manage ZFS datasets below the jail root.

So if you are still in the design phase, you could of course use that, create all the sharing datasets inside the jail as suggested by the zrepl documentation, and then set up all your shares and other services to use the datasets that are child datasets of your jail ...

HTH,
Patrick

P.S. Come to think of it, that's actually quite clever The <pool>/iocage dataset is hardwired into iocage and TrueNAS, but for datasets for SMB, iSCSI ZVOLs etc. TrueNAS does not enforce any particular layout. That's what confuses some beginners - you need a bit of experience to come up with <mypool>/share/<something> and <mypool>/vms/<some-vm> ... etc. and recognize you want to treat these things differently e.g. with respect to snapshots in the first place.

P.P.S. You will be limited to a single pool, since everything needs to reside "below" your jail.

 

[Heracles]

Thank you for your input, I'm not sure how your question gets you any closer to telling me if it is possible to expose all my pools and datasets to a jail? I think you are jumping to conclusions because I don't want to use the built-in system that I must be doing something strange or wrong?

To understand a setup is important. Just to tell someone how to drive a car on a race track does not make it safe and sound. Whenever someone asks about offroad driving, to be sure the situation is properly understood is important not to fast forward him to catastrophe.

As you do not seem to appreciate the care we take for your data, do as you wish then...

I have two pools just like 99.9% of other TrueNAS installs. A boot pool and a data pool.

No need to do anything about the boot pool. Just backup your config. It is actually much safer and much easier. So you actually have only a single pool to manage here.

This is why we are asking questions...

Respectfully, I'm just looking for advice to know if exposing zfs pools and datasets in a jail is possible under TrueNAS.

So do whatever you wish if you do not appreciate the care we show about your setup, data and safety....

 

[Ericloewe]

Ah ... now I get what you are intending. No, you cannot do that in a jail to my knowledge. There is the "jail_zfs" property to iocage and friends for a jail, but that only allows you to create and manage ZFS datasets below the jail root.

So if you are still in the design phase, you could of course use that, create all the sharing datasets inside the jail as suggested by the zrepl documentation, and then set up all your shares and other services to use the datasets that are child datasets of your jail ...

HTH,
Patrick

P.S. Come to think of it, that's actually quite clever The <pool>/iocage dataset is hardwired into iocage and TrueNAS, but for datasets for SMB, iSCSI ZVOLs etc. TrueNAS does not enforce any particular layout. That's what confuses some beginners - you need a bit of experience to come up with <mypool>/share/<something> and <mypool>/vms/<some-vm> ... etc. and recognize you want to treat these things differently e.g. with respect to snapshots in the first place.

P.P.S. You will be limited to a single pool, since everything needs to reside "below" your jail.

I don't think they strictly need to be children of the jail's root dataset, though I may be mistaken. What I know for sure is that you can't go overboard and jail every dataset and that the mountpoint needs to be inside the jail's filesystem.

 

[scott2500uk]

Ah ... now I get what you are intending. No, you cannot do that in a jail to my knowledge. There is the "jail_zfs" property to iocage and friends for a jail, but that only allows you to create and manage ZFS datasets below the jail root.

So if you are still in the design phase, you could of course use that, create all the sharing datasets inside the jail as suggested by the zrepl documentation, and then set up all your shares and other services to use the datasets that are child datasets of your jail ...

HTH,
Patrick

Oh poop. Thanks, I understand it better now. I'm going to do some more testing with that in mind and see if I want to go through all the effort to get zrepl working. I have already migrated ~35TiB to this system I am building but it isn't out of the question to restructure at this point.

 

[Patrick M. Hausen]

You can rename a dataset including the entire path.

zfs rename mypool/shares/some-large-data mypool/iocage/jails/zrepl/root/shares/some-large-data works as long as mypool/iocage/jails/zrepl/root/shares already exists.

 

[Patrick M. Hausen]

What I know for sure is that you can't go overboard and jail every dataset and that the mountpoint needs to be inside the jail's filesystem.

I thought for snapshots they need to be child datasets but I may be mistaken. This calls for more investigation, apparently.

 

[scott2500uk]

You can rename a dataset including the entire path.

zfs rename mypool/shares/some-large-data mypool/iocage/jails/zrepl/root/shares/some-large-data works as long as mypool/iocage/jails/zrepl/root/shares already exists.

Thanks. I've just made a test dataset in the root of my pool called z and set a jail and used jail_zfs=1 and jail_zfs_dataset=z

The jail sees the dataset as expected and can do all the zfs stuff to it that you'd expect. With the jail running, in the truenas GUI I went and created a child dataset. On clicking create, it appeared to do nothing. Clicking create again, it came back with an error saying the dataset had already been created. Clicking cancel to go back to the pools page in the gui, the child dataset was indeed listed but I couldn't set any permissions on the dataset. It was like it was locked out and all the control was given to the jail.

If this is the case then this won't work for us as we would need to retain control in the GUI on all the datasets to continue to administrate them.

I'm going to experiment some more but I feel like this is going to be a dead end.

I know this is a big no-no in the truenas world but would it be a very crazy idea to install zrepl on the host system? I know updates will break it yada yada, but once this goes into production it is very unlikely to get touched.

 

[Patrick M. Hausen]

What exactly are you missing in TrueNAS replication tasks? Maybe a festure request?

P.S. As far as I understood the underlying mechanism, as soon as you delegate a dataset to a jail, all management must be done from within. Including the creation of child datasets. Michael W. Lucas probably has some more information in "FreeBSD Mastery: Jails".

 

[scott2500uk]

What exactly are you missing in TrueNAS replication tasks? Maybe a festure request?

The documentation around the built-in replication of TrueNAS doesn't say much about what it can and cannot do so some of the features that I want from zrepl may actually exist in TrueNAS.

Why zrepl wins for me:
1) TCP transport without the SSH overhead. My servers are connected over a wireguard tunnel. Truenas does have SSH+netcat which I suspect is somewhat similar but meh
2) Automatic retries. Again TrueNAS might do this but I cannot see any documentation that it does do this.
3) Resumable sends and receives. As this is a ZFS feature I'm sure TrueNAS does this but again it isn't documented.
4) Automatic bookmark & hold management for guaranteed incremental send & recv. Let's say I want to snapshot every minute and send the smallest amount each minute. Realistically I only want to store minute snapshots for an hour. If there is some outage (for example moving the backup server offsite after seeding locally) then I would want the snapshots to be kept until they are synced. I'm pretty sure TrueNAS doesn't hold snapshots, I may be wrong.
5) Easier and more flexible snapshot management. Easier to setup age-based fading (grandfathering scheme) snapshots.
6) Cross-platform tool.

I could go on but I just feel like I'm regurgitating zrepl main feature list.

Really the feature request would be to either implement zrepl into TrueNAS or add its main features that TrueNAS is missing.

 

[Ericloewe]

Random thought: doesn't zrepl support a remote mode, in which it pulls the data from the destination, in addition to the control stuff?

 

[scott2500uk]

Random thought: doesn't zrepl support a remote mode, in which it pulls the data from the destination, in addition to the control stuff?

yes it supports remote mode and can do pulls but from what I understand you still need to have a sink setup on the other end to allow the connections etc. So the requirement is that zrepl is at both ends.

The configuration example it has here https://zrepl.github.io/quickstart/continuous_server_backup.html is pretty much the same as how I would have it set up for my datasets.

 

[sretalla]

I saw this after installing the plugin version...

Install Notes:
zrepl is now installed
A ZFS dataset must be delegated into the zrepl jail. Create a dataset on the
host if one was not delegated during jail creation, stop the plugin jail,
and set the following iocage property:
jail_zfs_dataset=
The "" should not contain the pool name.
Then start the plugin.
Now the configuration file located at "/usr/local/etc/zrepl/zrepl.yml"
within the zrepl jail can be edited and zrepl can be used.
For zrepl documentation see https://zrepl.github.io/



Separately to that (and why I was installing the plugin in the first place) I wanted to test if you could just run it on the host directly (since that seems to me to be what you really want) from inside the jail install...

/mnt/tank/iocage/jails/zrepl/root/usr/local/bin/zrepl --config /mnt/tank/iocage/jails/zrepl/root/usr/local/etc/zrepl/zrepl.yml daemon

Seems to me that you can...

 

[scott2500uk]

Separately to that (and why I was installing the plugin in the first place) I wanted to test if you could just run it on the host directly (since that seems to me to be what you really want) from inside the jail install...

/mnt/tank/iocage/jails/zrepl/root/usr/local/bin/zrepl --config /mnt/tank/iocage/jails/zrepl/root/usr/local/etc/zrepl/zrepl.yml daemon

Seems to me that you can...

Ah yes, good shout. I will test that and report back.