Confused with Boot Environments: clone or create ?

[toadman]

I did both Clone and Create. "I now issue a "Clone" for 11.0-U4 and call it "11.0-U4-Clone" and also issue a "Create" and call it "created-via-create-button".

So both show up at datasets, which is what I would expect. And they are different. (Well, at least a hash of the recursive file list is different.)

 

[Black Ninja]

Perhaps one is like create snapshot of running/active boot other clone the snapshot it self.


More recently I study how EXSI snapshots work , and I am now even more confused about snapshot on zfs from the time I learn about them. it turns out:

snapshots in ESXI - BAD
snapshots in ZFS - GOOD
snapshots of esxi VM machine on ZFS datastore - TO BE DETERMINED

 

[toadman]

Perhaps one is like create snapshot of running/active boot other clone the snapshot it self.

That's what I was thinking as well. But there are implications of that I suppose. I think it means:

• Snapshot is created on upgrade to new version
• Changes to boot environment exist on active dataset. (For example if I change a tunable like vfs.zfs.arc_max or something.)
• If I "Clone" a boot environment it clones the snapshot, which would NOT have the changes in it. (So if I changed vfs.zfs.arc_max it would not show up in the clone.)
• If I "Create" a boot environment it creates a snapshot of the current environment, and would therefore have any changes it in. (So the vfs.zfs.arc_max tunable would be in it.)

More recently I study how EXSI snapshots work , and I am now even more confused about snapshot on zfs from the time I learn about them. it turns out:

snapshots in ESXI - BAD
snapshots in ZFS - GOOD
snapshots of esxi VM machine on ZFS datastore - TO BE DETERMINED

What do you mean? In what context? ESXi shapshots are very useful. For example anytime I am worried about an upgrade I take a shapshot of the VM. Upgrade doesn't work? Revert to the snapshot.

Snapshots in ZFS are obviously good. Especially for non-open files. Snapshots in ZFS are can be bad if snapshotting an esxi datastore. Why? The captured state of the VM on disk may not be stable. If there are items in memory that have not been written to disk there may be a problem. If you take the VM out of the cloned ZFS snapshot it may fail to run.

Precisely why the vmware snapshot function is implemented in Freenas. You snap the VM from within esxi or vcenter, which quiesses the file system and captures the memory image. Then you snap the ZFS dataset. Then you delete the exsi snapshot. This way if you restore the VM from the ZFS snapshot you have an esxi generated snapshot to revert to if the VM won't run.

 

[Black Ninja]

What do you mean? In what context? ESXi shapshots are very useful. For example anytime I am worried about an upgrade I take a shapshot of the VM. Upgrade doesn't work? Revert to the snapshot.

They are very useful, no doubt I use them too, but perhaps you know that in ESXI running of snapshot for a longer time is not recommended. It suppose to be ran just for a short time to test an upgrade of something and then snapshot should be consolidated. Because of the way snapshot is created in ESXI: the original file is split to delta file , so you have the original read-only and the changed file that system have to read from both then write to the delta.. and bla bla. So if you have a snapshot ( or more than one is even worst) and this is how normally running the system is not good (per vmware ) and the performance hit is huge. (I have SSD datastore and writes suffer big time)

So that what I meant snapshots are "BAD" for ESXI , cause you are not suppose to normally run system of snapshot, but on freenas-zfs on other hand (since snapshots there are different) it seems that is fine to have snapshots all the time and even many of them. It seem that don't cause problem like ESXI snapshot does.

the the third I said: "snapshots of esxi VM machine on ZFS datastore - TO BE DETERMINED", you can forget about this one , cause we will drift too far from the topic if we got there.

 

[Black Ninja]

• Snapshot is created on upgrade to new version

• Changes to boot environment exist on active dataset. (For example if I change a tunable like vfs.zfs.arc_max or something.)

• If I "Clone" a boot environment it clones the snapshot, which would NOT have the changes in it. (So if I changed vfs.zfs.arc_max it would not show up in the clone.)

• If I "Create" a boot environment it creates a snapshot of the current environment, and would therefore have any changes it in. (So the vfs.zfs.arc_max tunable would be in it.)

I really like your logic. It's make very good sense, it must be the truth how it should work.

So when I get to the point where my freenas is configured as general template condition (but no specifics like- pool or shares are added) THEN I have to use "create" button at that moment to make perfect template copy from which I can later branch/fork 2 separate specific purpose freenas machines.

 

[toadman]

So that what I meant snapshots are "BAD" for ESXI , cause you are not suppose to normally run system of snapshot, but on freenas-zfs on other hand (since snapshots there are different) it seems that is fine to have snapshots all the time and even many of them. It seem that don't cause problem like ESXI snapshot does.

Ah, gotcha. ESXi snapshots are bad due to implementation and resulting performance impact. Yes, makes sense. :)

 

[Black Ninja]

Ah, gotcha. ESXi snapshots are bad due to implementation and resulting performance impact. Yes, makes sense. :)

And the risk for running out of space :
"Delta files can grow to the same size as the original base disk file, which is why the provisioned storage size of a virtual machine increases by an amount up to the original size of the virtual machine multiplied by the number of snapshots on the virtual machine."

Anyways, Let ses where we were at with the boot environments:

Base on this very good point you made:

• Snapshot is created on upgrade to new version

• Changes to boot environment exist on active dataset. (For example if I change a tunable like vfs.zfs.arc_max or something.)

• If I "Clone" a boot environment it clones the snapshot, which would NOT have the changes in it. (So if I changed vfs.zfs.arc_max it would not show up in the clone.)

• If I "Create" a boot environment it creates a snapshot of the current environment, and would therefore have any changes it in. (So the vfs.zfs.arc_max tunable would be in it.)

Then, it seems the best way to preserve the current progress is to use "create" button, on your active environment, instead of selecting it make "clone" ?

 

[toadman]

So I went ahead and fired up a VM and did a new install of 11.1-U2 and played around a bit. First here is what I did:

• Boot into default environment (actually labeled "default" on the Boot menu)
• Made several changes to the system (to see what would propagate through a Clone or a Create)
• Made a CLONE of "default"
• Made a CREATE environment
• Booted into both new boot environments to see what propogated through. All changes had propogated through in both cases.

My conclusion:

• CREATE snaps the current ("Active") environment at that moment and creates a new boot environment you can select.
• CLONE snaps the selected environment and clones it in it's current state. If the selected environment is the "Active" environment it functions the same as CREATE.

Obviously it's not extensive testing, but good enough for me. I am going to re-read the docs and submit a bug to have it cleaned up if it's not clear enough.

 

[toadman]

So what I would do if experimenting with system changes is:

(1) After install clone the active environment to "Clean-Install" or something. Now you have a clean baseline
(2) Make desired changes to the system.
(3) Rename the active environment to "Changes-Made-1" (probably from "default")
(4) (Optional reboot.) Test system with changes to see if it works as desired.

If Yes:

1. You are good to go.
2. Keep Changes-Made-1 as the active environment. (Or rename it to whatever you want.)

If No:

1. Effectively revert back to Clean-Install by FIRST issuing a CLONE of Clean-Install and call it Changes-Made-2.
2. Set the Active environment to Changes-Made-2 and reboot.
3. Go back to (4) at the top and test the changes.
4. Delete Changes-Made-1 if desired as it's no longer needed.

 

[toadman]

Bug filed. 28649

 

[Ericloewe]

Boot environments are always clones of snapshots, that's why they're writable.

If I'm not mistaken, "clone" clones an existing snapshot to create a new boot device and "create" snapshots the currently-active environment and clones it to create the new environment.

 

[toadman]

I think you are correct. Create is just confusing because it's (potentially) redundant to doing a clone on the active environment.

 

[Black Ninja]

Boot environments are always clones of snapshots, that's why they're writable.

If I'm not mistaken, "clone" clones an existing snapshot to create a new boot device and "create" snapshots the currently-active environment and clones it to create the new environment.

So boot environments are just different snapshots of boot volume , is that it ?


P.S. I have re-read again about snapshot on zfs , how they work , cause I had ESXI snapshots more fresh in my mind and I am confused now. On freenas , when I select boot environment Am I running of s snapshot ? Is it ok to run this way indefinitely ?!

 

[Chris Moore]

On freenas , when I select boot environment Am I running of s snapshot ? Is it ok to run this way indefinitely ?!

Because of the copy on write nature of the ZFS file system, and how it creates snapshots to begin with, and destroys them when they get destroyed, it is perfectly fine to run from a snapshot. The basin functionality is so different. Any changes are written to disk and read from disk in the same way any other data in ZFS is written to and read from disk. All that really happens is that the pointers to data in other snapshots is retained instead of being marked as free space.

 

[Black Ninja]

Because of the copy on write nature of the ZFS file system, and how it creates snapshots to begin with, and destroys them when they get destroyed, it is perfectly fine to run from a snapshot. The basin functionality is so different. Any changes are written to disk and read from disk in the same way any other data in ZFS is written to and read from disk. All that really happens is that the pointers to data in other snapshots is retained instead of being marked as free space.

And the performance shouldn't have an impact ?

 

[Chris Moore]

And the performance shouldn't have an impact ?

Where ESXi has a disk file and a dif file, which can create a performance issue, ZFS just writes new nodes on the file system in the same way it normally does any time it stores data. In a 'copy-on-write' file system, new data is written to an available space on the disk, not over the old data. So, any writes happen in the same way they would have and the reads do to. No performance impact. All a snapshot does is tell the file system to not free the 'old' data space on disk and maintain a set of pointers to the location where the data is that was replaced by the new write. It is basically a set of pointers that keep track of all the nodes on disk that represent what was and another set of pointers that represent what is now and the two can have an overlap of data that is common to both sets.
I have a 250TB array at work were we keep about 40 snapshots of the data so we can go back if someone accidentally deletes something. It is part of the design of the ZFS file system and it works very well.

 

[Black Ninja]

It is part of the design of the ZFS file system and it works very well.

That's why we are loving it Chris !:)

So as far as the boot environments which way would you do it , if the goal is:

After new fresh freenas installation you do a general setting and customization , but not adding any pools or further changes. You want to preserve this state and after that you want to fork 2 separate ways:
-one set of hdds with pools set, shares etc.
-second set of HDD with totally different settings and pools

and the idea is to switch between them by swapping drives and change the boot environment on next boot so you can have to totally different freenases , using the same hardware (except drives) but having totally different setup. And it take literary one reboot time, to switch between them ?

 

[Chris Moore]

After new fresh freenas installation you do a general setting and customization , but not adding any pools or further changes. You want to preserve this state and after that you want to fork 2 separate ways:

After the initial configuration, I would create two clones. That gives me the initial, plus the two clones, a total of three. Name the clones something meaningful. Then I would boot into the clones, each in turn, and make the customization that are required. For all three of the clones, the base and two customized ones, you will need a separate backup of the config file. This should all work just as you intend and you can access the list of boot images from the grub menu to select which one to boot into.
Something you will need to do, when it comes time to upgrade to a new version of FreeNAS, you will probably need to delete the two clones, upgrade in the base install, recreate the clones, boot into each of them and restore the config file that is appropriate to the clone you are in...
This is theoretical, as I have not done it, but I am thinking through it and I feel sure it would work.

 

[Black Ninja]

I would create two clones.

2 ways to accomplish that :
1. I select select active-default environment and make 2 clones by pressing "clone" button and give them deferment names.
2. While in active-default environment, without selecting anything just pres "Create" button twice and give them different names.

Would it matter which way is done ?

Something you will need to do, when it comes time to upgrade to a new version of FreeNAS

The good thing is I don't do upgrades,so won't have to worry about it, but what you are saying makes perfect sense.

 

[Chris Moore]

The good thing is I don't do upgrades,so won't have to worry about it, but what you are saying makes perfect sense.

I upgraded without any negative impact. I can't guarantee that you won't have any problem, but I don't understand the idea of not installing upgrades, even if a problem does crop up.

Would it matter which way is done ?

We have been covering this for a couple days. I thought it was already decided.