Help upgrade for deduplication (keeping chassis)

[dxun]

I wouldn't expect a NVMe drive to go in a 3.5" bay anyway.

I am actually doing exactly that - using this nifty, tooless Supermicro drive caddy to host both the Optanes and the Intel P4510s. Another option is an adapter like this. The real trick is to either have a backplane that supports NVMe or to be creative with the power/data cabling.

The easy and safe option is to have one drive (Optane would be best, but a regular SSD could do) to host the DDT as persistent metadata L2ARC. This will install in a M.2 or PCIe slot.

Oh, this looks interesting - so you can actually have L2ARC host DDTs? This may be the sole reason why you'd want to mirror your L2ARC. But the dedicated dedup special vdev still seems like the better way to go, if you don't need L2ARC.

 

[Davide Zanon]

At the risk of sounding impolite, with the information provided there is a good chance that snapshots would be an equally suitable solution. But it is of course your decision whether or not you want to pursue this approach.

Please BE impolite if I'm being stubborn :), what I don't understand is how snapshots could replace dedup:
I'm backing up every night my Windows fileserver to the NAS, I have 6 days (mon-sat) differential backups
and one (sunday) full backup, so I have the same files backupped many times across the year;
this lead me to think that I want/need dedup because it can help me retain way more data, my actual dedup
ratio (3.25x) tells me so: I'm storing 43.5TB in just 18.2TB (am I understanding this all wrong?).
If I can achieve the same result with snapshots without dedup and all the hassle/burden it takes... then
please, please do advice on how to do it, and if you want/have time explain how and why is it possible,
I'm all about undestanding things and learning from my mistakes.
Keep in mind that this system originally had a Netgear proprietary OS based on Solaris (I think?) and it
came with dedup already enabled, so all I did a few years ago was to replace that OS with FreeNAS
when the internal USB storage decided to die for good, imported the pool and kept it that way.

 

[Arwen]

The way backups and snapshots can work, is that you do this;

1. Create a dedicated dataset for each client, like "pool/backups/CLIENT"
2. Backup the client to it's backup dataset using a differential backup program like RSync
3. At the end of the backup snapshot the client's dataset with the backup's date

Each snapshot only takes up the amount of space that is different from the source dataset. Thus, if a server uses 50GBs but only 5GB changed, then you only have to store 5GBs of additional space for that one backup. Any file that did not change, is not written to the backup again. Sort of a poor man's de-dup.

Next, the snapshots represent the exact file layout of the server at the time of the backup & snapshot. Thus, if you need a prior state of a client server, you can mount a snapshot and get files from an earlier date. So each snapshot is sort of a full backup.

In my case, I simply keep dozens of snapshots on the backup disks and NAS. This goes back years, (for my home computers). When the disk or NAS pool gets full, I will delete the oldest backup snapshot to recover space.

 

[Davide Zanon]

The way backups and snapshots can work, is that you do this;

1. Create a dedicated dataset for each client, like "pool/backups/CLIENT"
2. Backup the client to it's backup dataset using a differential backup program like RSync
3. At the end of the backup snapshot the client's dataset with the backup's date

Each snapshot only takes up the amount of space that is different from the source dataset. Thus, if a server uses 50GBs but only 5GB changed, then you only have to store 5GBs of additional space for that one backup. Any file that did not change, is not written to the backup again. Sort of a poor man's de-dup.

Next, the snapshots represent the exact file layout of the server at the time of the backup & snapshot. Thus, if you need a prior state of a client server, you can mount a snapshot and get files from an earlier date. So each snapshot is sort of a full backup.

In my case, I simply keep dozens of snapshots on the backup disks and NAS. This goes backup years, (for my home computers). When the disk or NAS pool gets full, I will delete the oldest backup snapshot to recover space.

But like this you're only doing differential backups, what about full backups?
I haven't taken a snapshot with TrueNAS yet, is it simple to mount it and browse the files?
Right now if I want to restore a file I just browse the backup directory via SMB and restore it.

 

[Arwen]

But they are FULL backups. Why backup something that already exists on the NAS backup server?

Example:
- You have 3 files on the client. 1 changes all the time, 1 changes occasionally, the the last does not change
- The initial RSync would backup all 3, because it's an empty dataset
- Any follow up RSync would always backup the file that changed alot, and occasionally backup the 2nd file. But, the 3rd file is still on the NAS, so why copy it again?

- Backup 1: All 3 files
- Backup 2: 1st file
- Backup 3: 1st file & 2nd file
- Backup 4: 1st file
etc...

ZFS snapshots and RSync saves both network time and NAS disk space.

Instead of having separate paths for each backup, which require duplicate space, we combine each client's backup in to the same space, but with snapshots representing the state of the client server at the time of the backup. So each snapshot / RSync backup is both a full on the NAS, and faster incremental on the client server.

Sometimes it's hard to wrap your head around a concept. Sorry if I am not explaining it well.

 

[Davide Zanon]

But they are FULL backups. Why backup something that already exists on the NAS backup server?

Example:
- You have 3 files on the client. 1 changes all the time, 1 changes occasionally, the the last does not change
- The initial RSync would backup all 3, because it's an empty dataset
- Any follow up RSync would always backup the file that changed alot, and occasionally backup the 2nd file. But, the 3rd file is still on the NAS, so why copy it again?

- Backup 1: All 3 files
- Backup 2: 1st file
- Backup 3: 1st file & 2nd file
- Backup 4: 1st file
etc...

ZFS snapshots and RSync saves both network time and NAS disk space.

Instead of having separate paths for each backup, which require duplicate space, we combine each client's backup in to the same space, but with snapshots representing the state of the client server at the time of the backup. So each snapshot / RSync backup is both a full on the NAS, and faster incremental on the client server.

Sometimes it's hard to wrap your head around a concept. Sorry if I am not explaining it well.

My problem with this concept is if I want to restore a file/folder a user accidentally deletes,
I have to mount each snapshot until I find that file/folder and restore it, instead if I keep on
using my actual implementation I have everything inside a folder that's easily searchable.
Again, I'm not at all accustomed to snapshots so maybe there's an easy solution to this,
that's why I'm asking all kind of (weird) questions about the subject.

 

[Arwen]

You can have all the snapshots mounted. I don't know how it works with SMB, but it's just a directory, again easily searchable.
Here is an example from my old laptop for my home directories' snapshots:

Code:

root@laptop:/home# ls -la
total 42
drwxr-xr-x  3 root    root    3 Aug  6  2021 .
drwxr-xr-x 23 root    root   23 Jan  2 23:43 ..
drwxrwxrwx  1 root    root    0 Jan  2 23:43 .zfs
drwx------ 63 myuser users  121 Jan  2 23:44 myuser
root@laptop:/home# ls .zfs/snapshot/
dow_0  dow_3  dow_6   hod_02  hod_05  hod_08  hod_11  hod_14  hod_17  hod_20  hod_23
dow_1  dow_4  hod_00  hod_03  hod_06  hod_09  hod_12  hod_15  hod_18  hod_21
dow_2  dow_5  hod_01  hod_04  hod_07  hod_10  hod_13  hod_16  hod_19  hod_22

Those snapshots are 7 DOW, (Day Of Week), and 24 HOD, (Hour Of Day). They are intended to help me recover deleted files and changes I want to back out. Not perfect because the old file has to be their up to 1 hour for the HOD snapshots to pick it up. In my case, I don't consider backing out changes more than 1 week old. Thus, I only keep 7 DOW snapshots.

Underneath each snapshot directory is a fulll R/O copy of my home file system at the time of the snapshot. Easy enough to search.

Note in the above example, "/home" is a separate dataset. Thus, the snapshots are on it, not the user home. But for client backups, (or user home shares), you would have separate ZFS datasets for each client and possibly user homes if kept on the NAS.

 

[Davide Zanon]

W

You can have all the snapshots mounted. I don't know how it works with SMB, but it's just a directory, again easily searchable.
Here is an example from my old laptop for my home directories' snapshots:

Code:

root@laptop:/home# ls -la
total 42
drwxr-xr-x  3 root    root    3 Aug  6  2021 .
drwxr-xr-x 23 root    root   23 Jan  2 23:43 ..
drwxrwxrwx  1 root    root    0 Jan  2 23:43 .zfs
drwx------ 63 myuser users  121 Jan  2 23:44 myuser
root@laptop:/home# ls .zfs/snapshot/
dow_0  dow_3  dow_6   hod_02  hod_05  hod_08  hod_11  hod_14  hod_17  hod_20  hod_23
dow_1  dow_4  hod_00  hod_03  hod_06  hod_09  hod_12  hod_15  hod_18  hod_21
dow_2  dow_5  hod_01  hod_04  hod_07  hod_10  hod_13  hod_16  hod_19  hod_22

Those snapshots are 7 DOW, (Day Of Week), and 24 HOD, (Hour Of Day). They are intended to help me recover deleted files and changes I want to back out. Not perfect because the old file has to be their up to 1 hour for the HOD snapshots to pick it up. In my case, I don't consider backing out changes more than 1 week old. Thus, I only keep 7 DOW snapshots.

Underneath each snapshot directory is a fulll R/O copy of my home file system at the time of the snapshot. Easy enough to search.

Note in the above example, "/home" is a separate dataset. Thus, the snapshots are on it, not the user home. But for client backups, (or user home shares), you would have separate ZFS datasets for each client and possibly user homes if kept on the NAS.

What about compressed files? Those will occupy real space on the pool instead of being deduped,
same thing for databases that send diff/full backups, that space will be used up.
I'm not trying to convince me that I need dedup, I'm just trying to understand if I can really live without it.
My goal is always to retain as much data as possible, if I can do it with fewer resoucers the better.
Also, why take snapshots anyway if you're sendind only diffs to the server?
Thanks

 

[Etorix]

The real trick is to either have a backplane that supports NVMe or to be creative with the power/data cabling.

This is where I expect a stumbling block…

Oh, this looks interesting - so you can actually have L2ARC host DDTs? This may be the sole reason why you'd want to mirror your L2ARC.

More precisely, you can force the DDT to remain in L2ARC no matter what (it is a prime candidate for ARC or L2ARC anyway).
There is still no need to mirror: The L2ARC caches a copy of the actual table in the pool, so it speeds up reads but writes still have to go to the master DDT in data vdevs. Benefits are that it is safe (L2ARC is performance-critical but not pool-critical), easy and reversible, since L2ARC can always be removed.

But the dedicated dedup special vdev still seems like the better way to go, if you don't need L2ARC.

Special vdevs accelerate reads and writes but are pool-critical, so require mirrors (quite possibly 3-way or more), and are an irreversible design decision if the bulk storage is on raidz, as would be expected if one is space-conscious.

 

[dxun]

More precisely, you can force the DDT to remain in L2ARC no matter what (it is a prime candidate for ARC or L2ARC anyway).
There is still no need to mirror: The L2ARC caches a copy of the actual table in the pool, so it speeds up reads but writes still have to go to the master DDT in data vdevs. Benefits are that it is safe (L2ARC is performance-critical but not pool-critical), easy and reversible, since L2ARC can always be removed.

Special vdevs accelerate reads and writes but are pool-critical, so require mirrors (quite possibly 3-way or more), and are an irreversible design decision if the bulk storage is on raidz, as would be expected if one is space-conscious.

Interesting - so the simplest and most performant way is to just host your [block storage] pool on Optanes :) With the on-going fire sale, this might be an actually viable (and cheaper) option. To summarise, here are the options with deduping:

DDT location	Dedup Reads accelerated?	Dedup Writes accelerated?	Resilience
vdev	N	N	Same as pool
L2ARC	Y	N	Same as pool
special vdev	Y	Y	MIN (vdev, special vdev)

 

[Etorix]

Correct, with the last item being: "Resilience: The weakest of data or special vdevs".
You don't want the special vdev to be an obvious weakest link, but a 4-way mirror as special vdev is of no use if the data is on 20 TB HDDs in 15-wide raidz1 vdevs…

 

[dxun]

Thanks, good call - updated the post.

 

[Arwen]

@Davide Zanon - Your database and pre-compressed files are the use cases where neither snapshots nor de-dup will help. A single bit different, and that block won't match the prior block, and the ZFS checksum would be different, so no de-dupping those blocks.

Of course if the pre-compressed files are from the same source blocks, and the compression algo is the same, and their is no metadata stored in compressed file, (like date / time, or such), then maybe the pre-compressed files will be de-dupable or snapshot friendly. Probably same for the database files.

 

[ChrisRJ]

Just to reiterate what @Arwen has written about snapshots and backups. This is exactly what I had in mind! It is an extremely easy and elegant way to handle backups in the sense of being able to restore older versions of a certain file, as long as the underlying storage is there. (Also a great approach to deal with encryption attacks.)

Of course, as far as I understand the situation, deduplication would be able to achieve the same result (not better in any way, though). But the amount of resources thrown at the problem would be bigger. Plus the sizing will be considerably more complex.

I have spent more than 30 years on the subject of backup (and more importantly restore), 25 of those in a professional context. And if there is one thing I have learned, than it is that simpler is better. Because less complexity reduces the risk for human error in critical and by that stressful situations.

 

[Davide Zanon]

What if I need to restore a full backup? Right now I will have to restore the last full backup taken and then restore
all the subsequent differential backup, but since I take a full backup every week the labour is not that much.
With this snapshots approach, if I understood correctly, I have only one initial full backup and from then on I will
have hundreds of differential backups to go through to restore everything, right? am I missing something?

 

[Etorix]

You restore the last snapshot, which includes by reference everything that is still current in all previous snapshots.

 

[ChrisRJ]

With this snapshots approach, if I understood correctly, I have only one initial full backup and from then on I will
have hundreds of differential backups to go through to restore everything, right? am I missing something?

It is sort-of the other way around and for this scenario even easier.

Let's make a concrete example with a timeline:

• Start:
  • No backup or snapshot yet
  • Target directory for a given machine is /mnt/pool/backup/machine1 and is currently empty
• Initial backup
  • By whatever means (let's assume rsync here) all files eligible for backup will be copied from machine1 to the target directory
  • To keep things simple let's say there are now 3 files:
    • file1.docx
    • file2.xlsx
    • file3.py
• Creation of initial snapshot for target directory
  • This is the baseline and conceptually comparable to a snapshot in e.g. ESXi. In other words, until something changes, it consumes more or less no storage space. Yes, some "metadata" will be stored, but not an amount that is worth talking about.
• Second backup
  • Let's assume that file file3.py was changed on machine1. So a traditional incremental backup job would copy only this file over. This is what rsync would also do here.
• So how are things looking now?
  • The target directory contains all files in their current version. Its contents are the equivalent of a full backup, although the job only copied over the equivalent of an incremental backup (i.e. the changed file).
  • The snapshot now "contains" the original version of file3.py, whereas the target directory contains the new one.
• Restore considerations:
  • For a disaster recovery you simply use the target directory.
  • For getting back a specific version, you go to the respective snapshot. Comparable to pulling out the tape from last Wednesday and restoring the file from there.

Hope that makes sense. There are some additional thoughts, but they are more on the conceptual/theoretical side and would likely confuse rather than help now ;-)

 

[Davide Zanon]

It is sort-of the other way around and for this scenario even easier.

Let's make a concrete example with a timeline:

So to put this in my scenario:
1) define a dedicated dataset for fileserver backup
2) make first full backup
3) take a snapshot
4) make diff backup
5) take a snapshot
6) repeat 4-5 every n our/day/whatever
Correct?

 

[Arwen]

So to put this in my scenario:
1) define a dedicated dataset for fileserver backup
2) make first full backup
3) take a snapshot
4) make diff backup
5) take a snapshot
6) repeat 4-5 every n our/day/whatever
Correct?

I'd describe it a little different:

One time setup per CLIENT:
a) define a dedicated dataset for CLIENT backup, (like pool/backup/CLIENT)
b) make first backup, which would act as a full since the new dataset is empty
c) take a snapshot with date name, (like zfs snap pool/backup/CLIENT@YYYYMMDD)

Normal backups:
1) make backup
2) take a snapshot with date name
3) repeat nightly if that is your incremental schedule

No need to consider a difference for "full" verses "incremental" when using a program like RSync.

Using the date the backup was taken for the snapshot name simplifies finding it. ZFS does store "creation" date & time inside the snapshot attributes. But with the CLIENT name as the sub-dataset name, their is little reason not to name the snapshots with the date.

For restore, simply choose the appropriate snapshot based on date. Or you can use the main dataset which has current, (as does the most recent snapshot). All snapshots would reflect the exact state of the source CLIENT at the time of the backup. So all snapshots would be "fulls".

Of course, RSync and other backup programs can be told to ignore certain paths or files. Thus, they would not be in the backups, (or backup snapshots), if that feature were used.

One last thing. If your pool gets fullish, then you can start trimming the oldest backup snapshots. Thus, unless you want to restrict the backups to 30 days, you can just keep collecting ZFS snapshots. I once wrote an auto-delete program for log files to keep a file system below a specific usage. Something similar can be done for ZFS & old snapshots.

None of this is intended to dissuade you from using ZFS De-Dup. This is just an alternative. (And may help other people in the future.)