What is the state of deduplication in 2023?

[viniciusferrao]

Hello, I'm considering enabling deduplication on a new pool that will serve some Linux repositories. Since there's a lot of common packages it would benefit from deduplication.

Unfortunately I only had the ideia after mirroring almost 9TB of data, so if I would go further with the ideia I'll need to redownload everything.

With that being said, I've done some testes with zdb and it seems that I can increase the space efficiency to up to 1.33x:

Code:

root@truenas-repos[/mnt]# zdb -U /data/zfs/zpool.cache -S repos0
Simulated DDT histogram:

bucket              allocated                       referenced         
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    45.3M   5.61T   5.48T   5.48T    45.3M   5.61T   5.48T   5.48T
     2    10.3M   1.27T   1.24T   1.24T    21.4M   2.64T   2.57T   2.57T
     4    1.27M    160G    156G    156G    5.57M    704G    686G    686G
     8     142K   17.7G   17.4G   17.4G    1.19M    152G    149G    149G
    16    8.48K   1.04G   1.00G   1.00G     167K   20.6G   19.7G   19.7G
    32    2.79K    355M    331M    331M     132K   16.4G   15.3G   15.4G
    64      999    124M    123M    123M    77.1K   9.57G   9.48G   9.48G
   128        9    774K    350K    370K    1.36K    113M   46.7M   50.1M
   256        1    128K      4K   6.73K      510   63.8M   1.99M   3.35M
 Total    57.0M   7.06T   6.89T   6.89T    73.8M   9.13T   8.91T   8.91T

dedup = 1.29, compress = 1.02, copies = 1.00, dedup * compress / copies = 1.33

I will also disable the zstd compression since it does not save anything and will stick with lz4.

If my calculations are correct I will need roughly 17,81GB of system RAM to hold the deduplication table in the current state of the data. The system has a total of 32GB of RAM which will compromise severely the available RAM, but I'm already running in a scenario where the disk space is already at 75%

Code:

root@truenas-repos[/mnt]# zfs list
NAME                                                      USED  AVAIL     REFER  MOUNTPOINT
boot-pool                                                2.65G  25.0G       96K  none
boot-pool/ROOT                                           2.64G  25.0G       96K  none
boot-pool/ROOT/22.12.2                                   2.64G  25.0G     2.63G  legacy
boot-pool/ROOT/Initial-Install                              8K  25.0G     2.63G  /
boot-pool/grub                                           8.20M  25.0G     8.20M  legacy
repos0                                                   8.97T  3.14T      209K  /mnt/repos0
repos0/.system                                            712M  3.14T      645M  legacy
repos0/.system/configs-40cc91dacae0491e84781ab81ded8ba4  2.64M  3.14T     2.64M  legacy
repos0/.system/cores                                      162K  1024M      162K  legacy
repos0/.system/ctdb_shared_vol                            162K  3.14T      162K  legacy
repos0/.system/glusterd                                   175K  3.14T      175K  legacy
repos0/.system/rrd-40cc91dacae0491e84781ab81ded8ba4      34.9M  3.14T     34.9M  legacy
repos0/.system/samba4                                     875K  3.14T      875K  legacy
repos0/.system/services                                   162K  3.14T      162K  legacy
repos0/.system/syslog-40cc91dacae0491e84781ab81ded8ba4   27.7M  3.14T     27.7M  legacy
repos0/.system/webui                                      162K  3.14T      162K  legacy
repos0/repos                                             8.97T  3.14T     8.97T  /mnt/repos0/repos

With that data I would like to hear the opinions about deduplication.

The question is: what is the state of deduplication today? In the past if we didn't have enough RAM to read the deduplication table, the pool cannot be mounted and effectively losing its data. Is this still true? Is deduplication worth in my scenario? It would reduce 33% of used disk space which seems good to me.

Please consider that:
* Performance is not that important. It cannot be neglected however, but since it's a local mirror it does not need to be extremely fast, being faster than the WAN is sufficient.
* I don't have nay dedicated SSDs on this pool nor I can add to the system.
* There's no other server to hold the data and I don't have the resources ($$$) to increase disk size.
* It's a single RAID-Z pool with 8x 2TB NL-SAS disks. Single parity.
* Running the latest version of SCALE: TrueNAS-SCALE-22.12.2

Thanks in advance for any opinions on this matter.

 

[blanchet]

ZFS deduplication does not work well.

• it will eat all your memory and ruin the performance
• it will increase the time to import the pool when booting
• the deduplication ratio is low

I have tried several time to use ZFS deduplication, and I have finally abandoned.

 

[Etorix]

The state is still the same: Deduplication is a RAM hog which is best not considered.
If considered anyway, the absolute minimum for RAM would be at least 64 GB—for CORE; make that 128 GB for SCALE because of the crippling behaviour of the kernel allocator. More would be even better.
A typical rule-of-thumb of 5 GB/TB would give 48 GB RAM to deduplicate 9 TB of data. I don't know how you came with 17.8 GB but the figure
doesn't feel right.

You do not have enough RAM, and a 1.33 ratio is frankly ridiculous. If it were 5 or 6 and you had at least 64 GB RAM, we might discuss.

I'm already running in a scenario where the disk space is already at 75%

I understand that the situation is desperate, but there's really no alternative here to increasing storage or deleting data.

 

[jgreco]

With that data I would like to hear the opinions about deduplication.

Would your data be suitable for file-level deduplication? This is cheap and super-easy, without any significant requirements. I like Phil Karn's dupmerge (look for "KA9Q dupmerge" to find the source code).

The code removes duplicate files and replaces copies other than the first with a hardlink back to the first. However, it isn't automatic, you have to run something like a script to cause it to do the deduplication. It also relies on you not changing the contents of the files, so it is mostly useful for archival file access.

Code:

#! /bin/sh -

find /mnt/storage0/wininstall/old -type f -print0 | /mnt/storage0/wininstall/bin/dupmerge -0

I have a bunch of versions of our Windows install repository that is many gigabytes large. This eliminates all the redundant files and sucks down more than a terabyte of space into less than 100GB. The main problem with this style of deduplication is that if you are going to make any changes in the file tree, you MUST REMOVE the file in question and then replace it. If you open an existing file for write and then write stuff to it, it will cause ALL the copies of the file to change, and that almost certainly isn't what you intended.

 

[Arwen]

Here is some suggested reading;

My experiments in building a home server capable of handling fast + consistent deduplication

AIM: To help people looking at deduplication on TrueNAS 12+, what I've found on the way making it work on mine. On sustained mixed loads, such as 50GB+ file copies and multiple transfers, using TrueNAS 12 with a deduped pool and default config...

www.truenas.com

Now I did write a resource giving misc. information, mostly discouraging people. But, it never the less has useful information both to help implementation and as a warning. If you have your heart set on ZFS De-Dup, then you might as well skip and read the above resource.

Resource - ZFS de-Duplication - Or why you shouldn't use de-dup

The TrueNAS forums occasionally have people who come across ZFS de-duplicate, and want to investigate its use. Or think it is a good idea, and want to implement it. Here are some suggested configuration details: Understand that you need CPU...

www.truenas.com

Good luck in what ever you decide.

 

[Whattteva]

If the repositories stay fairly static, why don't you just use zfs clone instead? It costs nothing in performance for the same effect. Of course, the problem is if they change a lot, then you will accumulate diff data overtime. Also, you wont be able to delete that snapshot without deleting all the ones that depend on it first.

 

[viniciusferrao]

Guys, thanks for all the answers, I'll follow your advices. @jgreco suggested a file level deduplication and yes, that definitely worth it.

I'll check it and probably use it.

Thank you all.

 

[winnielinnie]

There's also "Czkawka", which has a GUI and command-line version. You'd have to run it locally (on your client, and later rsync to the server with hardlinks preserved) or run it directly on the server. The former method you can use the user-friendly GUI. There's a port for it on FreeBSD (i.e, run it under a jail in TrueNAS Core, and grab it via "pkg install czkawka").

Not sure about SCALE, though.

 

[celinedion]

@mapquest said
Guys, thanks for all the answers, I'll follow your advices. @jgreco suggested a file level deduplication and yes, that definitely worth it.
I'll check it and probably use it.

Thank you all.

While modern implementations handle deduplication table loading better, there's still a potential risk if the table becomes too large.

 

[Whattteva]

While modern implementations handle deduplication table loading better, there's still a potential risk if the table becomes too large.

@jgreco solution isn't an actual dedup. It's just a bunch of hard links... I don't remember 100%, but I think this probably also assumes it's working on the same dataset because I don't think you can make hard links across different file systems.

 

[MrTyton]

Hi,

I'm in a semi-similar position, where I want to effectively dedup by creating a lot of hardlinks.

However, I can't seem to get the dupmerge to run. The shell (running Truenas Scale 22.12.3.3) doesn't have an apt, doesn't have a yum, doesn't have anything like that. Can't compile the script because of that, can't compile it on another system and run it there because it doesn't have all the libraries. Don't see it in applications, or anything like that.

Is there something stupid that I'm missing? How exacctly am I supposed to run that from within Truenas? Or is the solution to run it from a different linux box and hope that the host system hardlinks actually change?

Thanks.

 

[Etorix]

Run the deduplication utility in a container, VM or "jail".