What are the usage scenarios for ZFS de-dupe?

[Linkman]

Just curious, not something I'm going to bump into either at home or at work, but what are the scenarios where using ZFS de-duplication is warranted? Huge amounts of RAM are needed, so does it save enough disc space to be worthwhile, or I/O volume, or what?

 

[DifferentStrokes]

Maybe a Web hosting server with 1000 WordPress installs running for their different customers.

 

[dlavigne]

Also, consider that dedup pre-dates LZ4 by some years.

 

[titan_rw]

Or those of us that like to tinker:

Code:

DDT-sha256-zap-duplicate: 4689269 entries, size 1174 on disk, 189 in core
DDT-sha256-zap-unique: 975227 entries, size 2498 on disk, 403 in core

DDT histogram (aggregated over all DDTs):

bucket  allocated  referenced
______  ______________________________  ______________________________
refcnt  blocks  LSIZE  PSIZE  DSIZE  blocks  LSIZE  PSIZE  DSIZE
------  ------  -----  -----  -----  ------  -----  -----  -----
  1  952K  119G  78.1G  80.2G  952K  119G  78.1G  80.2G
  2  418K  52.2G  38.5G  39.4G  1004K  126G  92.1G  94.3G
  4  623K  77.9G  57.0G  58.4G  3.04M  389G  290G  297G
  8  353K  44.1G  30.9G  31.8G  4.19M  537G  371G  382G
  16  416K  52.0G  40.7G  41.6G  8.70M  1.09T  863G  882G
  32  502K  62.8G  55.7G  56.2G  23.1M  2.88T  2.58T  2.60T
  64  466K  58.2G  50.7G  51.3G  42.3M  5.28T  4.65T  4.70T
  128  1.75M  225G  206G  207G  307M  38.4T  35.4T  35.6T
  256  4.67K  598M  288M  310M  1.53M  196G  94.3G  102G
  512  340  42.4M  27.4M  28.6M  228K  28.4G  18.3G  19.1G
  1K  26  3.25M  2.31M  2.35M  34.2K  4.28G  2.88G  2.94G
  2K  10  1.25M  219K  280K  25.9K  3.24G  509M  668M
  4K  18  2.25M  1.18M  1.30M  80.2K  10.0G  5.14G  5.66G
  16K  1  128K  1K  7.99K  26.4K  3.30G  26.4M  211M
Total  5.40M  691G  558G  567G  392M  49.0T  44.4T  44.7T

dedup = 80.87, compress = 1.10, copies = 1.01, dedup * compress / copies = 88.58

This is a backup destination for my vmware host. Each day it copies the full vmdk's over the network to freenas. With dedupe, only the blocks that change need to be stored on disk.

I don't really need to store so many deduped full copies of VM's, but it's cool seeing freenas dedupe it all. And yes, I keep an eye on the total number of deduped blocks so as to watch ram usage. I have 64 gigs of ram, so I'm comfortable with it so far.