Deduplication Stats

[douglasg14b]

I'm trying to figure out how to see the effectiveness of my deduplication (On VM backups). zpool status seems to give helpful output, but I need some help interpreting the results. I also want to figure out why the Storage UI seems to show the pre-dedup size?

Code:

zpool status -D Data_Drive

 dedup: DDT entries 1249263, size 904 on disk, 291 in core

bucket			  allocated					   referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
	 1	12.5K   1.55G	790M	790M	12.5K   1.55G	790M	790M
	 2	1.15M	147G   86.7G   86.7G	2.30M	294G	173G	173G
	 4	26.2K   3.28G   1.85G   1.85G	 120K   15.0G   8.59G   8.59G
	 8	3.90K	499M	298M	298M	38.5K   4.81G   2.89G   2.89G
	16	1.69K   216M	148M	148M	32.5K   4.06G   2.78G   2.78G
	32	 429   53.6M   34.3M   34.3M	15.6K   1.95G   1.24G   1.24G
	64	  15   1.88M	368K	368K	1.26K	162M   33.6M   33.6M
   128		6	768K	240K	240K	1.19K	153M   49.9M   49.9M
   256		4	512K	252K	252K	1.10K	141M   66.2M   66.2M
   512		1	128K	   4K	  4K	  990	 124M   3.87M   3.87M
 Total	 1.19M	152G   89.8G   89.8G	2.51M	322G	190G	190G

What does each column represent? What do I need to look at to determine the dedup effectiveness?

In the FreeNAS web UI, it shows that 190GB of space is used on disk, is this the amount actually used? From some tests, it seems that the UI shows the amount of space the data would take up pre-deduplication. If I had a 1TB drive, and added 1TB of data that was deduped down to 200GB, can I still add more without ZFS getting upset, even when the UI shows that 1TB is used?

 

[Nick2253]

Your dedup seems to be doing well. To figure dedup ratio, you're looking at the LSIZE referenced (total amount of data) vs the LSIZE allocated (total on disk), in the "total" row. That points to a dedup ratio of 50%.

 

[douglasg14b]

Your dedup seems to be doing well. To figure dedup ratio, you're looking at the LSIZE referenced (total amount of data) vs the LSIZE allocated (total on disk), in the "total" row. That points to a dedup ratio of 50%.

Thanks for the info, any idea why the FreeNAS UI shows 190GB used for that dataset if it's 152GB?

 

[Nick2253]

Can you run just zpool status and post the output here?

 

[wblock]

Note: put output in code tags and it will line up correctly. I tried to fix the table in post #1 with limited success.

 

[pro lamer]

referenced (total amount of data) vs the LSIZE allocated (total on disk)

Isn't it the opposite? Allocated values in the OP are bigger than referenced...

EDIT: and here too: http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-113-size-zfs-dedup-1354231.html

 

[kdragon75]

What are you using for backup? any good backup software should offer dedupe on your backups. It may not offer cross VM dedupe but even with just lz4 compression I get 1.4 ratios on my VMs.