Dedup on. What should I be seeing?

[praecorloth]

So I was experimenting with deduplication last night. Just doing some tests to see what kind of CPU and memory overhead I could expect. Unfortunately I couldn't get dedup to do anything. So here's what I did.

I fired up a fresh install of FreeNAS on a server with a pair of quad core Xeons and 8GB of memory. Not a ton of memory, but should be enough for some function testing. Made a RAIDZ of three 250GB SAS drives and set up a Samba share. I copied over a few video files which came out to about 11GB. I turned deduplication on and was watching the CPU usage to see if it was doing anything. It was not.

So then I made a new directory and copied the same 11GB of data. No extra CPU usage. No extraordinary amount of memory usage. When the copy was done, about 22GB of space was used. So I deleted the extra directory, and then rebooted the FreeNAS machine in case it needed a reboot to enable dedup.

I copied over the same 11GB of data, with the same result. I created a third directory and copied the same data. Still no extra CPU usage, no extra memory usage, and 33GB of space used on the share.


So while doing all of this, my expectations were that I would see CPU and memory usage that was out of the ordinary, and that when running a df -h against the RAIDZ, that only the space actually used would appear. Am I incorrect in my expectations? What should I be seeing with dedup on, and how can I test to make sure it's working properly?

 

[ser_rhaegar]

You need to use this command to check space:

Code:

zfs list -o space

I don't believe df will show proper results with dedupe.

 

[warri]

Code:

zpool list

will give you a dedup rate as well.

 

[gpsguy]

Be careful with deduplication. The memory thumbrule calls for at least 5Gb of RAM for each Tb of deduplicated data. Read the warnings about deduplication in the manual.

 

[praecorloth]

Excellent, thank you everyone! I knew I had to be missing something. Also, yes, I want to approach dedup carefully. That's why I'm doing these tests. I want to throw various scenarios at it and see how various hardware performs with dedup on.