ZFS Deduplication on and off supported

[dirkme]

Dear FreeNAS Friends,

what I mean by that is, if I try ZFS Deduplication and would feel it takes too much of performance, can i then switch it back to off without losing data?

As much as I understand ZFS Deduplication makes kind of a link to a duplicate and if I switch back to off, will that link be replaced again with a file?

 

[Thomas102]

Hi,
No it won't be. the "links" won't be replaced and it may not help performance.
New data copied to the disk won't use deduplication. you may remove dedup on a file by copying it but not sure it helps performance either

 

[dirkme]

Hi,
No it won't be. the "links" won't be replaced and it may not help performance.
New data copied to the disk won't use deduplication. you may remove dedup on a file by copying it but not sure it helps performance either

The idea was like on NextCloud, if a Guy shares 1 file to 10 users, it would store only one file on the storage, thinking if you then switch off dedup, would those 10 users still have the file in their NextCloud? Otherwise you switch it on, you are stuck or you have to move the data to a new dataset, but you would still lose duplicated files which were in multiple folders, sigh.

 

[Thomas102]

switching off dedup does not change anything to the file system.
Deduped files (blocks is more accurate) will stay dedup.
It only apply to data created after switching off.

The key point is that if you have a performance issue, switching off is not enought.
You will have to recreate the pool.

 

[dirkme]

switching off dedup does not change anything to the file system.
Deduped files (blocks is more accurate) will stay dedup.
It only apply to data created after switching off.

The key point is that if you have a performance issue, switching off is not enought.
You will have to recreate the pool.

I will play around on a small dataset and will see what it does to my performance and if it is worth switching it on at all.

I will update this post with my findings, and if anyone has more explanation, advantages or best usage, that would be great. Also, experience reports would be great

 

[pschatz100]

Just curious... Why are you interested in dedup? What kind of data do you plan to store?

 

[dirkme]

Just curious... Why are you interested in dedup? What kind of data do you plan to store?

I have a few 100 MB on photos, some are sorted and some are still in folder and sub-folders.

I trust there are many duplicates and I thought it would save significant space.

However, I tested around with a 99 GB dataset and I saved somewhat of 9 GB at most.

But the performance decrease is massive and in my opinion not worth it.

 

[Jailer]

It takes an insanely large amount of RAM for deduplication on ZFS. That's likely why your experiment showed a significant performance hit.

http://www.oracle.com/technetwork/articles/servers-storage-admin/o11-113-size-zfs-dedup-1354231.html

 

[pschatz100]

Dedupe requires quite a bit of CPU power and lots of memory. Your system may not have sufficient resources to do it effectively.

I'm not really surprised by your results. Dedupe is not designed for media, and generally has limited benefit on groups of files that are already compressed such as mp3, mp4, and jpg.

Thanks for sharing your interesting experiment.

 

[Ericloewe]

I have a few 100 MB on photos, some are sorted and some are still in folder and sub-folders.

I trust there are many duplicates and I thought it would save significant space.

However, I tested around with a 99 GB dataset and I saved somewhat of 9 GB at most.

But the performance decrease is massive and in my opinion not worth it.

You're looking at the wrong tool. Your static set of duplicate-containing data is best suited to a client-side solution.

I've used dupeGuru successfully in the past, and it also has versions that can look for similar images (and audio) even if the files themselves are different (different type, resolution or encoding).

 

[dirkme]

You're looking at the wrong tool. Your static set of duplicate-containing data is best suited to a client-side solution.

I've used dupeGuru successfully in the past, and it also has versions that can look for similar images (and audio) even if the files themselves are different (different type, resolution or encoding).

Is that available for FreeNAS?

 

[Stux]

Is that available for FreeNas?

Seems to be a Ubuntu version... and Ubuntu works in a VM.

 

[Rattlebattle79]

If you ask if it's a good idea to use dedup, don't use it. You have to be at a much higher level as a sysadmin before you start fiddling with dedup. It could have severe consequences if you don't have a good plan and strategy and know exactly what you are doing!

 

[Ericloewe]

If you ask if it's a good idea to use dedup, don't use it. You have to be at a much higher level as a sysadmin before you start fiddling with dedup. It could have severe consequences if you don't have a good plan and strategy and know exactly what you are doing.

Hence my suggestion for a sort of offline (from the server's perspective) dedup.

 

[fracai]

I'm currently using rdfind from a jail to dedupe images.
I've also had good results with duff.