Deduplication after volume creation?

[Havock2]

I'm planning my first FreeNAS. I have a question about deduplication. Can it be added to a volume after it has been created and is in use? I would like to save some money on the initial build and add memory and dedup later, if possible.

 

[Ericloewe]

Yes, but the likelihood of it being a good idea is close to zero.

Note that only new data gets deduped.

 

[gpsguy]

Here's a quote from the manual "If you plan to use ZFS deduplication, ensure you have at least 5 GB RAM per TB of storage to be deduplicated."

Many users don't heed this advice and end up in trouble. Two weeks ago we had a fellow that ran into this problem. He had to task his hard disks to work in order to find a machine that had sufficient RAM to load his pool. He didn't discover the problem until he shut his server down and couldn't bring it back up.

 

[Havock2]

I plan to have 12TB of storage. Will 64GB of Registered ECC memory be enough?

 

[Ericloewe]

I plan to have 12TB of storage. Will 64GB of Registered ECC memory be enough?

Nobody can give you a definite answer. You could need much more or significantly less.
Again, the marginal benefits of dedup are only very rarely greater than the significant risks and performance issues. Compression gets you 50% of the benefits (luckily, redundant sets of data also tend to be compressible) at zero cost.

 

[Havock2]

How will I be able to tell if I have enough memory?

 

[gpsguy]

For some information on the topic, see jgreco's response in message #11 & 12 in this thread -

https://forums.freenas.org/index.ph...dware-goals-before-i-start.42613/#post-277338

How will I be able to tell if I have enough memory?

 

[jgreco]

I plan to have 12TB of storage. Will 64GB of Registered ECC memory be enough?

You plan to have ... 12TB of raw storage? 12TB of pool space? 12TB of usable pool space (60% rule)? 12TB of data stored after dedup gets done with it? 12TB of data stored before giving it to dedup?

These are each very different things.

 

[Havock2]

I will have 12TB in a vdev that will be mainly used for my Plex libraries. Right now, I have just over 3TB to put there.

What is the 60% rule?

 

[cyberjock]

Plex libraries won't dedup well... you're better off avoiding kicking a hornet's nest and not using dedup.