tl-dnr:
De-dup really is a specialized use case. If you think you need it, you don't. You will know when you need it and can actually implement it wisely.
There have been a few posts about using de-dup. So, let's first get some things out of the way;
De-dup is a memory hog, as the de-dup...