RAM needed for compression gzip9 + dedup

[prescott]

Hello.

I'm building FreeNAS with Supermicro 1150 with a Xeon and 32GB ECC RAM. In this build there's going to be 6x 4TB WD Black in raidz2 and I have a few questions before I finish it completely. The server is going to be used at home with 5 family users, with almost never high loads.

I have Dual-WAN at home, so that is going to answer alot of questions regarding dedup (torrenting).

1) How much RAM does dedup AND compression need in my situation if I can't go over 32GB? I'm thinking dual seedboxes with only 1 data written onto the array. EDIT: "I think" in this subject I'm going for "heavy dedup", like 6TB+ torrents dedup-ed...

2) Can 1) be solved with L2ARC?

3) If it can be solved with L2ARC, does it need to be SSD? I have few 160-320GB HDDs laying around and I could put them in just for that. Performance of the array is not that important!

4) Maybe a hard one: I'm building a very self-"maintained" server, for that I want the OS drive (HDD) to be in RAID1. I'm thinking Supermicro X10SL7-F with that 2 ports in RAID1 just for the OS drive. Question: What is stored on the OS drive and not on the arrays? Specifically, if the OS drive fails, what should I then restore? Samba shares? What configuration should then be restored? I hope I'm very clear on this subject...

EDIT2:
5) In this particular situation I'm also going to use another server (Hyper-V) which is going to run a few VMs. This server will have all VMs stored on the FreeNAS for "data security". I was thinking on doing iSCSI, but I'm also asking you guys, what is better in case of a power failure: iSCSI or classic SAMBA for this?

The server is going to be used like forever till it dies, so when I make the first build it must be perfect, I hope you understand :-(

Thank you!
Luka

 

[cyberjock]

WOW.. you are scaring me with what you are writing.

What does "dedup" and "torrenting" have to do with each other in the same sentence? Just reading that makes me think you have no understanding of what dedup is or how it works.

1. dedup needs about 5GB of RAM per TB of storage. And that's just a thumbrule. There is no upper limit. If you ever have an unclean dismount of the pool you will need however much RAM(not l2arc, not zil, only RAM) is necessary to hold the ddt. If you don't have enough, your pool doesn't mount. This is why I tell people that use the word dedup they are playing with fire, that they are borderline irresponsible with their choices, etc. You will have to get enough ram for your ddt before you will see your data again. Considering you have 32GB of RAM, if you want to do more than 4-6TB of disk space I would not even consider dedup unless you are willing to spend $1k+ on just RAM. This is why dedup is outrageously expensive for us mere mortals. :(
2. not unless something changed recently. Just look at the warnings against dedup in the freenas manual.
3. no, you'll want SSDs. using hard drives will make performance so low you will be unable to use it.
4. FreeNAS is designed for a USB stick. Not a hard drive, not a RAID1. A USB stick. If you want to do your own thing, all I can say is "good luck". For how to restore, please consult the manual at doc.freenas.org. (in fact, if you check out the manual, all of this except #3 is answered in there).
5. support for hyper-v here is almost non-existent. I had no idea hyper-v even supported samba.. so my answer is /shrug.

 

[prescott]

I understand it was designed for a USB stick. Does USB stick have SMART attributes? Does it warn you a little bit before completely dying? I had several USB sticks dying on me without warning, that's why I'm overly cautious about this. If it dies, what everything is stored on it that has to be configured again?

 

[prescott]

Dedup and torrenting:

I have 100x 40GB or more torrents and dual WAN. Instead of load balancing on router itself I want to make two seperate torrent seedboxes with two identical torrents, thats 2x 100x 40GB. And there comes dedup to the rescue for 1x written files instead of 2x....

 

[prescott]

Hyperv:

I know exactly where support for hyperv is..

FreeNAS server (bare metal #1) + HyperV (bare metal #2) =>
Bare metal's #2 virtual machine files/configuration and virtual hard drives are stored over LAN on bare metal's #1 ZFS array. The question is: what's better way to do it, iSCSI or CIFS, incase there's a power failure on either and on both sides? Does the zvol get damaged in cases of power failures?

 

[crumbz]

Dedup and torrenting:

I have 100x 40GB or more torrents and dual WAN. Instead of load balancing on router itself I want to make two seperate torrent seedboxes with two identical torrents, thats 2x 100x 40GB. And there comes dedup to the rescue for 1x written files instead of 2x....

Why can't both your seedboxes point to the same backend storage (FreeNAS)?

 

[prescott]

Why can't both your seedboxes point to the same backend storage (FreeNAS)?

I will try that, but don't think its possible, because files might get locked by one instance and cant use them on other instances... That's where dedup would work magically!

 

[prescott]

So what do you lose when USB stick dies? On Windows' Storage Spaces everything is stored on the drives: array info, file permissions... Except samba shares must be done over again if you put all the drives in another machine. What's the story with ZFS and FreeNAS?

 

[crumbz]

I will try that, but don't think its possible, because files might get locked by one instance and cant use them on other instances... That's where dedup would work magically!

That depends on how your seedboxes access the files stored on your FreeNAS server. SMB, FTP, NFS, or something else? I'm not an expert on this subject, but I would try everything else before deduplication. I sounds nice in theory, but as previously stated by cyberjock: you are playing with fire.

So what do you lose when USB stick dies? On Windows' Storage Spaces everything is stored on the drives: array info, file permissions... Except samba shares must be done over again if you put all the drives in another machine. What's the story with ZFS and FreeNAS?

Keep a backup of the FreeNAS configuration somewhere safe, and you shouldn't lose anything. Just reinstall FreeNAS on a fresh USB stick, auto-import your volumes and restore your configuration from backup.

 

[cyberjock]

Keep a backup of the FreeNAS configuration somewhere safe, and you shouldn't lose anything. Just reinstall FreeNAS on a fresh USB stick, auto-import your volumes and restore your configuration from backup.

Actually, you don't even have to import your volumes. Simply upload the config file and you are done!

 

[no_connection]

I would use more storage instead of dedup if it is needed.

Are you sure the torrent client can not handle dual WAN?
Load balancing would be rudimentary but with enough clients it should still work out to saturate both connections. And would not be that much more unbalanced than running two clients at the same time.

As far as I can see the files are not locked when seeding, so you should only experience some extra load when client #2 rechecks all data before it starts seeding.
Both clients can obviously not download at the same time, and dedup would not solve that.