From DrivePool/SanpRAID to FreeNAS

[kayot]

Background - I have been building a hard disk set for about nine years. Back when I started I had no idea what RAID was. By the time I learned, my disks were near full and I was dumping my whole paycheck on school loans. So I created two sets, one was a backup and one was the main. I used Dynamic Disks due to a lack of knowledge.

After losing the array three times due to MS's poor implementation and my own ignorance, I went with a DrivePool/SnapRAID setup which I've been using till this day. It was inexpensive and did not require a backup set while allowing the set to grow faster then my budget could handle it. The primary downsides were a lack of Hard Link support and no ability to use any sort of advance filesystem features.

Now I'm sitting here with 12 2TB drives with 9 for data and 3 for SnapRAID parity.

I'm looking into purchasing 4 6TB WD Green drives WD60EZRX (with wdidle set to max) or 4 6TB Red drives WD60EFRX and using the 2TB drives as a backup solution. I want to use a 90GB SSD for the De-duplication record. The majority of the total space used is large files like movies.

I've used Linux (Debian[Mint]/Arch), but not FreeBSD and by extension FreeNAS.

System Specs of the NAS:
AMD FX 4130 Black Edition 3.8GHz Quad-Core Socket AM3+ Boxed Processor
ASUS M5A78L-M LX PLUS Socket AM3+ 760G mATX AMD Motherboard
Crucial Ballistix Sport 8GB DDR3 1600 MT/s (PC3-12800) CL9 @1.5V UDIMM 240-Pin Memory
OCZ Technology Vertex 3 SSD Series 90GB 2.5" Solid State Drive

I plan to use this for archive usage. The array will never experience long term stress. I don't want anything besides the De-duplication data (and if possible, the OS) on the SSD to save wear-writes. I plan to re-silver the array to make it bigger. It won't grow beyond four/five disks; what ever I start with.

My questions are as follows:

- Is a RAID-Z1 with monthly/biweekly backups good enough for the four disk set, or should I really consider a 5 disk RAID-Z2?
- Should I bother with a dedicated RAID Card with JBOD or is the cheap Motherboard good enough for my needs?
- What Backup Software should I use on the 2TB disk set?
- If I lose a drive and there is an Unrecoverable Read Error on another disk during a repair, do I lose the set or will it rebuild around it (so I can avoid a full backup recovery by just re-silvering a replacement drive and replacing the bad files from backup)?
- Should I bother with De-duplication? Is it worth it and will the SSD keep up with it?
-- I know De-duplication is a complicated question. I'm wondering; with a mixed archive of movies, music, and games, is it worth the overhead on the SSD. Please, only people that have done this. I've read of ton of speculation and doom-saying and I want real world results.
- Does ZFS Support Hard Links? (In case De-duplication isn't possible)

I'm making a vbox to test out possible scenarios so I can answer the rest of my questions.

 

[dlavigne]

Answers to some of your questions below. Note that most of your questions are "depends on you and what you can tolerate wrt downtime".

- Is a RAID-Z1 with monthly/biweekly backups good enough for the four disk set, or should I really consider a 5 disk RAID-Z2?

- If I lose a drive and there is an Unrecoverable Read Error on another disk during a repair, do I lose the set or will it rebuild around it (so I can avoid a full backup recovery by just re-silvering a replacement drive and replacing the bad files from backup)?

That's up to you and what you can tolerate. RAIDZ1 will provide more capacity but if you lose 2 disks or a second disk dies during resilvering, it's backup time...

- Should I bother with De-duplication? Is it worth it and will the SSD keep up with it?

IMHO, deduplication is never the right answer :)

- Does ZFS Support Hard Links? (In case De-duplication isn't possible)

What are you hoping to gain with hard links? Snapshots and clones might be what you want.

 

[Ericloewe]

The fact that the requirements for deduplication state "at least 5GB of RAM per TB of dedpulicated data, with no upper bound" should be enough to scare you away from it. If it isn't, read that again and let it sink in.

 

[depasseg]

If I were you, I would use RAIDZ1 on the new NAS and use daily (or even more frequent) ZFS replication to your 12x2TB drive array, but that assumes that you are going to run FreeNAS on those as well. Is that part of your plan? If not, then ZFS replication isn't possible and you'll need to look into Rsync.

I'd avoid de-duplication. It's not worth it for the type of data you have.

If you are using RAIDZ1 and lose a drive, you can correct file errors if the metadata copies=2 is set, if there aren't 2 copies of metadata, then you will get a read error and need to restore from backup, the resilvering is not affected however.

I don't know the other answers. Do a search for you MB model. There are other users using it.

 

[kayot]

I was thinking about using a Solid State drive to handle the absurd ram requirements on de-duplication. The reason I asked about hardlinks is because I often put a file in two places and it causes duplication issues. I'm not so much as interested in block level de-duplication as file de-duplication. It just seems that it wont do file level (at least automatically) and the only way to do that would be to write a script that will go and do it for me.

When using ext4 I used a script that would hash the whole system, find duplicates, deep level compare, remove the offending file, and then hardlink a replacement. It was time consuming, but effective. I only bothered running it once a month since the duplication isn't crippling, just annoying and preventable.

My 12-2TB drives are going to be used to keep a backup in case I lose the array. I have no intention of putting these disks together if I can help it. What I would like is software that will fill a drive to the brim and keep note of what is on each drive. As I removed/add files to the array, the backup software would keep track of this and during the next sync it would have me cycle the drives one at a time for file removal and addition. It will also note duplicate files and only store them once. If such software doesn't exist, I'll have to make it. I have an idea how I would go about doing so, but would prefer a solution that already exists.

Questions:
Does ZFS have a mode that will notice two identical files and only store them once?