Introducing.. me! (and my future build pending your awesome feedback)

[Dangins89]

Hello FreeNAS Gurus! My name is Dan, I'm an avid IT enthusiast and network engineer, currently pursuing my Masters of Computer Science. I’ve been reading through the forums for months, this is my first time posting! There’s quite a plethora of information to read and digest. I’ve been trying to be a big boy and figure out my build all by myself, but I still have concerns and would really benefit and appreciate any suggestions or validations you can offer me. I’m sorry ahead of time for the length of this post… but appreciate your feed back!

First, some background on my objective and use-case.. From what I’ve been reading, this does not seem to be the typical home usage.. but I feel like FreeNAS is still the right solution for me. People always seem to specify that a RAID is not a BACKUP.. So, if after you read through this, if you have any recommendations on what does equate to a good backup, I’m really interested in knowing! Now, here are my 3 reasons for this build.

Business Purposes:
I have 4 family businesses, at separate locations (all in NJ, USA). Their total combined business data is under 200GB and consists of Word documents, some images and audio files and some quickbooks, access databases, and other basic office documents. At each office location, I have a primary machine that stores all the data and shares out folders to the other office machines for data access. I would like to maintain a copy of all their data within FreeNAS with a snapshot or three for rollbacks and data recovery from user error. In addition, I want to perform Windows Backups for all computers at each location; this is not a function currently implemented, only business data is backed up.

Extended Family Purposes:
Along with the duties of being the family IT guy, I’ve now been dubbed as the precious family memories curator.. Family members will be given a set of instructions to send family related content to me, I will then organize and maintain it within FreeNAS and plan to use Plex or OwnCloud to give family and friends access to this content via the web. I don’t think this will extend beyond 4TB any time soon.. but, who knows!

Personal Purposes:
I have 5 users in my house, with several networked devices, whom I’d like to be able to stream via Plex. I also have the most data, currently around 6TB, that I would like to maintain on the FreeNAS as well; this is music, movies, raw video footage and photos and thousands of ebooks and pdfs. I’m curious about Plex, OwnCloud, Sickbeard, Couch Potato and some of the other plugins; are they worth running on my FreeNAS or should I run them on my desktop and migrate data from there?

Here are my main questions & concerns.

Disk Size:
I envision this unit someday holding 20TB of data (likely mirrored, so 40TB of drives, but this number is really high… might never actually exceed 16TB) I’m not sure if there’s any dangers using larger drives, aside from longer rebuild times. I was thinking 4TB drives seem to be a mid point between capacity and resilvering time. Is it safe to just use 6TB drives or larger to maximize storage capacity within a single large case? Is the resilvering process substantially longer between 2, 4 and 6TB drives? To the point where it warrants buying 1 size over another?

RAID1 Vs. RAID10 Vs. ZFS2.. (I’m pretty sure that’s the title of at least 2 articles I’ve read)

I’ve spent hours and hours reading through RAID posts and trying to understand the benefits and caveats of each..The dreaded Raid5 write hole lead me to ZFS2, which seems wonderful! Except for how slow the resilvering process can be, and the potential for data loss during that time. I am currently of the mindset that RAID1 arrays are the best way to go, for me, as they offer normal write speeds (I’ll be doing bulk writes, but then small incremental writes as new data is created at each location, so I figure internet bandwidth will be a bigger bottleneck than disk write speeds), better than single disk read speeds, and seemingly much quicker resilvering times in the event of a failure. I would have a Raid1 for Business, Extended Family, and Personal (so 3 groups of 2 disks to start) and feel that simply adding 2 more disks as an additional Raid1 would make expansion much easier than trying to extend a ZFS2 array. Does my logic here make any sense? Most of what I’ve read seems to show users implementing ZFS2. The big downside I see to my method, is as “Personal” grows beyond the size of the Raid1 to a second Raid mirror, they will show up as 2 different volumes and I’ll have to either split my data up accordingly or just know to check both folders. My concern is, if a drive fails in a portion of my “Personal” data, I would prefer to have half the data still available, and just have to rebuild that disk. I would also like to remain somewhat system agnostic; in the event of any weird hardware or software failures, I’d like to be able to pull out my drives and pop them in another machine (possibly FreeNAS, possibly Windows)

Requirements (with more questions)

1. On & Off-site weekly (or every 2 weeks) backup of Windows machines to FreeNAS
   1. Done over-night and throttled speeds
   2. incremental, I hope..
   3. 12 machines, can be scheduled for various days and times to minimize bandwidth consumption
   4. ~ 3TB of storage needed
      
2. On & Off-site access to data (streaming movies, music, 1080p videos & Windows & Mac file shares)
   1. Ability to support at least 5 1080P streams simultaneously on and off-site (I’ve got a 50MB down 25MB up service at the moment, which may be a bottleneck) Most streaming will be done on-site (connected via Gigabit switch)
      
3. On & Off-site access to data via network shared folders.
   1. 95% Windows 5% Mac..
      
4. Disk Size & Type - My heart says 4TB, my brain says 6TB..
   1. Seems WD Reds are the best way to go.. but what about the HGST Deskstar NAS drives?
   2. Is there a significant resilvering time between 2TB, 4TB & 6TB drives?
   3. What do you guys personally use and feel safe with?
      
5. Backup - extra piece of mind.. This device will be used to maintain backups of data, but I would like to make a backup of the backup.. Well.. two actually. I would like to have the whole system backed up. My initial thought was a second identical FreeNAS box at the most heavy use family business and RSYNC the two together. I am now thinking about using Crashplan Central (or any other cloud service based on your suggestions) instead. In addition, I would like to maintain a set of physical drives offline. A friend of mine does this process by simply removing a drive from his mirror, and letting it rebuild. The removed drive then becomes an offline backup; this process would be done weekly or every other week, but seems very intensive on the disks.. is there a better approach to do this is an incremental way that doesn’t require resilvering every time??
   
   
6. Disaster Recovery - Here’s my fall back plan..
   1. End-user system dies, I can restore from backup (remotely would be lovely, but I’ll likely hop in the car with an external drive with the image from the FreeNAS)
   2. 1 drive in a mirror fails, resilver mirror
   3. both drives in a given mirror fail, resilver from off-site, non cloud, backup
   4. FreeNAS explodes (or is stolen/fire/flood) (and non cloud backup also fails), restore from cloud service.
7. Raid
   1. ZFS2 pool of disks or several Raid1 Mirrors?
      1. raid1 seems easier and quicker to repair
      2. how can I mirror this FreeNAS to a set of offline disks?
         1. break raid and resilver, creating a backup of the data from that time to be maintained offline, and eventually cycled back into the raid the next time a offline backup is done.

Will the following hardware, in combination with my Raid1 vs RaidZ2 choice, successfully meet these requirements I’ve outlined above?

Case: $100 Fractal Define 4
Motherboard: $250 X10SL7-F
CPU: $270 LGA 1150 Intel Xeon E3-1230V3 Haswell
Ram: $250 (2x) Crucial 16GB Kit (PC3-12800) DR x8 ECC UDIMM Server Memory CT2KIT102472BD160B/CT2CP102472BD160B
PSU: $60 SeaSonic G550 80+ Gold (Is this enough to power a potential 10-12 HDDs?)
Hard Drives: (qty:x) 4TB WD Red NAS WD40EFRX
Total Cost (Minus Drives) ~ $930

Thank you in advance for taking the time to read this and any feedback you can provide me!

Cheers,
Danno
(Sincerely, anyone who took the time to read ALL of this, I really do appreciate it!)

 

[jgreco]

Personally I like to look at the cost of the entire system vs the amount of usable terabytes when figuring out what size drives to get. This sometimes yields unexpected results.

Suggest using ZFS terminology for maximum forum user happiness. Not clear what "ZFS2" is. ZFS has a concept similar to RAID1 called "mirroring." ZFS does not do "RAID10" but you can make several mirror vdevs and ZFS will intelligently load balance (not stripe) between them.

We do not suggest the use of RAIDZ1 on larger disks (TB or larver) because there's more of a chance that a second unrecoverable read error will pop up. ZFS relies on redundancy to maintain pool integrity. RAIDZ2 is probably sufficient for most users. RAIDZ3 is for the more cautious. Some of us do RAIDZ3 with a warm spare disk waiting in the chassis for when a disk fails. It isn't paranoia when dealing with disk drives. They do fail, and fate looks for the low hanging fruit.

You can create a single vdev of whatever type you wish (mirror of two 4TB drives, RAIDZ2 of four or six 4TB drives, etc) to start with and then later add another vdev. We normally suggest that you expand a pool with vdevs of the same type that you initially started with; there are performance issues otherwise.

I think more highly of the HGST Deskstar NAS drives than the WD Red drives, but everyone has a different opinion.

You probably want to figure out a strategy that uses ZFS replication for backups.

 

[Dangins89]

Hi jgreco,

Thank you for your quick reply. Sorry for my improper terms, I'm still learning! :) Your Terminology and Abbreviations Primer is great!

I've thought about your advice, and did some more reading and testing..

-Personally I like to look at the cost of the entire system vs the amount of usable terabytes when figuring out what size drives to get. This sometimes yields unexpected results.

I will try these calculations once I finally decide between a RaidZ3 (or Z2) or several Mirrors

I’ve done more research, and even rewrote this reply twice now. I think I’ve got a more solid understanding of this system and my implementation. My initial confusion was about how I would be able to backup a 16TB RaidZ2 (or Z3) to a single external drive. This is in part why I was considering Mirrors; I'd always be able to write to an external disk the same size. Now, I believe I would simply backup each separate dataset to external HDDs as needed and not try to cram 16TB in any one spot. This still leaves me with some confusion; If I do end up with a dataset that extends beyond the storage limitations of a single drive, say 10TB, how do I go about backing it up? Is there any way to preserve that dataset, or would I need to create 2 datasets (each containing half the data, somehow divided..) and then manage it as 2 datasets going forward in order to back it up?

Here’s my redundancy & backup plan:

1. Business 1 contains 3 computers; 1 computer is determined to be the primary machine and all other computers will use network shares to add/edit files in this single folder.
   
   
2. This “Business Data” folder will backup to a “Business Data” dataset on the FreeNAS box, likely using RSYNC over the internet (initial backup may be time consuming, but consistent rsync throughout the day shouldn’t impact network performance on either end much.) This dataset will have snapshots frequently to allow for rollback for accidental file deletion and to prevent against encryption virus.
   
   
3. I’d like to replicate this dataset to a 3rd offsite location (my home PC, Windows 7 x64 NO ECC) to an external disk that will be stored offline and updated once a week.

Does this seem like a good plan for implementation? Should I be concerned about my last step, replicating the dataset over the internet, to a non EEC machine? I’m thinking, since I don’t have the money for a second freenas, to run a virtual instance on my home machine, simply to get the data to the disk for offline storage. Does my lack of EEC RAM kill this idea?

-RAIDZ3 is for the more cautious. Some of us do RAIDZ3 with a warm spare disk waiting in the chassis for when a disk fails. It isn't paranoia when dealing with disk drives. They do fail, and fate looks for the low hanging fruit.

I’ve never actually had a hard drive fail on me personally in my 20 years working with computers… but I have had to give customers that very same bad news on multiple occasions (without backups none the less..) Where do the overly paranoid fit into this? Do they prefer RAIDZ3 or a bunch of Mirrors?

Also, I based my hardware selection on other's builds and suggestions on here, I'd just like some validation on it, to ensure I'm not blindly missing any important parts.

Thanks so much,
Danno

 

[Dangins89]

An additional question, has anyone here gone through any of the 4 Freenas.org training courses? Did you benefit from them greatly or have you come here with previous knowledge of ZFS and FreeBSD?

 

[Robert Trevellyan]

If I do end up with a dataset that extends beyond the storage limitations of a single drive, say 10TB, how do I go about backing it up?

As you originally proposed, either to another FreeNAS box, or to the cloud.

Should I be concerned about my last step, replicating the dataset over the internet, to a non EEC machine? I’m thinking, since I don’t have the money for a second freenas, to run a virtual instance on my home machine, simply to get the data to the disk for offline storage. Does my lack of EEC RAM kill this idea?

You should be concerned about backing up to a non ECC machine, but only you can determine your risk tolerance. Don't even think about backing up to a virtual machine unless you are a FreeNAS virtualization guru. If you can't afford a 2nd FreeNAS, you probably can't afford to virtualize FreeNAS successfully either.

Where do the overly paranoid fit into this? Do they prefer RAIDZ3 or a bunch of Mirrors?

The overly cautious will typically favor RAIDZ3 over a bunch of mirrors. With RAIDZ3, you can lose any 3 drives without losing data. With the latter, if you lose both drives from one mirror, you lose the pool.

 

[DrKK]

An additional question, has anyone here gone through any of the 4 Freenas.org training courses? Did you benefit from them greatly or have you come here with previous knowledge of ZFS and FreeBSD?

I certainly arrived into the FreeNAS world with no previous knowledge of ZFS or anything else. There is a learning curve.

I don't want to blaspheme anyone, but I think I speak for many of us in the community when I say that we would not particularly recommend the "courses" offered on the freenas.org home page. That's all I will say about that.

I suggest investing the time, reading the forum, and just learning from experience. I had my FreeNAS in a virtual machine just to learn about FreeNAS for something like 6 weeks before I even dared to build a real one.

If you are a book learner, then Allan Jude and Michael Lucas have a very nice book: FreeBSD Mastery:ZFS that is inexpensive. Also, Cyberjock's original ZFS presentation "for noobs" is excellent (and is by the far the most referenced resource for people new to ZFS on planet Earth---nothing else is even close).

 

[solarisguy]

You can get a feel for FreeNAS and ZFS by using any machine with 8GB RAM. Install, re-install. Work on becoming comfortable setting up shares, saving configuration, restoring configuration.

Create RAID-Z2 or RAID-Z1 (depends on the number of SATA ports the motherboard has). Make it fail it by powering the system off and taking one disk out, then powering the system up. Learn how to recover from such a situation. (If possible practice both of the following options: putting the original disk back in; putting another disk in its place). How would such a failure show in your e-mails?

Keep documenting for yourself what needs to be done and what is the order of operations.

ZFS documentation available on Internet might not describe the recommended practices for your version of FreeNAS 9.3.1, so that is why you have to document what you did to make your system work. (The official FreeNAS 9.3.1 documentation is fairly complete in describing the current, 9.3.1, version.)

There are very many pieces of ZFS knowledge that are not required in order to be a successful FreeNAS user. For example, origin of the ZFS name, what COW (copy-on-write) is and why, history of ZFS, default block size, meaning of multitude of parameters FreeNAS sets up by default when creating a ZFS pool / ZFS filesystem, etc.

 

[jgreco]

An additional question, has anyone here gone through any of the 4 Freenas.org training courses? Did you benefit from them greatly or have you come here with previous knowledge of ZFS and FreeBSD?

For the most part, you can learn a ton just by hanging around and participating in discussions. Those of us who are a little more "goo-rue" and have been doing ZFS for a long time may correct you now and then, but that's to be expected on something so insanely complicated. I hang around here because I like seeing people learn and do cool stuff.