BUILD Build for reliable backup

[spierce7]

Hey guys, so I'm looking to make a build where I can:
1. Set up a space for my wife and I to have our mac time machine back ups.
2. Store several GB of family photos
3. Store several TB of family videos
4. Create a simple site that uses the FreeNAS API to allow my entire family to upload new images / videos, and search through all the images / videos that have been made.

There are a few questions that I have:
1. Is FreeNAS right for this? Should I be at least considering maybe an easier option?
2. How powerful of a machine do I need? I've been going through the hardware recommendations, only to find that I don't know if I'm buying something that's too powerful, or not powerful enough.
3. Are there any solutions that already exist for #4 above? I'm a software engineer, and wouldn't mind rolling my own, but surely I'm not the only one with this desire.
4. I was considering raid 5 or raid 6. I've read in some other places that raid is not a backup solution. Assuming that I want to be pretty certain that I won't lose this data, do I need to create a 2nd device that will create a snapshot every night or so of everything stored on the freenas? What do people normally do for this?
5. How much money should I be looking to spend? I want something with at least 4+ TB to start, with the ability to grow in the future if need be. I'm not looking to cut corners, but I'm not wanting to spend more than I need to.
6. Are there any previously documented, relevant builds that would suit my need well? I've found some online, but they are somewhat old.

 

[Robert Trevellyan]

1. FreeNAS is a viable option. There are others. Ideally, test out a few and see which you prefer.
2. For the requirements you listed, nothing fancy. Entry level for something off-the-shelf would be a Dell PowerEdge T20 or a Lenovo TS140.
3. Looks like a job for ownCloud. There is a FreeNAS plugin for it, if you decide to go with FreeNAS.
4. In FreeNAS you'd say RAIDZ1 or RAIDZ2, because the filesystem is ZFS. You are correct, RAID is not an excuse for not having a backup. Bad things can happen on any platform, so anything you can't bear to lose must be backed up.
5. Referring back to #2, as little as $179 plus RAM plus hard drives. To store 4TB comfortably and allow for some growth, you could start with 4x 6TB. If data growth is slow enough, you could even get away with 4x 4TB.
6. If you want to roll your own, spec out a system and post it for review.

 

[Robert Trevellyan]

4 (continued). People use all kinds of solutions for backup, depending on many factors ... volume of data, their tech level, budget etc. Some use cloud backup, some build a 2nd system, some hook up external drives via eSATA and rotate them offsite.

 

[Dice]

Hello, welcome to the Forums.

3. Are there any solutions that already exist for #4 above? I'm a software engineer, and wouldn't mind rolling my own, but surely I'm not the only one with this desire.

I've not heard or seen any such or even remotely similar solution for FreeNAS. Other contributors of the Forum may be better informed about the capabilities of coding such.
What ever you can get to run on a FreeBSD machine can be setup inside FreeNAS 'Jails'. May that be a webserver running your own app..

2. How powerful of a machine do I need?

I'd say you could definitely get away with a G4400 CPU and start off with 16GB of RAM, preferrably 32GB. Put this on any SuperMicro X11 board and you'll be fine. X11-SSL-F (try getting the -F option for IPMI)

I was considering raid 5 or raid 6

Proper terminology here is RaidZ1 and RaidzZ2.
The most common option for your typical usecase would be to pick 6-8 drives, put them in raidz2.
Select the drive size (typically recommended WD REDs), that will get you to the desired usable space (this is the goto calculator: https://jsfiddle.net/Biduleohm/paq5u7z5/1/embedded/result/ )

I've read in some other places that raid is not a backup solution. Assuming that I want to be pretty certain that I won't lose this data, do I need to create a 2nd device that will create a snapshot every night or so of everything stored on the freenas? What do people normally do for this?

Good, this is the correct state of mind!
Some people build mirror/lower performance systems, locate them at a friends house. Others send their data to backblaze, dropbox unlimted or crashplan.

want something with at least 4+ TB to start, with the ability to grow in the future if need be

Expanding FreeNAS is typically done by adding a second vdev to the same zpool. Using the same number of drives. This means that while going with higher drive count (in raidz2), the better the space utilization efficiency but also the steeper upgrade cost.
A rule of thumb that people keeps returning to (for reasons) is to start out with about 3x as much usable space as you think you would use. Or if you've a better estimation on how much data you are producing, get enough to hold you covered until you foresee the next upgrade...
Planning ahead is KEY. Not doing so will sure enough, but eventually bite you in the rear. (I took that bite way sooner than expected...).

Cheers /

 

[Robert Trevellyan]

My data grows slowly, and my T20 does everything I need without breaking a sweat. @Dice's suggestions are at least as valid, and give you more room to grow. The thing about having a server is that once you have it, you will keep finding more ways to make use of it, so planning for growth can be tricky.

 

[spierce7]

The thing about having a server is that once you have it, you will keep finding more ways to make use of it, so planning for growth can be tricky.

So I'd love to setup a server and use it for multiple things. I'm under the impression that FreeNAS is an OS though, and so it'd be difficult for me to use my FreeNAS system for anything but data backup. I'd invest in a higher performance system if I could run websites off of my server and use it for other things, I'd invest in a higher level system with a xeon processor. Is that a good idea though? I just assumed I'd need to get a separate system for that.

Good, this is the correct state of mind!
Some people build mirror/lower performance systems, locate them at a friends house. Others send their data to backblaze, dropbox unlimted or crashplan.

Locate them at a friends house? Is it a bad idea for me to simply put the 2 machines side-by-side? I wasn't planning on building anything for this. I was planning on just getting an external hard drive and pushing to it.

Expanding FreeNAS is typically done by adding a second vdev to the same zpool.

You lost me. vdev? zpool? Not understanding this made me not understand the rest of what you were saying here as well.

Looks like a job for ownCloud. There is a FreeNAS plugin for it, if you decide to go with FreeNAS.

I'll look into it. Thanks for the recommendation!

 

[Robert Trevellyan]

I'm under the impression that FreeNAS is an OS though, and so it'd be difficult for me to use my FreeNAS system for anything but data backup.

When I said 'server', I meant it as a generic term - an always-on computer, delivering services to users on a network. FreeNAS is indeed an OS aimed specifically at storage, but it can do a lot of other things.

First off, there's data storage and data backup. The 1st item on your list is backup, the 2nd and 3rd are data storage. So, an example of a newly discovered use would be storage of things you hadn't considered storing before, e.g. DVD images for easy playback on any device in the house, vs having to put the physical DVD in a player attached to a specific TV.

Next, through the use of jails, plugins (preconfigured jails) and virtualization, you can run other things on FreeNAS. The ownCloud plugin is one example of this.

websites ... Is that a good idea though?

In general, exposing FreeNAS to the internet is discouraged for security reasons, but running a web server in a jail is viable. I'm not clear whether you want #4 to be accessible over the internet, but if you do, the ownCloud plugin is just that, i.e. a web app running in a jail.

 

[spierce7]

running a web server in a jail is viable

Would I have any problems allocating a portion of the memory specifically for that jail? If I was to host a database in one jail, could multiple other jails communicate with that database?

 

[Sakuru]

https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/
https://forums.freenas.org/index.ph...guration-for-small-freenas-deployments.27062/
https://forums.freenas.org/index.php?threads/zfs-primer.38927/
https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/
http://doc.freenas.org/9.10/freenas.html

Welcome to FreeNAS :D

 

[Mirfster]

Locate them at a friends house? Is it a bad idea for me to simply put the 2 machines side-by-side? I wasn't planning on building anything for this. I was planning on just getting an external hard drive and pushing to it.

No that is not a bad idea. For a lot of people there is a "Rule of 3" in regards to backups. Aside from the other concepts in that, there is having "On-Site", "Near-Site" and "Off-Site" copies of the the vital data.

So in your situation, what would happen if your house caught on fire and you have both those Servers in there?

How far you want to go into ensuring the safety of your data is entirely up to you, but this is just "food for thought". It can get confusing and seemingly overwhelming, but at the end of the day only you know what is best for your situation and level of comfort.

 

[Dice]

Read the links by @Sakuru. They explain what I mentioned regarding vdevs and zpool. It is more or less a mandatory read.

The thing about having a server is that once you have it, you will keep finding more ways to make use of it, so planning for growth can be tricky.

I definitely agree. But - given the OP's 'own production of data' predictability might be a lot better than for random data hoarders (like me).

@spierce7 What's you estimated annual data growth in retrospect?
Would it make sense to use the last 2-3 years worth of data growth, and extrapolate the data per year development an additional 3 years a head of today?
- what required space would you end up with?

I'm not sure I got that point across correctly. I'll help you with a guesstimated example that takes somewhat into account increasing data use by year from.. more pics, new devices with higher definition frame rates etc:

To start off:
2013 Total 2TB data accumulated
2014 + 500GB
2015 + 800GB
2016 + 1 TB
This would generate a total of 4.3TB used space.

2017 + 1TB
2018 +1.5TB
2019 +2 TB
Adding another 4.5TB over the next years.
A grand total of around 9TB would be a great size to shoot for.

Now, granted you'd want to build your machine upfront to cover you through 2019 without thinking about adding new space. What drive configuration would be neat?
Following @Bidule0hm's calculator; 6x3TB drives in a RaidZ2 would be ideal in my opinion, giving you about 9.6TB usable space. If the prize difference to 4TB's drive is acceptable, then you would end up with about 12.6TB TB usable data.
Then when you eventually run out of space, you have the option to upgrade by getting another batch of 6x drives (any size), and probably a HBA controller (100USD off ebay).

Cheers

 

[Robert Trevellyan]

Would I have any problems allocating a portion of the memory specifically for that jail?

That's now how jails work - they're more focused on security than on resource allocation. If you need more control of resources, you can use virtual machines.

If I was to host a database in one jail, could multiple other jails communicate with that database?

Not directly, because each jail is isolated from the rest of the system, by design. A jail could make a database available at an IP address on your LAN, which another jail could then access.