Hi, (first post here; I promise I've been reading like crazy and trying to read/think carefully to get to this point)
Critical advice needed (please):
Advice is welcome on any aspect of the endeavor, of course.
I know not to rely on RAIDZ1. Is RAIDZ2 "safe enough" to rely on, or should I jump straight to RAIDZ3?
Assume (because it's true) that I already have more 10 TB hard drives on hand than I currently have a clear need for. I intend to keep one on the shelf at each location as well for the eventual day a replacement is needed. Any surplus I can easily off-load to friends of mine who missed the early Black Friday promo.
I can imagine doing anything between a RAIDZ3 6-drive vdev (feels a little too small) and a RAIDZ3 10-drive vdev. Without any great level of in-depth reasoning, 6-drive RAIDZ2 or 8-drive RAIDZ3 are "feeling right" to me. (6-Z2 because it uses only half of the slots in the R720xd; 8-Z3 because it gives me a substantial multiple of our current usage, more resiliency (I think), but it's not obvious down the road how to further grow that with only 4 slots left)
I'm considering running only one large NAS here (in MA) and the other at my parents' place (in NC). Any special considerations for a NAS that I can arrange good remote access to, have local semi-skilled hands (skilled with tools, not with IT topics), but will only have hands-on access to a couple times per year generally? I'm considering this not so much from the power cost spread, but rather from the fault-isolation spread. A fire or burglary here puts my primary machines and the NAS (and the backup NAS) all at risk all at once. Thoughts?
For the burglary situation, what's the current thinking/recommendation about encrypting data-at-rest? Both CPUs support AES-NI.
Thanks to everyone who has read all of the above and many thanks for anyone who provides advice!
"IT guy" (computer programmer by trade, then went into managing operations [before it was called "devops"], now back on the management side of software engineering), married with two kids (so lots of family photos, some videos, corresponding media consumption demands, and some documents to store and safeguard), intermittently busy/traveling for work (so this setup needs to run for weeks at a time without any oversight or fiddling), not worried about rock bottom pricing; worried a lot more about data security and integrity, backups against loss of treasured family photos, and a good user experience for my very smart but not IT-technical wife and kids.
I pay ~$0.20/kWh for power. I'm not going to be bent out of shape if NAS1 costs me $500/year in electricity (~300W average). Even a bit more than that would be fine. Double that and I'll have to give it some real thought, but will still do it.
One possible off-site location [my parents] pays about $0.11/kWh
I pay ~$0.20/kWh for power. I'm not going to be bent out of shape if NAS1 costs me $500/year in electricity (~300W average). Even a bit more than that would be fine. Double that and I'll have to give it some real thought, but will still do it.
One possible off-site location [my parents] pays about $0.11/kWh
In roughly decreasing order of importance:
G1. Family photos and videos (irreplaceable data) to be stored in a highly available store locally, backed up to two other highly durable stores, at least one of which is off-site
G1. Family photos and videos (irreplaceable data) to be stored in a highly available store locally, backed up to two other highly durable stores, at least one of which is off-site
- Current intention here is FreeNAS zfs, at least RAIDZ2 for the main store
- Secondary store is likely to be another FreeNAS (location TBD), again at least RAIDZ2
- Tertiary store is likely to be Amazon Prime Photos
- Considering additional replication of photos only to S3, B2, Glacier, or rsync.net
- Considering additional replication to a Synology NAS off-site
- Intention for primary and second store as above.
- Tertiary store likely to be rsync.net
- Considering additional replication to S3, B2, Glacier, tarsnap, or the off-site Synology.
- Primary and second store as in G1
- Tertiary store is unlikely due to data size (and with the computer and two zfs Time Machines, we have 3 copies already, though perhaps none off-site)
- Primary store will be on FreeNAS (probably on the RAIDZ2/Z3 bulk volume shared as above)
- No additional copies of these are needed, as we have the DVDs
- Considering Plex ease-of-use apps like: Ombi, Sonarr, Radarr, Tautulli (need to read more)
- Stored as in G2.
- Accessed with Next Cloud or Own Cloud, or Resilio (happy to take advice here)
- For reference, Synology CloudStation worked great for this
- Storage - striped mirrors or RAIDZ2 here?
- Highly desirable use cases: Ubiquiti UniFi controller, Plex, PiHole, Asterisk, Mediawiki (or other wiki), git server, PXE Boot and iSCSI capability for Raspberry Pi and other diskless workstations, Minecraft server, libreNMS
- Considering use cases: self-hosting mail (inc webmail) for my family, database hosting (utility hosting for other apps for the household only), syslogd, PDF auto-OCR, text indexing
- In exchange for them acting as a target for my data, I'd like to offer the same in return
- Primary concern here is security (of the networks and of the data at rest; yes, they're family/friends, but I have to assume their network and hardware is compromised and might or might not be able to give me my data back, but I want to make sure neither they nor their attacker can make use/sense of the data)
- Take each old drive (that I can still mount) and dd it to the NAS with as much information as I can find from the label, maybe scan the filesystem for the latest mtime and name the block file with that information. Realistically, this is probably a graveyard, but it's a graveyard that takes less space than currently (once I take apart the old drives or otherwise destroy them)
- Primary and secondary backup on NAS devices. Might put select old hard drives onto B2 or Glacier
- 6.6 TB is currently in use across two volumes.
- Realistically, if I made an effort to get all the family photos, videos, and other random files onto there, that might swell to 8 TB.
- Kids are 7 and 9 and I expect they'll both get into computing/science (as their parents have) and that we're the types to want to digitally packrat a lot of the memories (even the ones that no one is likely to ever see again).
- I don't currently do any IP cameras. Adding that could add an additional amount of utilization, obviously.
Not many. My time is somewhat limited in terms of I can't sustain a system that regularly demands 2+ hours of my attention at a randomly chosen time. I can plan to burn 50-100 hours during the initial setup and tweaking, but need the on-going support burden (on me) to be reasonably low.
I have purchased some of the hardware. If that hardware isn't suitable, please tell me (and take a stab at the "why"). Cost is a distant second to security and data durability.
I have purchased some of the hardware. If that hardware isn't suitable, please tell me (and take a stab at the "why"). Cost is a distant second to security and data durability.
I've been studiously reading about FreeNAS and have a pile of equipment that research suggests to me is suitable and I've been running a "prototype" FreeNAS server for about 3 weeks now.
FreeNAS1 will be my primary NAS, located in the basement of our house, with wired Ethernet (multiple drops as needed) available. The network security gear and cable modem is also here. Definitely will publish SMB file services. I haven't decided on AFP or NFS (even though we have no Windows machines in the house currently, everyone seems to be able to deal with SMB). I do currently have nfs turned on on the synology (primary fileserver today) because that plays very nicely with the FreeNAS1 test unit during my testing.
I think I can use FreeNAS1 as a jail host, docker, and VM host and use the second Dell R720 as another FreeNAS box. I'm undecided as to whether to locate that also at our house, at my parents' house (700 miles away), or in a local colo (least likely).
I've been testing and playing around with FreeNAS and some of the critical services and am now ready to blow away the current setup on FreeNAS1 and re-create that machine as the intended-for-production (after testing and burn-in).
FreeNAS1 will be my primary NAS, located in the basement of our house, with wired Ethernet (multiple drops as needed) available. The network security gear and cable modem is also here. Definitely will publish SMB file services. I haven't decided on AFP or NFS (even though we have no Windows machines in the house currently, everyone seems to be able to deal with SMB). I do currently have nfs turned on on the synology (primary fileserver today) because that plays very nicely with the FreeNAS1 test unit during my testing.
I think I can use FreeNAS1 as a jail host, docker, and VM host and use the second Dell R720 as another FreeNAS box. I'm undecided as to whether to locate that also at our house, at my parents' house (700 miles away), or in a local colo (least likely).
I've been testing and playing around with FreeNAS and some of the critical services and am now ready to blow away the current setup on FreeNAS1 and re-create that machine as the intended-for-production (after testing and burn-in).
- FreeNAS1-Dell R720XD (12 3.5" drive chassis), 128 GB ECC RAM, 2 x E5-2660-8 cores ea @2.20GHz (Passmark of 1397 per thread and 11098 per socket [two sockets full]), H200 card flashed to LSI HBA mode, 4 x Gbe card, currently running the prototype/play around with FreeNAS install
- FreeNAS2-Dell R720 (8 3.5" drive chassis), 96 GB ECC RAM, 2 x E5-2637 v2-4 cores ea @3.50GHz (Passmark of 1831 per thread and 9406 per socket [two sockets full]), H200 card not yet unpacked from Ebay, 4 Gbe on-board, additional card with 2 (presumed) Gbe, currently never even powered it on
- 2 x Chelsio 10Gb fiber ethernet cards - 110-1088-30 B0
- (up to) 20 Western Digital 10 TB drives from the BestBuy Western Digital EasyStore Black Friday promotion - I was considering to use 6 of these in a RAIDZ2 (~36 TB net and about 29 TB to 80%) in NAS1. I'm open to how to use the other slots and drives. I'll take any advice here, please. Does 8 make more sense? What about RAIDZ3?
- 8 2.5" Intel SSDs (from NAS2 when I bought it). 4 x 160 GB Intel DC S3500, 4 x 200 GB Intel DC S3700. I haven't yet powered up the machine (it arrived today) to see the SMART stats on the drives, but I do know they ran for about 1376 days (3.75 years) on an unknown workload. So, they're getting up there, but probably still have some useful life left in them. I would like to reserve two of them for another lab project unrelated to this.
- Synology DS1815+ - full of 8 x 2 TB green drives, currently serving as production fs1 for the household, including TimeMachine, CloudStation, and Plex. Will be retired as a primary NAS though perhaps relegated to off-site backup target duty. As above, 6.6TB is in use.
- Synology DS214 - available as a scratch NAS for temporary additional backups. Probably will keep around and powered down once this is all done.
- 3 x APC Smart UPS 1400 (rack mount unit) - 2 fully working with batteries replaced; 1 working with new batteries but lights the "site fault" red LED in back (still carries the load just fine in testing); 1 for parts. 3 APC AP9619 network cards are en route; 1 APC AP9512 temp and humidity sensor for the AP9619 card is on-hand.
- 1 x APC 7930 networked PDU - I'm intending to run this off the UPS supply and use this as primary gauge of power consumption during testing. Because both servers have dual AC supplies, I'm considering running them off dual UPS for additional run-time.
- Ubiquiti ER-X, SW8s, and AC-Pro access points. I mention this only because of any tie-in to network security or network segmentation needs. I also have a load of older Cisco gear I bought from a Blackberry/RIM closing-down auction. I'm not opposed to switching to Cisco, but only intend to do that if the Ubiquiti gear (including possibly additional UBNT gear) can't do the job. Here's a case where I'll throw money at ease-of-admin.
Critical advice needed (please):
Advice is welcome on any aspect of the endeavor, of course.
I know not to rely on RAIDZ1. Is RAIDZ2 "safe enough" to rely on, or should I jump straight to RAIDZ3?
Assume (because it's true) that I already have more 10 TB hard drives on hand than I currently have a clear need for. I intend to keep one on the shelf at each location as well for the eventual day a replacement is needed. Any surplus I can easily off-load to friends of mine who missed the early Black Friday promo.
I can imagine doing anything between a RAIDZ3 6-drive vdev (feels a little too small) and a RAIDZ3 10-drive vdev. Without any great level of in-depth reasoning, 6-drive RAIDZ2 or 8-drive RAIDZ3 are "feeling right" to me. (6-Z2 because it uses only half of the slots in the R720xd; 8-Z3 because it gives me a substantial multiple of our current usage, more resiliency (I think), but it's not obvious down the road how to further grow that with only 4 slots left)
I'm considering running only one large NAS here (in MA) and the other at my parents' place (in NC). Any special considerations for a NAS that I can arrange good remote access to, have local semi-skilled hands (skilled with tools, not with IT topics), but will only have hands-on access to a couple times per year generally? I'm considering this not so much from the power cost spread, but rather from the fault-isolation spread. A fire or burglary here puts my primary machines and the NAS (and the backup NAS) all at risk all at once. Thoughts?
For the burglary situation, what's the current thinking/recommendation about encrypting data-at-rest? Both CPUs support AES-NI.
Thanks to everyone who has read all of the above and many thanks for anyone who provides advice!
