BUILD Advice on build to replace 3-2-1 storage solution

[eexodus]

Background
I'm a PC systems specialist at a university and second-in-command for a department of ~150 users. We're looking to replace our aging 3-2-1 storage solution. Currently we have an old Xserve running OS X which mainly acts as an LDAP server. We don't plan on using FreeNAS to replace it, instead we will likely buy a Dell server running Debian/Ubuntu or openSUSE and OpenLDAP.

Multiple hardware RAID arrays (4 HDD bays) are plugged up to the Xserve over FireWire 800/eSATA. User home directories (infrequently used) and SMB/AFP shares (frequently used) use these arrays which total ~50TB. A second set of arrays are plugged up to the Xserve and act as a primary backup (another ~50TB). A third set of arrays are plugged up to a Mac mini server off-site which acts as a secondary backup (another ~50TB). We plan to replace these arrays with FreeNAS pools; we have access to 10Gb/s so bandwidth between FreeNAS and the Linux server should be good (the pool shares would be mounted as network drives on the Linux server).

In the past, my boss has just added more arrays as needed. It is becoming a headache for a few reasons:

1. The Xserve has unreliable uptime.
2. The arrays don't have redundant PSUs so when one dies the users that have data on it are unable to access it until a new PSU comes in.
3. They are hardware RAID (this isn't a big issue beyond unlikely silent data corruption) in RAID 5 (this also isn't a big issue because we have two backups, UPSs, and generators).
4. They are expensive to expand. For example, it would cost us $1,726.57 to buy a 24TB Buffalo TeraStation which is $71.94 per TB vs. buying new Western Digital 6TB hard drives to put into the empty bays of our future FreeNAS server at $39.69 per TB.

We considered cloud block storage for primary/secondary backups
...but we decided against it. If anyone has experience in this area I'd be extremely interested in any insight. AWS EBS would be $0.045 per GB per month (not including the cost of an AWS EC2 instance required to attach it to) and Google Cloud would be $0.026/GB/mo. Compared to a $10,000 FreeNAS server that will last 5-8 years it just doesn't compare. For example, Google Cloud would cost $1,300/mo for 50 TB/mo. Maybe I'm missing something but cloud block storage doesn't seem "there yet" to replace local storage for small/medium businesses like our department. When compared to pricey SANs or NASs with rock-solid warranties, maybe, but I feel our unique situation of having free electricity, bandwidth, rack space, battery/generator backups, plenty of free time, etc. pushes us towards local storage because it is certainly cheaper to replace a few dead HDDs/PSUs a year vs. monthly payments of $1,300.

The build (and questions)

Keep in mind we will likely be buying three of these builds (primary data, primary backup, secondary off-site backup)

• Chassis: SuperMicro 6048R-E1CR36L
  
  • Motherboard: SuperMicro X10DRH-iT
  • SAS Controller: AOC-S3008L-L8i (supports pass-through IT mode)
• HDDs: 18x Western Digital WD60EFRX 6TB in RAIDZ2 pools of 6 (would add 6 drives at a time later)
  
  • Is this the best setup? Or would larger pools/RAIDZ be OK with our multiple backups? I'm also not sure if I should be afraid of rebuild times.
• RAM: Samsung MEM-DR416L-SL01-ER21 (16 GB) x 4 for a total of 64 GB (might add another 64GB later)
• CPU: 1x Xeon E5-1650 v4 or Xeon E5-2630 v4 (might add a second CPU later)
  
  • I'm leaning more towards the E5-1650 because I know GHz will help SMB, but beyond that I don't know how CPU choice affects FreeNAS besides obvious speed benefits.
• FreeNAS SSDs: does anyone have any recommended enterprise SSDs I could mirror for FreeNAS?

Do you think we could also skimp on the CPU/RAM in the two backup FreeNAS servers since users won't be accessing them? They would likely just be in a live > daily backup > weekly backup layout.

 

[Nick2253]

First off, welcome! Hopefully we can help make this a reality for you.

Just to summarize: You are looking at three identical FreeNAS servers. These will provide primary access and primary/secondary backup for your ~50TB of data.

I am curious, what are the shares used for? Is it a lot of read data? Write data? Both? Big/small files?

Assuming this is correct, then here are my comments:

• On your hard drives: I think 6-drive RAIDZ2 vdevs would work pretty well. I would put all of them in the same pool to maximize I/O. You could go to 9 drive RAIDZ3 vdevs, but you'd lose the extra I/O from the third vdev. Also, it's much easier to add 6 disks at a time than 9 disks.
  
• I think 64GB of RAM is insufficient for the hard drive space you are proposing. You may be able to get away with 64GB for your backup servers, but I would imagine you'd want at least 128GB of RAM for your primary server. Depending on the demand, you may want closer to 256GB.
• I would probably chose the E5-1650 over the E5-2630, though the extra cores may be a big benefit if you have a large number of simultaneous users. What's your share use look like?

On skimping for your backup servers: I would say it depends on your disaster recover plan. If your disaster recovery plan in the event of a NAS loss is to promote your primary backup server, you'll want to make sure that the primary backup can serve your users (and in that, you may tolerate lower performance). That might mean more RAM than you "need" for your primary backup, but you tolerate a lower processor than your main server. For the remote backup, you can probably get away with a wimpier processor.

As far as the memory is concerned, you could try it at 64GB, but I would be ready to upgrade. I'd say that I'm 75% sure that you'll need 128GB for your main server to get the performance that you want, though I'm around 20% that you'd need more RAM for your backups, which means you could probably get away with 64GB, with the knowledge that it might be insufficient.

 

[gpsguy]

I read you message a couple of times, but still aren't 100% sure how you plan to use FreeNAS in your environment.

You haven't said anything about snapshots and/or replication. I don't *think* you can replicate to two targets, nor replicate from primary to secondary and secondary to tertiary.

Depending on your needs, you might consider using rsync to your onsite secondary server and replicating to your offsite server.

We're looking to replace our aging 3-2-1 storage solution

 

[Mirfster]

If you are getting up there in RAM, then you may want to give consideration to a SLOG and possible L2ARC.

Also, for the Secondary Backup, you may even consider RaidZ3.

 

[Nick2253]

I don't *think* you can replicate to two targets, nor replicate from primary to secondary and secondary to tertiary.

There is no problem replicating to two targets. The simple solution would be to schedule a replication to the first target, and schedule a replication to the second target. Or, probably the way I'd go, is replicate to the first target, and let the first target replicate to the second target.

 

[MatthewSteinhoff]

Multiple hardware RAID arrays (4 HDD bays) are plugged up to the Xserve over FireWire 800/eSATA

Whatever you buy is going to be leaps and bounds ahead of the Xserve solution. So, even if you under-buy or under-spec, you're way ahead of where you are now. We had Xserves years ago and, even then, they seemed slow.

User home directories (infrequently used) and SMB/AFP shares (frequently used) use these arrays which total ~50TB.
(the pool shares would be mounted as network drives on the Linux server)

Am I understanding that you're using Linux servers as front-ends the the NAS? So your 150 users will never contact the NAS directly? You're using NFS for the Linux server to NAS connection?

SAS Controller: AOC-S3008L-L8i (supports pass-through IT mode)

Please double-check that card can act as an HBA. I quickly looked at the user guide and didn't see anything other than RAID described.

Is this the best setup? Or would larger pools/RAIDZ be OK with our multiple backups? I'm also not sure if I should be afraid of rebuild times.

'Best' is hard to say. RAIDZ2 in six-disk groups is what I would do. It'll be faster than what you have now with a reasonable amount of redundancy.

I'm leaning more towards the E5-1650 because I know GHz will help SMB, but beyond that I don't know how CPU choice affects FreeNAS besides obvious speed benefits.

With no power concerns and a price delta that is negligible, I'd go with the E5-1650 in a heartbeat. Given where you're coming from, either processor will fly.

RAM: Samsung MEM-DR416L-SL01-ER21 (16 GB) x 4 for a total of 64 GB (might add another 64GB later)

Start with 128 GB (or 256GB) in your primary host.

If your plan is that the secondary NAS acts as your primary in a disaster situation - and this has to happen quickly - match its memory to the primary. Otherwise, I'm fine with the secondary limping along with 64 GB until you can add more memory or get the primary working. It sounds like the existing system is somewhat unstable and your users are used to limping along. Even limping along with 64 GB, you'll be faster than where you are now. (Sense a theme?)

You can absolutely live with 64GB of memory on the tertiary NAS. It'll accept replicated data just fine with 32 GB of RAM. You can also use a slower, less expensive processor in the tertiary.

To save money, build the primary and secondary NAS using new drives, get them up, running and proven then take the drives out of your Buffalo TeraStations for use in the tertiary host. Even if you're using twice as many slots because of the 2/3TB drives, it will be years before you have to replace them with larger drives given your existing utilization. Finally, while you don't want to fill your primary pool more than about 50% to (reasonably) maximize performance, your tertiary NAS will happily chug along at 80% full.

Chances are, your new NAS will be FAR more reliable than the existing system and you'll never touch the secondary machine let alone the tertiary NAS. So, spend money on the primary and skimp on the secondary and throw used, spare parts at the tertiary.

Replication: the primary can send snapshots directly to the primary and secondary hosts with little performance impact. I'd much rather do that than daisy chain from one to two to three.

Cheers,
Matt

 

[eexodus]

...

Thanks for the welcome. I've been really surprised by how friendly and knowledgeable everyone is here.

The shares are used for storing research: documents, spreadsheets, video files... the most unique thing about our use case is sometimes we're dealing with ~24 hour video files.

Based on yours and others input I'm definitely going to put 128 GB RAM in the primary server. We can try 64/32 GB of RAM on the second/primary backup servers because unlike some of you have suggested we don't plan on using it as a "hot spare"; especially because of how unlikely it would be for the primary server's pool to completely collapse (as some of you have said). This brings me to another question on disks and vdevs. Because of rebuild times on such a large pool of 3x 6 drive RAIDZ2 vdevs, and because all the drives will be purchased at the same time (bad batches) would it be wiser to do 9x 2 drive mirror vdevs instead to shorten rebuild time? I still am unfamiliar with how long it would take to rebuild a RAIDZ2 vdev made a 6x 6TB drives.

Also from what I've read IO will be as fast as the slowest disk in the pool, and if the pool is made of entirely new and identical drives then I assume IO in a mirrored setup would be the same as a 3x RAIDZ2 setup. Of course, we'd lose half our storage.

Thanks for yours and others input regarding the Xeon too. I'll go with the E5-1650 v4. I don't admin our Xserve so I can't comment on if more cores would help with concurrent users/connections, but like Matthew points out compared to our Xserve the 1650 will be great.

...

Sorry for any miscommunication on my part. I didn't go into much detail on backup strategies and replication because we haven't gotten there yet. I posted this thread to mainly help pick out parts so I can get a solid build together and do a price comparison with alternatives. Second, I'm still new to FreeNAS so I just had a few questions in my OP mostly regarding pool layout and CPU GHz.

Since I don't think my boss would be interested in using the primary backup as a hot spare we likely would use the structure you outlined: primary pool >rsync> primary backup >replication> secondary backup; or, like others have suggested sending snapshots to both backups from the primary pool.

...

Yes we'll consider RAIDZ3 for the secondary backup. I actually considered asking about that shortly after posting. It would be used for disaster recovery so reliability is paramount. We might be forced to go with a different layout though if we reuse old TeraStation drives.

...

The Xserve is actually doing OK. It isn't particularly slow because we're basically only using it for LDAP and file sharing, but it is no longer supported and Apple has abandoned the enterprise so there's not much hope there.

Yes you understood me correctly about using a Linux server as a front end and connecting to the primary pool with (likely) NFS. That would be over 10Gb/s on a 10Gb/s switch (we would have to buy the switch). Would you recommend something else? I looked into Thunderbolt 2, but unfortunately SuperMicro doesn't widely support it.

Thank you for the last part of your post. I didn't know FreeNAS liked being at 50% capacity or lower. That seems rather low to me and makes it even less likely we'd go with vdevs made of mirrors (which would lose of 50% capacity right off the bat). Would 70-80% capacity affect performance? Like you say, FreeNAS should be a lot more reliable than our current setup. I'm probably just being anxious about rebuilding times, etc. and either way we'll be in a good position of having two backups.

Is FreeNAS finicky about adding a second CPU later?

 

[Mirfster]

Also from what I've read IO will be as fast as the slowest disk in the pool, and if the pool is made of entirely new and identical drives then I assume IO in a mirrored setup would be the same as a 3x RAIDZ2 setup. Of course, we'd lose half our storage.

Actually a Pool of 9 Mirror vDevs will have more IOPS that a Pool of 3 RaidZ2 vDevs. Think of it this way, a single RaidZ2 vDev will basically have the performance of a single drive; but provides 2 drive Redundancy. Mirrors are faster and will re-silver faster as well, but have a single drive redundancy (unless you do 3-Way Mirror) and will cost you 50% capacity (66% if you do 3-Way Mirrors).

IOPS are gained by the number of vDevs in a Pool, but in a one on one speed test a single Mirror is natively faster than a RaidZ2 since it does not have the overhead of writing to multiple drives.

Each vDev has its Pros and Cons and there is always a decision to be made between Speed, Space and Redundancy.

There is great information in the ZFS Primer and it contains links to other articles such as:
Getting the Most out of ZFS Pools
A Closer Look at ZFS, Vdevs and Performance

I didn't know FreeNAS liked being at 50% capacity or lower. That seems rather low to me and makes it even less likely we'd go with vdevs made of mirrors (which would lose of 50% capacity right off the bat). Would 70-80% capacity affect performance?

The 50% comment only pertains if you are using iSCSI. For any Pool/Volume additional things can be done to help speed things up; More RAM, Settings (like turning off atime and case-sensitivity), SSD SLOG, SSD L2ARC, etc.

Per the ZFS Primer:

At 90% capacity, ZFS switches from performance- to space-based optimization, which has massive performance implications. For maximum write performance and to prevent problems with drive replacement, add more capacity before a pool reaches 80%. If you are using iSCSI, it is recommended to not let the pool go over 50% capacity to prevent fragmentation issues.

Is FreeNAS finicky about adding a second CPU later?

Nope, it will happily incorporate it.

 

[Stux]

Actually a Pool of 9 Mirror vDevs will have more IOPS that a Pool of 3 RaidZ2 vDevs. Think of it this way, a single RaidZ2 vDev will basically have the performance of a single drive; but provides 2 drive Redundancy. Mirrors are faster and will re-silver faster as well, but have a single drive redundancy (unless you do 3-Way Mirror) and will cost you 50% capacity (66% if you do 3-Way Mirrors).

IOPS are gained by the number of vDevs in a Pool, but in a one on one speed test a single Mirror is natively faster than a RaidZ2 since it does not have the overhead of writing to multiple drives.

Each vDev has its Pros and Cons and there is always a decision to be made between Speed, Space and Redundancy.

There is great information in the ZFS Primer and it contains links to other articles such as:
Getting the Most out of ZFS Pools
A Closer Look at ZFS, Vdevs and Performance

The 50% comment only pertains if you are using iSCSI. For any Pool/Volume additional things can be done to help speed things up; More RAM, Settings (like turning off atime and case-sensitivity), SSD SLOG, SSD L2ARC, etc.

Per the ZFS Primer:


Nope, it will happily incorporate it.

Of course it's more complicated than that.

Raidz2 has the sequential read/write performance of n-2, where as mirror has write performance of 1 and read of n. Per vdev.

I might be slightly wrong in the n-2, the point is raidz sequential performance scales with number of drives.

 

[Mirfster]

Of course it's more complicated than that.

Yes, I was just providing a 10 mile view (is it 10 or 5 mile?... Too early...). :)

 

[Nick2253]

unlike some of you have suggested we don't plan on using it as a "hot spare"; especially because of how unlikely it would be for the primary server's pool to completely collapse (as some of you have said).

I agree with your point on the likelihood of total pool failure, but don't forget other reasons that servers are offlined: non-redundant hardware failure (e.g. motherboard), network disconnect (e.g. damaged switch, network room fire), or total server loss (e.g. accidental damage, fire/earthquake).

Not trying to tell you your business, just give you some more context.

 

[Spearfoot]

Please double-check that card can act as an HBA. I quickly looked at the user guide and didn't see anything other than RAID described.

I share Matt's concern about the AOC-3008L-L8i card, and whether or not it can be flashed to IT mode. But it's based on the LSI SAS3008 chip, for which Supermicro has both IR and IT firmware available on their FTP server:

ftp://ftp.supermicro.com/driver/SAS/LSI/3008/Firmware/

(Sorry, I can't make the forum software embed a URL to an FTP site.)

I would confirm with Supermicro tech support before purchasing the card, but I have a high level of confidence that it can be used in IT mode.

 

[depasseg]

I don't have the AOC card, but an on board 3008 based LSI chip on my supermicro motherboard. I've flashed it using the stock LSI/Avago firmware (because SM was months behind providing an update). No guarantees that the AOC will work, but if it were me, I'd be comfortable with it.