Advice on an X8/Xeon 5500 system

[friolator]

We want to build a FreeNAS system to act as a rolling backup for our high performance RAID-Based SAN. Basically, we want a clone of our SAN for disaster recovery purposes and nothing more. Currently we've got about 256TB of space on the SAN, but may expand it in the future. On the FreeNAS side, performance isn't a huge concern, since it's really just about having a backup.

I happen to have several brand new Xeon 5500 CPUs in the closet, so I'm thinking of a system that looks something like this:

SuperMicro X8DTH-iF
192GB DDR3-1333 ECC Registered DIMM
Dual Xeon 5500 (each 4 core)
Intel or Chelsio 10Gbe NIC

I'd pick up a used 36-drive server chassis to house this

We have several LSI cards kicking around from when we last played with FreeNAS a few years ago, and that motherboard supports quite a few PCIe cards, so we'd probably add some more. And we have many, many drives of various sizes and speeds in the closet, ranging from 2TB to 8TB, both 3gbps and 6gbps. We'd likely have a mix of them. As we expand the SAN's RAID with larger drives roughly every year, the decommissioned drives would move into the FreeNAS box, so our available backup space would increase.

My plan is to make an SMB share on the FreeNAS box to correspond with each shared volume on the SAN. One of our Windows servers would run incremental backups to move the data from the SAN to the appropriate bucket on the FreeNAS box each night.

This is not intended to be a long-term backups solution, but more of a mirror of the SAN, in case of a failure of the main system.

My questions are:

1) Does the above system sound like it'd work well? I know it's not the most up to date, but we just want stability here and we're on a budget. Speed is secondary in this case.
2) Can we use fewer controller cards and a chassis with port expanders?

Thanks!

 

[PhiloEpisteme]

1) Does the above system sound like it'd work well? I know it's not the most up to date, but we just want stability here and we're on a budget. Speed is secondary in this case.

Your board is quite old and you'll see a higher power draw for the same performance as compared with a newer board. If the IOPS isn't a huge issue you may choose RAIDZ2 with drives of matching size for your vdevs so that you get a reasonable amount of parity and a better available storage to parity ratio.

2) Can we use fewer controller cards and a chassis with port expanders?

If you have an expander backplane you can absolutely go with fewer controller cards. If you get a direct-attach backplane or one without an expander you'll need n/4 internal SAS connectors (assuming you go all SAS connectors) where n is the number of drives. The best configuration will depend on the exact backplane model you get. Once you select one go ahead and post it.

Chelsio 10Gbe NIC

If I were you I'd hold off on this to start; see if you can saturate your 1G ports on your board first. If you're planning to do sync writes you'll likely need a SLOG to achieve this.

Currently we've got about 256TB of space on the SAN

Are you trying to match this available storage in your backup? If so it might be a bit difficult with only a 36-bay chassis. Consider that even with say 12-drive RAIDZ2 vdevs leaving you ~30 drives worth of available storage and the suggestion to stay at or under 80% storage means you'll be aiming for a total available storage space of ~320TB or 10.6TB per drive.

 

[friolator]

Your board is quite old and you'll see a higher power draw for the same performance as compared with a newer board. If the IOPS isn't a huge issue you may choose RAIDZ2 with drives of matching size for your vdevs so that you get a reasonable amount of parity and a better available storage to parity ratio.

Thanks - yeah, definitely not current, but I don't really care about speed here. this is just a failsafe system in case the SAN goes down. We want something faster than, say, an LTO autoloader, but it doesn't need to be able to move 1800+ MB/s like the SAN itself. Other than the power usage though, it should be an acceptable hardware setup?

If I were you I'd hold off on this to start; see if you can saturate your 1G ports on your board first. If you're planning to do sync writes you'll likely need a SLOG to achieve this.

We'll be using the 40GbE network because of the network topology. Our 1Gb network isn't connected to the SAN, so we'd put this machine on one of the 10GbE ports (using copper DAC breakouts cables).

Are you trying to match this available storage in your backup? If so it might be a bit difficult with only a 36-bay chassis

Sorry - I probably should have been clearer about this. Our TigerStore license for the SAN is for 256TB. Of that, we're currently using about 170T for stuff that would need to be backed up regularly. We have a ton of LTO-7 sized volumes on the SAN, which we use as temporary holding bins for stuff that will be written to tape for long-term backup, before being deleted. Or for client work that requires delivery on LTO (common in the film world). None of those LTO volumes require frequent backup, since what's in them is just a temporary copy of stuff that exists on the main work volumes (which would be backed up to the FreeNAS box).

So the actual requirement would be for probably about 180TB for now. When we upgrade the SAN for higher capacity, which would likely happen sometime next year, the 8 and 10TB drives currently in the SAN would likely be replaced with 12 or 14TB drives (depending on what's cost effective at that time), and those old drives would then be moved over to the FreeNAS system to increase its holding space with an external JBOD, a couple of which we have in storage.

Thanks!

 

[PhiloEpisteme]

Other than the power usage though, it should be an acceptable hardware setup?

I imagine you can make it work, yes. I didn't mean to have my post read as "DON'T DO THIS", more of just a caution on power usage. The only other thing I'd keep in mind is that IF you want to upgrade later you may want to consider using HBAs etc that can migrate with you to another board. Its a minor point though as most likely the HBAs can migrate but the CPUs and memory wouldn't.

So the actual requirement would be for probably about 180TB for now. When we upgrade the SAN for higher capacity, which would likely happen sometime next year, the 8 and 10TB drives currently in the SAN would likely be replaced with 12 or 14TB drives (depending on what's cost effective at that time), and those old drives would then be moved over to the FreeNAS system to increase its holding space with an external JBOD, a couple of which we have in storage.

This seems very reasonable. Do keep in mind that even if you don't care about performance going above 80% can dramatically trash your performance; and completely filling your pools can cause all sorts of headaches including ruining your backup in the worst case. So I'd take that 180TB requirement and treat it as a 225TB requirement. Using my example 3x RAIDZ2 volume setup as before you're looking at an average size of 7.5TB. You can go with RAIDZ1 to eek out 3 extra drives worth of space; in my opinion it won't make a huge difference on the drive size requirement and will leave you with only single-disk failure protection.

A thing to keep in mind is that if you fill up every single bay in this machine it'll be hard to burn in new drives with the same machine. It may be worthwhile having new, burned in drives at the ready in the event one fails so that you can replace it right away rather than waiting days or over a week to complete all the smart and badblocks tests you'll need to do. You can certainly use another machine for this or you can pick up a cheap external 2-bay enclosure; though I'd recommend avoiding any USB enclosures and going with SAS enclosures which will bring prices up a bit due to an HBA with external ports being required and a 4-bay enclosure. Though, now that I write it out; if you end up with 1 extra SFF8087 port inside your computer you can use a simple passthrough card to get an external port; then the added cost is only the external enclosure.

It sounds like you're not going to have to purchase much to put this machine together, right? If so I'd say give it a shot. If the board etc are too slow for your use case or the electricity bill comes to 4 million dollars a month you can decide to upgrade components or just eat the cost of the chassis and call it a day; not much lost.