Any Experience With 50+ Drives/45Drives' xl60?

[Mirado]

(I apologize if a lot of this is uninformed or very wishful thinking, but that's why I'm asking first rather than crying later.)

I've hit the limit of what my poor little Baby's First FreeNAS Box can do. I designed a very small form factor build: six drives (in RAIDZ2), mini-ITX board, Node 304 case, etc. It was never going to be what you "should" build; I'm not using ECC RAM or a mobo that supports it (I know I should), it's not on a UPS (I know it should be), and I didn't leave any possibility to expand it from the one Vdev (although that was never the goal).

At least I used mirrored boot drives. :D

My use case also rapidly expanded; first designed just for cold storage, it started running a Plex instance (which also ballooned from transcoding one 1080p stream to serving three, something I'm surprised the little i3-3220 can handle), and then a Virtualbox jail, and then one for owncloud (with lighttpd), and then it starting serving two workstations, and then four. Everything ballooned, except for the capabilities of the little box running FreeNAS.

So, this time we're going to do things right. Not six drives, but sixty. Redundant power supplies, ECC RAM, a good UPS, the works...unless I shouldn't, and that's why I'm here. Don't worry about the money portion of it; I'm not looking to burn a stack of cash just because I can, but rather I want to invest money in a system that will handle my inability to forecast my needs (I always wind up underpowered, so we're here to fix that this time), at least for a while. I also understand that a NAS isn't a proper backup; those will be handled offsite, don't worry. ;D

I've been eyeing up 45Drives' xl60 Turbo and BackupPods' StoragePod 6.0, as they both hit the raw storage I want in a form factor that is appealing for a home user. I pretty quickly disqualified the StoragePod as it has a number of red flags that I've identified (non-redundant PSUs, older generation of mobo/CPU, single boot drive, possibly flaky/no-name backplanes), and even if some of those are correctable, the xl60 from 45Drives has a number of advantages:

• newer generation of motherboard (X10DRL)
• dual processors (perhaps overkill, but I'd rather go over than under)
• the option to use the well-loved LSI9201 for the HBAs
• the option to configure 256GB of ECC RAM (but more on that in a bit)
• redundant PSUs and boot drives

Obviously, the xl60 Turbo's price tag far surpasses that of the StoragePod 6.0, but if we're going to build a system that can handle that many drives, the drives are still going to wind up as the largest part of the expense (I'm aiming for density, so at least 6TB HDDs, with the possibility of 8TB+ if they are available to consumers in volume and are proven reliable). However, I don't really know, or at the very least I don't have any experience when it comes to a build of this size. Before I make a big mistake and toss $50k out a window, I've got a number of questions:

1. Does anyone have experience with 45Drives, particularly with the xl60?
2. The power supplies are (proper 2N) redundant, but they are from a manufacturer that I have zero experience with. (Zippy?) I assume they wouldn't sell me a $10k box that will just up and light itself on fire, but seeing as I'd rather figure that out before it does just that, is that a cause for concern?
3. Does anyone have experience operating 50+ drives with FreeNAS, particularly in a home environment? Enterprise experience is welcome, but keep in mind I'm making something for under 10 users, and the technical support/administration staff consists of...me. I'm trying to discover any possible pitfalls that FreeNAS might have at that size, since I'm used to <10 drives where everything runs just fine.
4. Would you recommend an alternate company, or perhaps something more custom? I'm not necessarily looking for cheaper; I can scale the number of drives I stuff into the system to fit our needs/budget at the start, but I want room to grow and the horsepower to handle it right out of the gate.
5. How would you best split up 60 drives into vDevs? Given the potential for failure when you start using that many moving parts, I don't think 10 six-drive RAIDZ2 Vdevs in one pool are going to cut it, so how would you balance usable space, failure tolerance, and reslivering speed? Assume we are working with 60 8TB drives right off the bat (and if you need more workload/usage context, assume I need to transfer 28TBs of mixed media from the old NAS, that we have four users at any given time who accumulate a total of 50TBs a year, and that at least two users will want to work on 4k video editing right off of the NAS simultaneously, so transfer speed is an important factor). We're looking into 10 Gigabit home networking (obviously another can of worms that I won't get into right now), so assume the NAS is the bottleneck.
6. On RAM: 45Drives offers up to 256GB. From what I've read, you should be aiming for 1GB per TB of storage, before adding in whatever nonsense I'm going to put the poor thing through on top of that. Should I be looking at 512GB (at least)? I'd have to use 64GB sticks which would be a big jump in cost, although I suppose I could look at something like a X10DRi-T instead, as that gives me double the slots to work with.
7. How would you expand this? Yeah, it's probably crazy to think about expansion as I'm jumping from six drives to sixty, but I assume the dual Xeon's will last me far longer than the space will stay free, so how would you go about adding more drives into a system that is already using all of its PCI-E slots on HBAs and a 10 gigabit NIC? At that point, would it better to build something completely custom?
8. How hot is this thing going to get? It'll be in a clean basement that stays pretty cool (70F) even during the (95F) summer (and has HVAC, plus the winter can hit 0F or below so ambient temperature isn't an issue), but I'd hate to discover this just isn't going to fly in a home if I'm grossly underestimating the space-heater factor. Again, forget electric bills, think more "we physically cannot keep this 500sqft room at operating temperature with residential equipment."
9. What kind of UPS would you slap on this thing? Would any consumer UPS be up to the challenge? I assume we'd be looking at a pretty slow spin-down time given the sheer amount of disks that would have to be parked (and therefore the UPS would have to last longer than average), but I'm just grasping at straws.

Right now, we're in the advice and planning stage, so I'm not about to make any foolish decisions without first ignoring plenty of reasonable advice. If I've missed anything major or if you need any more information, please let me know. Nothing is set in stone beyond:

1. I want a dual Xeon board, price be damned.
2. I'm looking at the best way to handle 50+ disks in FreeNAS.
3. I'm open to either a ready-made solution like one from 45Drives, or something custom. I'm not looking to build it myself from scratch.

Thank you in advance.

 

[depasseg]

I looked into a 45 drives chassis, and between the high cost and backplane expanders, I decided to go a different route.

I found a better solution is to get a great 2U supermicro server (12 drive), and a 4U 45 drive SuperMicro JBOD enclosure (you can also get a 40-ish SuperMicro server). And I can keep adding enclosures (via external SAS connections). I got mine on ebay, but I'm not sure if that is available for you. This way I have all the "enterprise" capabilities I need like remote management, dual psu's, supported expanders and hot swap drive enclosures. RAM will be dependant on the total usable storage space, but you will definitely be north of 128GB and likely closer to 256 if not beyond.

I have a bunch of other gear in my basement and temps will go up to 80 (we have A/C upstairs, not in the whole basement though), so I had to figure out a solution (open door to main house, and recently added a little duct off the main A/C trunk to blow into the front of the rack. Temps are now near 70.

Performance will need more analysis, but as a rule of thumb, a Vdev will have the bandwidth of the sum of the data drives and the IOPS of the slowest drive. And vdevs are additive, since they are striped.

I'd look into a larger UPS (probably in the 3000VA range).

 

[danb35]

I think I'd suggest something similar to @depasseg. I lucked into my rig (details in my sig) for about $1100 on eBay; I haven't seen that price pop up again, but I've seen similar units for $2000 or less. Before you observe (correctly) that the 847 chassis only supports 36 drives, note that this unit already comes set up with external SAS connectors wired in to the backplanes. Adding drive shelves (whether the SuperMicro 45-bay units or something else) is as simple as plugging in a SAS cable, repeating as many times as necessary.

Edit: Heck, you can get that server new for under $2700. Of course, you'd add CPUs, RAM, and a 10G NIC (to match the 45drives.com unit), but you'd still be $6k or more under the price of the 45drives.com unit. Add a 45-bay JBOD for under $1k shipped (or new for $2k), and you've got room for 81 drives in a well-supported, solid configuration, and you've saved $5k (though at the expense of another 4U of rack space).

 

[cyberjock]

So you are going to be talking quite some cash here no matter what. 45 drive chassis are just... expensive. :/

Do consider that maybe you're better off with 2 servers instead of 1 very large server. This may make financial sense just because the 64GB DIMMs are so much more expensive than the 32GB DIMMs. I haven't done the math, but at least leave yourself open to the possibility and do the math. It isn't as "nice" to have to admin 2 server, and definitely won't help your e-peen. But 2 can be MUCH less expensive, and if you have some data that is super critical, you can also zfs replicate between the 2 servers if need be.

If you are going with 6-8TB disks, you'll probably be fine with 256GB of RAM. Its really about what your demands are. If your intention is to run 100+ VMs, you may want more RAM. If you plan to do file shares for a few dozen users, you'll almost certainly be fine with 256GB of RAM.

 

[danb35]

SuperMicro also has a 60-bay server that looks a bit less expensive than the 45drives.com unit, and which I think I'd probably trust more. They even have a 90-bay unit, but that's getting much more expensive (though I note that the $19k price includes 45 drives, which appears to be the minimum configuration in which they'll sell this machine--that'd put the "bare bones" price at about $11k). With any of those, though, as well as the 45drives unit, I'd be concerned about cooling. Even my 847 runs a bit warmer than I'd like.

 

[danb35]

I mentioned it in my last post, but should probably expand on it: drive cooling is important, and the more densely you pack the drives, the more important (and harder) it gets. My 847 chassis sounds like a jet engine, only has one layer (for lack of a better word) of drives (three of the four columns of drives on the front panel are full; the fourth has the dummy drives in place to help with airflow), and still struggles to keep the drive temps at or below 40 deg C, which is the warmest you want them on any regular basis. I don't see how the 45drives units (or the SuperMicro 60/90-bay units) can get enough airflow over the drives to cool them adequately.

Housing the server in a cool basement will help, but that drive layout just sounds like it's asking for cooked drives.

 

[Mirado]

Very, very helpful replies, people. Thanks a ton!

I think I'd suggest something similar to @depasseg. I lucked into my rig (details in my sig) for about $1100 on eBay; I haven't seen that price pop up again, but I've seen similar units for $2000 or less. Before you observe (correctly) that the 847 chassis only supports 36 drives, note that this unit already comes set up with external SAS connectors wired in to the backplanes. Adding drive shelves (whether the SuperMicro 45-bay units or something else) is as simple as plugging in a SAS cable, repeating as many times as necessary.

Edit: Heck, you can get that server new for under $2700. Of course, you'd add CPUs, RAM, and a 10G NIC (to match the 45drives.com unit), but you'd still be $6k or more under the price of the 45drives.com unit. Add a 45-bay JBOD for under $1k shipped (or new for $2k), and you've got room for 81 drives in a well-supported, solid configuration, and you've saved $5k (though at the expense of another 4U of rack space).

This is fantastic! I had a hunch that an all-in-one solution would be far more expensive, but if you are telling me I can potentially save $5k at the cost of 4U of rackspace (trivial, really), and have the potential for that kind of expansion (something that worried me about the xl60), that fits exactly what I was hoping to do.

I looked into a 45 drives chassis, and between the high cost and backplane expanders, I decided to go a different route.

I found a better solution is to get a great 2U supermicro server (12 drive), and a 4U 45 drive SuperMicro JBOD enclosure (you can also get a 40-ish SuperMicro server). And I can keep adding enclosures (via external SAS connections). I got mine on ebay, but I'm not sure if that is available for you. This way I have all the "enterprise" capabilities I need like remote management, dual psu's, supported expanders and hot swap drive enclosures. RAM will be dependant on the total usable storage space, but you will definitely be north of 128GB and likely closer to 256 if not beyond.

I have a bunch of other gear in my basement and temps will go up to 80 (we have A/C upstairs, not in the whole basement though), so I had to figure out a solution (open door to main house, and recently added a little duct off the main A/C trunk to blow into the front of the rack. Temps are now near 70.

Performance will need more analysis, but as a rule of thumb, a Vdev will have the bandwidth of the sum of the data drives and the IOPS of the slowest drive. And vdevs are additive, since they are striped.

I'd look into a larger UPS (probably in the 3000VA range).

Well, the one thing that I love about the Supermicro route is the ability to go with a larger form factor, 8 DIMM slot motherboard (or more). which should keep my RAM costs down if I want to hit 512GB. I think I've been pretty well convinced that it may be better to do this in two pieces as you describe. Also, a huge thank you for including your temperature experience; we've got a HVAC zone set up in the basement so I'm hoping temperatures remain consistent, and our winters are cold enough that I might actually wind up saving money during those months, but hopefully I don't burn it out during the summer or I'll probably never hear the end of it. I'll keep the UPS size in mind as well.

So you are going to be talking quite some cash here no matter what. 45 drive chassis are just... expensive. :/

Do consider that maybe you're better off with 2 servers instead of 1 very large server. This may make financial sense just because the 64GB DIMMs are so much more expensive than the 32GB DIMMs. I haven't done the math, but at least leave yourself open to the possibility and do the math. It isn't as "nice" to have to admin 2 server, and definitely won't help your e-peen. But 2 can be MUCH less expensive, and if you have some data that is super critical, you can also zfs replicate between the 2 servers if need be.

Yeah, I kind of resigned myself to spending first class international airline tickets level of dollar bills no matter what route I went. In the end, money is (thankfully, I understand how lucky I am) secondary to getting the right setup, and I'm already on somewhat shaky ground with the complexity of one server, let alone two. You are probably right about the data replication bit, but anything that critical is usually backed up off site, and the vast (95%) majority of the data isn't mission critical. It's going to be a home server, even if the hardware is starting to straddle the line into enterprise territory. (Or, y'know, leap right over it.)

I mentioned it in my last post, but should probably expand on it: drive cooling is important, and the more densely you pack the drives, the more important (and harder) it gets. My 847 chassis sounds like a jet engine, only has one layer (for lack of a better word) of drives (three of the four columns of drives on the front panel are full; the fourth has the dummy drives in place to help with airflow), and still struggles to keep the drive temps at or below 40 deg C, which is the warmest you want them on any regular basis. I don't see how the 45drives units (or the SuperMicro 60/90-bay units) can get enough airflow over the drives to cool them adequately.

Housing the server in a cool basement will help, but that drive layout just sounds like it's asking for cooked drives.

That was another worry; I'd be willing to make sure the drives stay at a reasonable temperature even if that meant cutting it down to a 45 drive JBOD expansion. We've got room for a few racks (not a server room level of rows upon rows, mind you, but...five? Six?) with the ability to still maneuver comfortably, so if I have to go wide rather than tall, I can do that.

 

[RichR]

The timing of this thread couldn't be better for me.....

Before getting into my needs, just a heads up. I have 8 of what used to be the former 45drives (Protocase) customized enclosures (4 in 2 data centers), and actually administer 3 more. Drive temperatures have never been a problem. Period. The only issues I've ever had have been LSI driver related, which has been pretty well documented on this and other bsd forums (unfortunately). The issue is if drives start to go bad, FreeNAS won't boot, but that's another story. Regarding temps, I ran a temperature script on one of the pods.....

Code:

[pacs@pod01] /mnt/Store1/pacs/scripts# ./temps.sh
System Temperatures - Wed Aug 24 08:52:30 EDT 2016
FreeNAS-9.3-STABLE-201504100216
 8:52AM  up 46 days,  7:11, 1 user, load averages: 0.19, 0.32, 0.28
CPU Temperature:

dev.cpu.0.temperature: 29.0C
dev.cpu.1.temperature: 25.0C
dev.cpu.2.temperature: 26.0C
dev.cpu.3.temperature: 24.0C
Drive Activity Status
ada0:FF  

HDD Temperature:
da0 22 (0 19 Z1F11FEV ST3000DM001-9YN166
da1 23 (0 20 Z1F0YFAA ST3000DM001-9YN166
da2 23 (0 20 Z1F14G4H ST3000DM001-9YN166
da3 24 (0 20 Z1F137W3 ST3000DM001-9YN166
da4 22 (0 19 Z1F11BAC ST3000DM001-9YN166
da5 22 (0 19 Z1F0YE6S ST3000DM001-9YN166
da6 25 (0 21 Z1F14GFN ST3000DM001-9YN166
da7 23 (0 20 Z1F13C0F ST3000DM001-9YN166
da8 23 (0 20 Z1F13CBB ST3000DM001-9YN166
da9 24 (Min/Max 21/34) PK1234P9HHN4JX HGST
da10 24 (Min/Max 21/34) PK2234P9HE6VPY HGST
da11 25 (Min/Max 22/34) PK2234P9HAK13Y HGST
da12 25 (Min/Max 22/34) PK2234P9HAB8AY HGST
da13 24 (Min/Max 21/34) PK2234P9H7DVVY HGST
da14 23 (0 20 Z1F1YWQR ST3000DM001-9YN166
da15 22 (0 19 Z1F14GKC ST3000DM001-9YN166
da16 21 (0 19 Z1F138HM ST3000DM001-9YN166
da17 21 (0 19 Z1F138PX ST3000DM001-9YN166
da18 20 (0 17 Z1F12SZV ST3000DM001-9YN166
da19 21 (0 18 Z1F14GEP ST3000DM001-9YN166
da20 23 (0 19 Z1F11FG9 ST3000DM001-9YN166
da21 19 (0 17 Z1F12JBA ST3000DM001-9YN166
da22 21 (0 17 Z1F1148J ST3000DM001-9YN166
da23 22 (Min/Max 18/34) PK1234P9HA606X HGST
da24 21 (Min/Max 18/34) PK2234P9HEWD5Y HGST
da25 22 (Min/Max 18/34) PK1234P9HHN5TX HGST
da26 21 (Min/Max 18/34) PK2234P9HAK18Y HGST
da27 21 (Min/Max 19/34) PK2234P9HAK0UY HGST
da28 22 (Min/Max 19/34) PK2234P9HA3RUY HGST
da29 22 (Min/Max 19/31) PK2238P3G4UH2J HGST
da30 24 (0 21 Z1F13CB7 ST3000DM001-9YN166
da31 21 (0 19 Z1F11QMV ST3000DM001-9YN166
da32 21 (0 18 Z1F11FE5 ST3000DM001-9YN166
da33 23 (0 21 Z1F140MZ ST3000DM001-9YN166
da34 24 (0 21 Z1F137E6 ST3000DM001-9YN166
da35 23 (0 20 Z1F14078 ST3000DM001-9YN166
da36 23 (0 20 Z1F115WY ST3000DM001-9YN166
da37 23 (0 20 Z1F13777 ST3000DM001-9YN166
da38 24 (0 21 Z1F140XP ST3000DM001-9YN166
da39 23 (0 20 Z1F13C1B ST3000DM001-9YN166
da40 23 (0 20 Z1F13C71 ST3000DM001-9YN166
da41 23 (Min/Max 20/33) PK1234P9HHAXZX HGST
da42 23 (Min/Max 21/35) PK2234P9HEXG6Y HGST
da43 25 (Min/Max 21/35) PK2234P9HAB4MY HGST
da44 26 (Min/Max 21/32) PK2238P3G4UVLJ HGST
ada0 30 (Min/Max 30/30) OCZ-3185WO95O1T41ZUA OCZ-VERTEX2
Done

Running in the low to mid 20's doesn't bother me at all

That being said, when I mentioned "customized" I meant truly customized. I redid all the wiring and was probably the first person to modify it to get rid of the 5-port port expanders on the bottom, and switched the drives to be wired from the top and give each direct access with an 8087 breakout cable. Although it's a slight mess, and a royal pita to service, they work quite well.

So now, we've grown up a little. It's going to be time at some point to move to something more serviceable. The biggest drawbacks of the current pods we have are:
1. They have to be shut down in order to service.
2. The rails they had at the time were total crap, so each pod is sitting on a heavy duty APC shelf so it takes up another 1U. In order to service it, it has to be removed from the shelf, put on a cart etc..... and they are HEAVY.

So I'm looking at alternatives. The new 45drives work on rails and can be serviced while running, although I still don't like the idea of pulling out a unit on rails while it's running, but that is an uninformed opinion. But, I'd still rather go with SuperMicro. The first question I have is, what is the throughput of a SAS cable connecting the suggested 2U server to a 45-drive Supermicro JBOD? It does seem to me to be another point of failure, where getting 36-drives per enclosure might be an ok tradeoff, each having it's own OS. Yes, more to administer, but that might be ok as well.

thoughts?

Rich