Connecting 24 Hard Drives to a Single NAS

[moochael]

Hello. Obligatory mentioning of this being my first FreeNAS build. I also want to apologize in advance if I ask any questions that have already been asked or if the questions are just silly in the first place. I'm getting a little disoriented with all of the options and questions that I've come across while doing some research about FreeNAS. I've seen very little information about how people connect 24 disks at the same time (maybe they don't for a good reason?)


My questions are about SAS Expanders and RAID controllers. I've done searching of these forums and found a lot of great information here and on google. But I still don't feel 100% confident that I'm ready to purchase hardware. I'm going to use 24 hard drives in a single RAIDZ2 array with FreeNAS. What is the most cost-effective way of connecting this many drives?

I've been looking at SAS Expander cards like this:

CK12803 Standard 28-port SAS Expander
http://usa.chenbro.com/corporatesite/products_detail.php?sku=73
~$230.00 before shipping

Intel RAID Twenty-four port Expander Card RES2SV240
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117207&Tpk=RES2SV240
$279.00 before shipping

Obviously bearing in mind compatibility with FreeBSD 8.2, can I use a single SAS Expander card to connect all my drives and will FreeNAS detect all the disks without a RAID controller behind the SAS Expander? Or am I required to use a RAID controller with the SAS Expander?


Another option I was thinking of (not as cheap as a single SAS Expander) is getting a motherboard with multiple PCI-E slots and then using cheap SATA controllers in conjunction with the SATA ports on the motherboard to get 24 ports. I'm having trouble finding reasonably priced 4-port SATA Controllers that support SATA 3, though.

HighPoint RocketRAID 640 PCI-Express 2.0 x4 SATA III (6.0Gb/s) Controller Card $96.00 before shipping
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115077

SYBA SY-PEX40008 PCI Express SATA II (3.0Gb/s) Software RAID Controller Card $55.00 WITH shipping
http://www.newegg.com/Product/Product.aspx?Item=N82E16816124027


I know to not set up a hardware RAID to run underneath the software RAID. That makes me feel even less confident about my research since the SATA controllers above also have built in RAID.


Thanks for you time and please feel free to also give better suggestions if I'm going about this completely wrong.

 

[freeflow]

It might be helpful if you can explain why you are setting up an array of 24 disks.

 

[matthewowen01]

you know you need a SAS controller to use a SAS expander right? the Intel one takes 2 SFF-8087 in, 4 SFF-8087 out. so it's kinda pointless unless your motherboard has SAS.

 

[moochael]

It might be helpful if you can explain why you are setting up an array of 24 disks.

It's for my boss. He just wants to get as much storage as possible with a minor amount of data redundancy in a 4U chassis. He wants to use one of these:

NORCO RPC-4224 4U Rackmount Server Case with 24 Hot-Swappable SATA/SAS Drive Bays
http://www.newegg.com/Product/Product.aspx?Item=N82E16811219038

SUPERMICRO CSE-846TQ-R900B Black 4U Rackmount Server Case 900W Redundant (1 + 1) Power Supply w/ PFC
http://www.newegg.com/Product/Product.aspx?Item=N82E16811152124

you know you need a SAS controller to use a SAS expander right? the Intel one takes 2 SFF-8087 in, 4 SFF-8087 out. so it's kinda pointless unless your motherboard has SAS.

I didn't know. But now that you say it, it does make a lot of sense and that is one of the many things I don't understand about SAS expanders.

Something like this would work behind it?:
Intel RAID Controller Card 6G SAS PCI-E x8 4 internal ports (RS2WC040)
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117172&Tpk=RS2WC040


Thank you for the replies.

 

[freeflow]

Your boss is a moron. Based on your questions above you don't appear to be competent to do the job he's asking you to do. I suggest you go and read the zfs administrators guide before you come back and ask more questions.

 

[moochael]

Your boss is a moron. Based on your questions above you don't appear to be competent to do the job he's asking you to do. I suggest you go and read the zfs administrators guide before you come back and ask more questions.

rofl

Part of your post was helpful. Thanks.

 

[moochael]

The manual for the Intel SAS expander says that it supports either 8 inputs and 16 outputs or 4 inputs and 20 outputs. I think that means I would probably want something more like this to go along with the expander:

Intel RAID Controller Card 6G SAS PCI-E x8 8 internal ports (RS2WC080)
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117171&Tpk=RS2WC080

 

[matthewowen01]

Your boss is a moron. Based on your questions above you don't appear to be competent to do the job he's asking you to do. I suggest you go and read the zfs administrators guide before you come back and ask more questions.

as mean as it sounds, it's kinda correct. it's one thing to fool around at home but if this is for a business, you've got to know what you're doing first, you can't learn as you go there. also a raidz2 of 24 drives is suicidal. to many eggs in 1 basket. go with 3 8 disk pools or 4 6 disk pools. also if this is for business, a real backup scheme should be used too.

The manual for the Intel SAS expander says that it supports either 8 inputs and 16 outputs or 4 inputs and 20 outputs. I think that means I would probably want something more like this to go along with the expander:

Intel RAID Controller Card 6G SAS PCI-E x8 8 internal ports (RS2WC080)
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117171&Tpk=RS2WC080

i don't really know enough about the expanders to give a good answer. i think if you decide to do the 4 in 20 out, you'll have performance issues if you have all the disks in the same vdev. each sata port (3 gb/s or 6 gb/s depending) is responsible for handling 5 drives. so each drive is severely limited on bandwidth.

 

[jgreco]

I'd say you can safely ignore freeflow. As someone who has built large storage arrays, it sounds to me like you just lack the practical experience with the hardware.

I haven't looked extensively at the current offerings, but I will note that you might find it easier to look at multiport SATA controllers, which are available in at least 8 port varieties. Years ago, we basically had to use RAID controllers in JBOD mode. That works too, but you typically pay a premium for the RAID feature. So for example, while you can use a pair of 3Ware 9550-12's to attach all 24 drives directly, using only two slots, that's going to be unaffordable.

If you don't need the ultimate in performance, look towards SATA port multipliers. You take a SATA port on your system (which has to be PM-compatible) and run it to this card, which in turn breaks out (typically) five more ports. So you have four SATA ports on the motherboard, plus four PM-capable SATA ports on an expander that go to four port multipliers, which gives you twenty more ports.

Go read http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ if you intend to do the port multiplier thing. If you don't understand it after reading all of that - don't try it. But it sounds to me like you're just having trouble with the implementation, not the concept.

 

[moochael]

as mean as it sounds, it's kinda correct. it's one thing to fool around at home but if this is for a business, you've got to know what you're doing first, you can't learn as you go there. also a raidz2 of 24 drives is suicidal. to many eggs in 1 basket. go with 3 8 disk pools or 4 6 disk pools. also if this is for business, a real backup scheme should be used too.

i don't really know enough about the expanders to give a good answer. i think if you decide to do the 4 in 20 out, you'll have performance issues if you have all the disks in the same vdev. each sata port (3 gb/s or 6 gb/s depending) is responsible for handling 5 drives. so each drive is severely limited on bandwidth.

It's not for a business. I didn't include who it was for in my original post because it seemed like it wasn't important.

My 'friend' and I have talked about doing 1 pool as RAIDZ2 and we've talked about 6 pools as RAIDZ1 (he thinks we need less redundancy than I do). We've also discussed reserving disks for hot-swap spares. We're also both aware that, regardless of how many pools or disks, that this has no data backup on its own. What we were most confused about was how to connect this many disks.

I think we'll take your advice and avoid a single vdev. You might think your help isn't the most valuable but it's more information than I have and it's also leading me to answers and other clarifications. You're being incredibly helpful so thank you, again.

 

[moochael]

I'd say you can safely ignore freeflow. As someone who has built large storage arrays, it sounds to me like you just lack the practical experience with the hardware.

I haven't looked extensively at the current offerings, but I will note that you might find it easier to look at multiport SATA controllers, which are available in at least 8 port varieties. Years ago, we basically had to use RAID controllers in JBOD mode. That works too, but you typically pay a premium for the RAID feature. So for example, while you can use a pair of 3Ware 9550-12's to attach all 24 drives directly, using only two slots, that's going to be unaffordable.

If you don't need the ultimate in performance, look towards SATA port multipliers. You take a SATA port on your system (which has to be PM-compatible) and run it to this card, which in turn breaks out (typically) five more ports. So you have four SATA ports on the motherboard, plus four PM-capable SATA ports on an expander that go to four port multipliers, which gives you twenty more ports.

Go read http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ if you intend to do the port multiplier thing. If you don't understand it after reading all of that - don't try it. But it sounds to me like you're just having trouble with the implementation, not the concept.

I'm so happy that you speak my language! This is exactly the kind of thing I've been looking for. High performance isn't really a concern so I would love to avoid the RAID controllers and SAS expanders and just use the SATA multipliers (I didn't know they existed.)

Thanks a ton.

 

[Milhouse]

Go read http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ if you intend to do the port multiplier thing. If you don't understand it after reading all of that - don't try it. But it sounds to me like you're just having trouble with the implementation, not the concept.

Also this critique of the Backblaze hardware design, which frankly sucks except for their very specialised usage case.

 

[jgreco]

Also this critique of the Backblaze hardware design, which frankly sucks except for their very specialised usage case.

The hell it does. The Backblaze hardware design is fine for what it is, and is suitable for a lot more than "their very specialised usage case." There are LOTS of storage applications involving large amounts of disk that do not need massive amounts of concurrency; the Backblaze design is exceedingly cost-reduced to the point of compromising away performance in the system board and busses, but that's a sensible compromise for them, and anyone can see how to reverse that particular engineering choice - by picking a different board and SATA interfaces. At the heart of it all, no conventional SATA disk is really capable of saturating a SATA 3Gbps interface, and so the use of a port multiplier is the biggest tradeoff - and that's not really a big impact. Oversubscription by a factor of two? So what. On a system like that, you'd need FOUR ten gigabit network cards to be able to max out traffic. Let's be real. The throughput thing can be made better, a lot, through different system board choices. The power supply complaint is basically stupid. It's a pragmatic choice on their part, which of course other implementations can do in other ways.

Basically, the Backblaze model is very much in the spirit of FreeNAS. It's not a maximal design (except in terms of space) and it wasn't meant to be, just as there are better, more expensive ways to implement NAS than FreeNAS. If I wanted to build a FreeNAS box with 45 drives, on the cheap, the Backblaze page is the first thing I'd look at, because someone else has done a lot of the work already and have figured out all the potential corners that can be cut. You can upgrade from that design as you see fit.

So what we really have is some Sun guy who is blogging about why Sun does it better than Backblaze. Fine, Sun does it better than Backblaze. But I can buy like five of the Backblaze units for the cost of one of the Sun units, who's laughing then?

There's a legitimate need out there for large amounts of storage that isn't heavily smashed on. I've got data around here that hasn't been accessed in a decade, but it's handy to have it for that time when I want it. Certainly more handy than having it offline somewhere. I'd really like to have a 12-bay NAS system that was low power, low volume. And by low power, I don't mean 300 or 400 watts.

 

[Milhouse]

The hell it does. The Backblaze hardware design is fine for what it is, and is suitable for a lot more than "their very specialised usage case."
...
The power supply complaint is basically stupid. It's a pragmatic choice on their part, which of course other implementations can do in other ways.

The power distribution design is a major flaw - possibly it's biggest - amongst others. If you lose one of the two PSU you lose either the board and half the disks, or just half the disks. And as "some SUN guy" has already pointed out in the linked article, even with RAID6 you can't configure the disks in such a way that the system continues to work when you lose only the PSU powering just the disks (if you lose the other PSU, you lose the entire system but either way lose a PSU and you're up sh1t creek). Basically, the system has very little redundancy worth talking about. Adding redundant power to the design isn't prohibitively expensive either, it's just one corner cut too far in this particular design.

As a hardware implementation it has several very major problems with too many single points of failure, meaning you either accept all of these problems and work around them in your application design by having multiple pods, or you choose a better and more redundant hardware design and software solution (redundant PSUs, NICs, battery-backed RAID5/6 or something like ZFS etc.).

Running a system from a single pod would be foolhardy in the extreme, but the Backblaze was designed for a single purpose and for that it's fine. However suggesting it's a good design for a more general purpose storage solution is naiive.

 

[jgreco]

The power distribution design is a major flaw - possibly it's biggest - amongst others. If you lose one of the two PSU you lose either the board and half the disks, or just half the disks. And as "some SUN guy" has already pointed out in the linked article, even with RAID6 you can't configure the disks in such a way that the system continues to work when you lose only the PSU powering just the disks (if you lose the other PSU, you lose the entire system but either way lose a PSU and you're up sh1t creek). Basically, the system has very little redundancy worth talking about. Adding redundant power to the design isn't prohibitively expensive either, it's just one corner cut too far in this particular design.

As a hardware implementation it has several very major problems with too many single points of failure, meaning you either accept all of these problems and work around them in your application design by having multiple pods, or you choose a better and more redundant hardware design and software solution (redundant PSUs, NICs, battery-backed RAID5/6 or something like ZFS etc.).

Running a system from a single pod would be foolhardy in the extreme, but the Backblaze was designed for a single purpose and for that it's fine. However suggesting it's a good design for a more general purpose storage solution is naiive.

If you don't like the amount of RAM they decided on, you add more. If you don't like the motherboard they used, you pick another. If you don't like the size of hard drive they decided on, you pick another. If you don't like the power supplies they selected, you pick another. Something that is trivial to fix in this manner is not a "major flaw". It's a pragmatic choice to control costs. They do make large redundant 1800W power supplies but they're *expensive*. You don't like RAID6? I would think a FreeNAS system would use ZFS - because we have it available. You don't like a single NIC? Add another. Geez.

The commercially available chassis alternative to the Backblaze offering, the RMC5D, is around $4000. Chassis and power supply only. Now go look at the Backblaze costs. It should be clear that they cut a lot of corners, but if it's not clear to you that they've also hit a home run by coming up with a design that serves as a good starting point for many applications, you need to look harder. For them to create what they did, and then offer the design for free to the public, is fantastic.

I wish that it had been around years ago. My storage server design from 2004, which some scaled up to the petabyte range, was at the time one of the most dense and inexpensive designs available. I can appreciate the tradeoffs that they've made, but I cannot figure out why you cannot see that most of the "flaws" in their design are actually pragmatic choices to control costs, choices for which you are free to select other options, especially since they're giving away their design for free.

 

[Milhouse]

If you don't like the amount of RAM they decided on, you add more. If you don't like the motherboard they used, you pick another. If you don't like the size of hard drive they decided on, you pick another. If you don't like the power supplies they selected, you pick another. Something that is trivial to fix in this manner is not a "major flaw". It's a pragmatic choice to control costs. They do make large redundant 1800W power supplies but they're *expensive*. You don't like RAID6? I would think a FreeNAS system would use ZFS - because we have it available. You don't like a single NIC? Add another. Geez.
...
I can appreciate the tradeoffs that they've made, but I cannot figure out why you cannot see that most of the "flaws" in their design are actually pragmatic choices to control costs, choices for which you are free to select other options, especially since they're giving away their design for free.

I'll state for a third time that I don't have a problem with their design for *their* use case, but as a general solution it's got too many shortcomings for it to be lauded as anything special - the only impressive feature of the Backblaze is the drive density.

You've admitted yourself, to overcome these problems and make it into a more practical solution you have to start from scratch, building and designing your own solution - all you would take from the Backblaze is the metal chassis and its backplane modules. If you're going to do that you may as well just get a Norco DS-24ER which would be perfectly adequate for the OP and his proposed 24 disks - and take it from there, the result wouldn't be any different and I'm perplexed as to why you think the Backblaze is so wonderful. Maybe it is if you're looking for a solution in their line of business with crazy high drive density, but I suspect most here are not.

 

[jgreco]

I'll state for a third time that I don't have a problem with their design for *their* use case, but as a general solution it's got too many shortcomings for it to be lauded as anything special - the only impressive feature of the Backblaze is the drive density.

You've admitted yourself, to overcome these problems and make it into a more practical solution you have to start from scratch, building and designing your own solution - all you would take from the Backblaze is the metal chassis and its backplane modules. If you're going to do that you may as well just get a Norco DS-24ER which would be perfectly adequate for the OP and his proposed 24 disks - and take it from there, the result wouldn't be any different and I'm perplexed as to why you think the Backblaze is so wonderful. Maybe it is if you're looking for a solution in their line of business with crazy high drive density, but I suspect most here are not.

My line of business pushes crazy high drive density.

One point is that there's nothing really wrong with the Backblaze design. You can choose other hardware particulars and wind up in various places in the low- to mid-range IOPS spread. It's not the way to go if you need network attached storage capable of 40Gbps to and from all drives simultaneously, but then again, at that point you're EITHER far beyond commodity hardware OR you've come to the realization that it's cost-effective to use gear that costs 1/5th to 1/50th the "heavy iron" solutions and instead engineer your requirements to accommodate the platform's tradeoffs. One lesson that people *still* learn is that the big magic proprietary server can - and does - fail, nothing's 100.0000000% uptime, so in some cases you get better reliability by going cheap-but-redundant. One only need look at the term RAID to understand this ... just in this case, a Redundant Array of Inexpensive Servers.

But the other point is that I brought this up while explaining port multipliers to a guy who was clearly struggling with the attachment hardware. In years of having done massive storage architectures, I can think of NO other example of a massive architecture that was so clearly and concisely described, for the general public, readily available for anyone to reference on the Internet, that described a practical platform in such an easy-to-understand way.

This guy started OUT with a Norco 24-drive case. He didn't understand the specifics of how to attach his drives. I gave him a pointer to a solution. You are just saying he "may as well just get a Norco DS-24ER which would be perfectly adequate" ... which is an exercise in useless recursion, since he's already figured that bit out.

Now, which one of us is helping the OP understand how to solve his problem?

 

[Milhouse]

Now, which one of us is helping the OP understand how to solve his problem?

The response from "some SUN guy" highlights the pitfalls of the Backblaze design which should help the OP in spades - it clearly and succinctly highlights the problems/drawbacks of port multipliers, and should help the OP come to a decision whether to use them or not, and if they are going to be used then how to avoid "overloading" them.

 

[jgreco]

The response from "some SUN guy" highlights the pitfalls of the Backblaze design which should help the OP in spades - it clearly and succinctly highlights the problems/drawbacks of port multipliers, and should help the OP come to a decision whether to use them or not, and if they are going to be used then how to avoid "overloading" them.

You do realize that that doesn't even begin to be a problem unless the guy has something greater than 3 gigabit ethernet interfaces fully saturated on his system, right?

I highly doubt that that's helping the OP in spades.

 

[Milhouse]

You do realize that that doesn't even begin to be a problem unless the guy has something greater than 3 gigabit ethernet interfaces fully saturated on his system, right?

I highly doubt that that's helping the OP in spades.

I do, and I also realise that network bandwidth isn't the only criteria when considering whether compromised disk performance is a good idea or not. Even though the OP has specified limited redundancy, knackering his disk IO will result in longer elapsed time to complete a scrub/resilver.