SAS expanders

Status
Not open for further replies.

Bigtexun

Dabbler
Joined
Jul 17, 2014
Messages
33
I'm starting a thread to talk about some technical details of SAS expanders, I was starting to discuss this on a SATA port multiplier thread, and while there was proper contect when I entered the thread, really this should be it's own thread. Port multipliers appear not to work any more under FreeBSD, and rather than belabor that issue, I think a fresh discussion of what hardware we should be using is called for.

So I have a handful of brand new LSI 9211-8i's on the way, and I was reading the installation guides for a couple Intel SAS expanders, one of which, the 24 port model, is recommended by a couple. There is also a 36 port version. I'm needing port density, so the 36 port card is the one I am interested in.

I'm told that the 24 port card is really 16 ports, and two of the 4-port SAS ports are reserved for connecting to the host controller card. But the documentation actually says that any port can be a drive or controller port, but only if you want the drive numbering to be sequential is this a requirement.

For the 36 port card, there seems to be some flexibility, and perhaps even downright ambiguity about how the ports are to be used. The documentation states:

"Each connector on the SAS expander card can be used as a “cable in” (SAS Controller to SAS Expander) or “cable out” (SAS Expander to Hot Swap Backplane) type connector. However, for contiguous drive mapping (0 – 16 or 0 – 24), cable routing differs when using a x8 wide-port capable 6 Gb SAS/SAS RAID Controller vs. using the embedded SCU ports."


So note the "0-16 or 0-24" reference. Later in the document they state that to be bootable, the drives must be directly connected to the controller, so without going into detail, they mention connecting one cable to the drive backplane, and the other to the expander.

The 36 port expander can have up to 24 contiguously numbered drives, and the remaining three ports only require one or two to be connected to the controller. So since all ports can be drives of controller ports, does this mean the unused ports can be connected to drives, just without contiguously numbered drives?

Keep in mind this document assumed in advance that you are using a specific chassis, which only has 24 drive slots, so the document doesn't address possibilities that might be encountered by someone with more slots. So all of the above may be simply to match the numbers in the drive bays, and in no way limits anything.

So has anyone tried either of these expanders with only one cable to the controller? Other than a reduction in throughput, does this work? Can I use all available ports for drives, so 32 drives on a 36 port board? Can I put a second expander on the second SAS connector, and get 64 drives on-line?

My computer case can hold 324 drives, and while I will never have that many drives in the box, I /do/ have more drives available than I am likely to ever be able to plug into a single FreeNAS system. So the more drive slots I can use, the more I /will/ use. In all of the NAS systems I'm trying to build, performance is a non-issue, this is zero-user bulk storage.

I'll be testing these questions myself, but I know other people have used these cards, so I'm curious how much expansion you can get on a 8-port LSI 9211-8i using these.

I just confirmed that someone else did in fact get a 24 port board working with only one controller cable... He also confirmed you cannot boot through the expander, and where you plug into the expander will influence the drive numbering, so if your first 4 drives are not through the expander so they are bootable, where you plug into the expander will cause the first drive to become drive 4. So perhaps the talk about how to hook the things up is mostly about getting the drive numbers right (I could care less about that). But that user did not try to use all ports, so the jury is still out on that question.
 

diehard

Contributor
Joined
Mar 21, 2013
Messages
162
Just curious, what case are you using? I'm assuming you didnt mean 324 drives..

Is it really a JBOD chassis? Most of those case's will have the SAS Expander on the backplanes...
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
324 drives! Screw you for not sharing buddy!

On a serious note I did read somewhere that someone did hook up a single 4xSAS link cable to an Intel SAS expander and populated all of the remaining links with hard drives and they all worked. They weren't bootable. This is how I'd expect SAS to behave. But I'll be damned if I can find the link where I read this. It was over a year ago. :/

Anyway, SAS is supposed to work on a link basis. What you connect to that link shouldn't matter. I say "shouldn't" because I'm sure there's some dishonest company out there that is trying to use some weird situation to force you to buy extra hardware so they can make more money.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
An SAS expander is kind of like an ethernet switch without spanning tree. In general there aren't too many restrictions as to how you build a topology as long as you avoid loops. You can have multiple controllers on an initiator (host), or multiple initiators (hosts), and you can chain them together in all sorts of creative and interesting ways - just always remembering that fragility increases as the number of parts increases.

A well-designed single 4U, 24 drive chassis might have a pair of 36 port SAS expanders, a pair of HBA's, and 24 dual ported SAS drives for complete redundancy in the paths from the host to the storage.

To answer the original question, you can stick as many drives on a SAS controller as the controller and expander(s) will allow, values for which you may have to check with the manufacturer's specs on.
 

Bigtexun

Dabbler
Joined
Jul 17, 2014
Messages
33
Three Hundred twenty four hard drives... but that assumes some minor case mods, using the back of the case for drives as well... and I'm talking about the laptop form factor drives... But no that is not a practical number, it is just what I could do if so inclined. The challenge would be to connect those drives to a computer, without going with a industrial backplane based computer.

The case is redonculous, obviously.

This case is a left over from what was originally a usenet news server I built in a previous lifetime. It's like a short side-by-side refrigerator. I bought it from 33comix.com many years ago. They are out of business now (or changed their name). The case cost $1500, and was meant for people building monsters. Since I can put 6 drives into each of the front 36 half-height 5.25" drive bays, the standard drive capacity is 216 without adding more drives in the back. The back has 6 panels that had a variety of pre-punched or blank configurations, which could become holes for more drive backplanes. Back panels with 2 power supply slots each were standard, as well as old style full size AT power supplies. And in the center are two vertical drawers for large full size (and smaller) motherboards. And fans all over. as well as several dozen LED's and switches. It is almost an antique, but it was just modern enough to support all of the current form factors... so I never threw it away. It's been recycled 4 times since it was first built.

And thank you both for the SAS expander info. That is what I was expecting to hear. The problem is the documentation makes assumptions as to what case you are using, so there is very little "general" info. So SAS expanders are just SAS switches... Well that SOLVES the problem of how you pack 324 drives onto a single computer... What could possibly go wrong with that? Sounds like a recipe for disaster.

Obviously performance suffers any time you use this kind of device... but for a home video server for two people that tend to only have one stream.

We are up to about 200 titles running out of plex, which I'm keeping well backed up because I don't know if I might decide to reformat and start over at some point... you learn so much as you build something like this, and the more complex it is, the more likely you will want to do it all again the right way. I'm still using my old network backup solution, but I'm getting ready to cut those over. I haven't decided yet how I will backup my library... redundant FreeNAS servers seems like overkill, and online storage is still subject to a malware-triggered massive data loss... So I'm leaning toward simple file by file backups to bare hard drives that are off-line and off-site 99.9% of the time...
 

diehard

Contributor
Joined
Mar 21, 2013
Messages
162
That sounds.. strange. It sounds like there is no way this enclosure wouldnt even have data or power connections on the backplane for any modern drive. How would you run cables? You might as well just put a fasten a bunch of drives to a rack and put a bunch of floor fans to blow over them!

There is no problem with the documentation making an assumption that you are using at least a somewhat modern case , when all other requirements are of modern hardware... or that you dont need to connect hundreds of drives.
 
Last edited:

Bigtexun

Dabbler
Joined
Jul 17, 2014
Messages
33
The case accepts power supplies of the current standard form factors, so being wired for power is not an issue, it will hold 8 modern 1000 watt power supplies, if that is what you want to put into it. It is a metal chassis that has the proper holes cut and drilled for form factors that are still in use today. The drive bays are simple rail type bays, with no back plane, and I'm finding several vendors with hot-swap bays designed to fit into the 5.25" chassis... It has no backplane, but there were backplanes you could buy for these at one point. had I done that, I would have had to buy a new one each time the old one went obsolete... I've owned this case across numerous scsi standards. Until SAS, scsi cables were all busses, so running custom cables was desirable in the olden days. With the serial protocols, everything is point to point, so multilane cables are desirable....

But I make my own cables where that is practical, and useful. Nothing fits better than a purpose made cable that is custom fit. But we don't do that with sata/sas...

There are some 3.5" drive bay options that can be bolted into 5.25" drive bays, but I'm using the 2.5" drives, and you can fit 6 of those into a5.25" slot, with a $75 backplane module like this:
View: http://www.amazon.com/gp/product/B005SIP20M


I have been seeing these in the $59 to $75 range. I have some thermatake branded ones, but they were discontinued, but this link is for an identical unit, just with different colors of plastic and a different brand name. The Thermatake was rated as 3meg sata2, but runs fine at 6meg. The one in the link is the same unit, but it now is rated for 6.

Or if you have the fatter 2.5" drives, you can put 4 per slot using bays like this:
View: http://www.ebay.com/itm/Aluminum-3-x5-25-to-12x2-5-SAS-SATA-6-0G-HDDs-Hot-Swap-Cage-Backplane-NEW-/400751175012?pt=US_Drive_Enclosures_Docks&hash=item5d4ea1a564


These are available in 4, 8 and 12 drive units. I bought 3 of the 12 drive units when they first went on sale, for $99 each. 5 minutes later the price went to $199, and has remained.

Each drive cage has integrated fans.

The Thermatake version of the 6 drive chassis I linked from Amazon has integrated drive LED's, so power and data required no wiring to get lights.... I assume the one in the link does too, but I won't know for sure until it arrives. (it arrived, and no wiring is required for the LED's). The reviews claim it is identical in every way except color.

The one I linked from Ebay requires you to run two wires for each traffic LED, if you want them, but power LED's are integral. All of these use individual sata data cables, so there is no multilane support in these... a disadvantage, but not the worst problem to have.

I prefer the 6 drive bay that I linked from Amazon, the only problem I have had with them is the Thermatake being discontinued. The one from ebay has more milled metal, and looks nicer because of that, but it doesn't /work/ better, and the drive led's will be a pain.

If I were not married to the 2.5" drives, I would probably do what many of you all did, and find a used 24 3.5" drive chassis.
 
Last edited:
Status
Not open for further replies.
Top