I'm starting a thread to talk about some technical details of SAS expanders, I was starting to discuss this on a SATA port multiplier thread, and while there was proper contect when I entered the thread, really this should be it's own thread. Port multipliers appear not to work any more under FreeBSD, and rather than belabor that issue, I think a fresh discussion of what hardware we should be using is called for.
So I have a handful of brand new LSI 9211-8i's on the way, and I was reading the installation guides for a couple Intel SAS expanders, one of which, the 24 port model, is recommended by a couple. There is also a 36 port version. I'm needing port density, so the 36 port card is the one I am interested in.
I'm told that the 24 port card is really 16 ports, and two of the 4-port SAS ports are reserved for connecting to the host controller card. But the documentation actually says that any port can be a drive or controller port, but only if you want the drive numbering to be sequential is this a requirement.
For the 36 port card, there seems to be some flexibility, and perhaps even downright ambiguity about how the ports are to be used. The documentation states:
"Each connector on the SAS expander card can be used as a “cable in” (SAS Controller to SAS Expander) or “cable out” (SAS Expander to Hot Swap Backplane) type connector. However, for contiguous drive mapping (0 – 16 or 0 – 24), cable routing differs when using a x8 wide-port capable 6 Gb SAS/SAS RAID Controller vs. using the embedded SCU ports."
So note the "0-16 or 0-24" reference. Later in the document they state that to be bootable, the drives must be directly connected to the controller, so without going into detail, they mention connecting one cable to the drive backplane, and the other to the expander.
The 36 port expander can have up to 24 contiguously numbered drives, and the remaining three ports only require one or two to be connected to the controller. So since all ports can be drives of controller ports, does this mean the unused ports can be connected to drives, just without contiguously numbered drives?
Keep in mind this document assumed in advance that you are using a specific chassis, which only has 24 drive slots, so the document doesn't address possibilities that might be encountered by someone with more slots. So all of the above may be simply to match the numbers in the drive bays, and in no way limits anything.
So has anyone tried either of these expanders with only one cable to the controller? Other than a reduction in throughput, does this work? Can I use all available ports for drives, so 32 drives on a 36 port board? Can I put a second expander on the second SAS connector, and get 64 drives on-line?
My computer case can hold 324 drives, and while I will never have that many drives in the box, I /do/ have more drives available than I am likely to ever be able to plug into a single FreeNAS system. So the more drive slots I can use, the more I /will/ use. In all of the NAS systems I'm trying to build, performance is a non-issue, this is zero-user bulk storage.
I'll be testing these questions myself, but I know other people have used these cards, so I'm curious how much expansion you can get on a 8-port LSI 9211-8i using these.
I just confirmed that someone else did in fact get a 24 port board working with only one controller cable... He also confirmed you cannot boot through the expander, and where you plug into the expander will influence the drive numbering, so if your first 4 drives are not through the expander so they are bootable, where you plug into the expander will cause the first drive to become drive 4. So perhaps the talk about how to hook the things up is mostly about getting the drive numbers right (I could care less about that). But that user did not try to use all ports, so the jury is still out on that question.
So I have a handful of brand new LSI 9211-8i's on the way, and I was reading the installation guides for a couple Intel SAS expanders, one of which, the 24 port model, is recommended by a couple. There is also a 36 port version. I'm needing port density, so the 36 port card is the one I am interested in.
I'm told that the 24 port card is really 16 ports, and two of the 4-port SAS ports are reserved for connecting to the host controller card. But the documentation actually says that any port can be a drive or controller port, but only if you want the drive numbering to be sequential is this a requirement.
For the 36 port card, there seems to be some flexibility, and perhaps even downright ambiguity about how the ports are to be used. The documentation states:
"Each connector on the SAS expander card can be used as a “cable in” (SAS Controller to SAS Expander) or “cable out” (SAS Expander to Hot Swap Backplane) type connector. However, for contiguous drive mapping (0 – 16 or 0 – 24), cable routing differs when using a x8 wide-port capable 6 Gb SAS/SAS RAID Controller vs. using the embedded SCU ports."
So note the "0-16 or 0-24" reference. Later in the document they state that to be bootable, the drives must be directly connected to the controller, so without going into detail, they mention connecting one cable to the drive backplane, and the other to the expander.
The 36 port expander can have up to 24 contiguously numbered drives, and the remaining three ports only require one or two to be connected to the controller. So since all ports can be drives of controller ports, does this mean the unused ports can be connected to drives, just without contiguously numbered drives?
Keep in mind this document assumed in advance that you are using a specific chassis, which only has 24 drive slots, so the document doesn't address possibilities that might be encountered by someone with more slots. So all of the above may be simply to match the numbers in the drive bays, and in no way limits anything.
So has anyone tried either of these expanders with only one cable to the controller? Other than a reduction in throughput, does this work? Can I use all available ports for drives, so 32 drives on a 36 port board? Can I put a second expander on the second SAS connector, and get 64 drives on-line?
My computer case can hold 324 drives, and while I will never have that many drives in the box, I /do/ have more drives available than I am likely to ever be able to plug into a single FreeNAS system. So the more drive slots I can use, the more I /will/ use. In all of the NAS systems I'm trying to build, performance is a non-issue, this is zero-user bulk storage.
I'll be testing these questions myself, but I know other people have used these cards, so I'm curious how much expansion you can get on a 8-port LSI 9211-8i using these.
I just confirmed that someone else did in fact get a 24 port board working with only one controller cable... He also confirmed you cannot boot through the expander, and where you plug into the expander will influence the drive numbering, so if your first 4 drives are not through the expander so they are bootable, where you plug into the expander will cause the first drive to become drive 4. So perhaps the talk about how to hook the things up is mostly about getting the drive numbers right (I could care less about that). But that user did not try to use all ports, so the jury is still out on that question.
