LSI SAS HBA topology mapping

[PeteKowalsky]

Hey Rand I know this was an old post but it came in quite handy for me trying to map out my SAS topology from the CLI. Looks like FreeNAS 11.2 already has the right kernel module and the sas2ircu tool is already here for me. :)

I don't suppose you know of any way to check to see how many backplane channels are negotiated with the HBA huh? Maybe that's crazy talk. I'm a network guy and this is my first FreeNAS build so I want to make sure I'm not doing anything stupid like over-subscribing my controller with too many drives or anything. 12 drives x 200MB/sec = 19.2Gbps, so clearly a single SAS2 6Gbps lane to an expander is not going to cut it. As I understand it, a SAS2 SFF8087 to 8087 carries 4x 6Gbps channels, so @ 24Gbps is enough. Just wanted to verify they're all "in use" before I start putting anything I don't wanna lose on here, LOL.

 

[jgreco]

I don't suppose you know of any way to check to see how many backplane channels are negotiated with the HBA huh? Maybe that's crazy talk. I'm a network guy and this is my first FreeNAS build so I want to make sure I'm not doing anything stupid like over-subscribing my controller with too many drives or anything. 12 drives x 200MB/sec = 19.2Gbps, so clearly a single SAS2 6Gbps lane to an expander is not going to cut it. As I understand it, a SAS2 SFF8087 to 8087 carries 4x 6Gbps channels, so @ 24Gbps is enough. Just wanted to verify they're all "in use" before I start putting anything I don't wanna lose on here, LOL.

While SAS theoretically supports use of a single lane to connect a downstream SAS expander, I don't think the real world implementations generally support it. They expect multilane connectivity, or more specifically wideport x4 or x8. I generally recommend the x4 since I've seen strange issues with x8. The HBA and expander automatically negotiate this. No need to worry about it. This also doesn't affect SAS connectivity, so even changing from a single lane to an x4 wideport to an x8 wideport has no effect on the data you've stored - just the speed at which it is stored.

Since you're a network guy, think of multilane or wideporting as LACP for SAS, except that unlike LACP it's non-idiotic and can automatically configure itself properly.

12 drives x 200MB/sec = 19.2Gbps in the case where you attach some rocket boosters to your array and launch it. In practice, you're not going to see that except in extremely unusual cases, because it assumes an entirely sequential workload. It would also require a filer that had networking in excess of 10Gbps. ZFS is a Copy-on-Write filesystem so there's generally seeks that break up disk I/O. You can safely toss 24 HDD on an x4 wideport. In the worst case, where you store large sequential files on the pool, and you never rewrite any data on the pool, you can get close-to-sequential disk access behaviours -- this leaves you in a situation where you can hit contention when doing a scrub, or if you have 40/100GbE ethernet connectivity, but all that's really going to happen is things go a bit slower.

 

[jgreco]

Moderator note: Poster revived a 5+ year old thread for a somewhat similar yet different topic. I have separated this into a new thread and edited the poster's message to contain a link back to the older thread.

 

[PeteKowalsky]

Thank you jgreco, I appreciate the thoughts and for moving this to a new thread.

LOL @ rocket boosters...but yes, I am in fact planning to be 2 x 40GbE connected to my core switch. I might rethink using lagg0 (2x40GbE) on my FreeNAS host since there's likely no way I'll even be close to 40Gb tput. I too cannot imagine a real life scenario where all drives would be reading/writing at full speed at the exact same time but thought it prudent to ask if I was setting myself up for trouble by using SAS2 and not SAS3 or having all drives on one HBA / expander link.

So, it's more common for expanders to use multiple SAS "channels" on a SAS2/SFF8087 connection - and that is the 4x wideport you mentioned. Do I have that right?

 

[Ericloewe]

When you're talking 40 Gb/s, I worry less about "do my disks have enough bandwidth" and more about "can literally everything else in my server and the rest of the network keep up".

So, it's more common for expanders to use multiple SAS "channels" on a SAS2/SFF8087 connection - and that is the 4x wideport you mentioned. Do I have that right?

Universal, really. Individual SAS connections are fairly rare, almost everything uses SFF-8087 or SFF-8643 (or their external equivalents) to a backplane.

While SAS theoretically supports use of a single lane to connect a downstream SAS expander, I don't think the real world implementations generally support it.

Anyone with a Supermicro X10SL7-F and an SAS expander, feel free try connecting the expander with just one of the four host connectors on a reverse-breakout cable and tell us how that goes.