LSI 9201-16e - Am I doing it wrong?

[Mirfster]

So, been experimenting with expanding using JBODs. Was working fine when I had two separate LSI 9200-8e cards. I then decided why not try to condense this to one LSI 9201-16e?

For the record all Servers are the same, Dell C2100/FS12-TY and they each have a SAS2 Expander Backplane.

Server 1
Currently is running ESXi 6.0 U2 and FreeNAS as a VM with the cards in "Pass-Through". Issue is not with ESXi as I can see that all the drives are not being detected in the SAS BIOS. If the whole "Virtualization is not supported" excuse is tossed my way, then I don't buy it but feel free to use it if you want... :P

This Server has three (3) HBAs (Dell Mezzanine Perc H200 - Flashed to LSI 9211-8i, LSI 9200-8e and LSI 9201-16e). While the Perc H200 is at P20.04, the other two are at P20.07.

Server 2 & 3
JBODs have the SAS2 Expander Backplanes connected to a Mini-SAS to Mini-SAS (SFF-8087 to SFF-8087) and then are connected to the external HBAs on Server 1 via SFF-8088 to SFF-8088 cables (think 1m or 1.5m..).

So when I connect all the JBODs to just the LSI 9201-16e, it fails to see all the disks in my JBODs. However, if I split up the JBODs to the separate HBAs then things are fine...

1. Is there something I am doing wrong or is this a limitation of some sort I am hitting and not understanding?
   
2. Do I need to check into "Active" vs "Passive" SFF-8088 cables (just saw some brief info but did not delve into it much)?
3. Could the issue be that one JBOD has all SAS Drives while the other has SATA Drives?
   • If so then I can live with that, but am looking to perhaps attach a 3rd all SATA JBOD and would wonder if just those two would be fine on one HBA then?
4. While I have two SFF-8088 connections going to each JBOD, I would think that I can get away with just one or perhaps even look into "Daisy Chaining" or "Cascading" (not too sure on this though, just want to get this part figured out)
   • Would however love to eventually connect all to a single HBA if that is feasible...

Sorry for the bombardment of questions, just a bit puzzled... :)

 

[depasseg]

I'm a little confused. Could you provide a diagram? It sounds like you've tested a 9201-16e in both/either server 2 &/or 3 with multiple JBOD's, but then in #4 you talk about have 2 connections to each jbod. Have you tried with just 1 connection to 1 jbod?

What model jbods are you using?

This should work, and daisy chaining should work. I think part of the issue might be the way they are cabled, hence the request for a diagram.

 

[Ericloewe]

While I have two SFF-8088 connections going to each JBOD

@jgreco says that such configurations are often problematic. I always found that to be weird, since it's a common configuration in Supermicro's manuals (for instance) and is a natural extension of what SAS already does every day (run four connections to the same expander).

 

[Mlovelace]

The only reason for two connections going to the backplane would be for SAS multichannel. You can daisy chain from jbod to jbod to NAS and for sata disks you only need one connection to the backplane.

 

[Ericloewe]

The only reason for two connections going to the backplane would be for SAS multichannel. You can daisy chain from jbod to jbod to NAS and for sata disks you only need one connection to the backplane.

No, using two (or rather, eight channels) is supposed to work, doubling upstream bandwidth.

 

[CookiesLikeWhoa]

No, using two (or rather, eight channels) is supposed to work, doubling upstream bandwidth.

I was looking that up and from what I read Supermicro does not support that. I don't know about other backplane manufactures, but it looked like Supermicro has no backplanes that support dual link. They support failover, cascading and multipathing but not dual link (doubling the bandwidth).

 

[Mlovelace]

No, using two (or rather, eight channels) is supposed to work, doubling upstream bandwidth.

You can test with eight channel connection to the jbod to see if four channel is bottlenecking, but as I said you only need a four channel connection. With single channel sata NAS drives I doubt you'd saturate a four channel connection.

 

[Ericloewe]

but as I said you only need a four channel connection. With single channel sata NAS drives I doubt you'd saturate a four channel connection.

That's true enough.

 

[Mirfster]

I'm a little confused. Could you provide a diagram?

Yeah, I am confused myself. :) Will toss together a pic a little later on when I get out from under some last minute work that popped up.

It sounds like you've tested a 9201-16e in both/either server 2 &/or 3 with multiple JBOD's, but then in #4 you talk about have 2 connections to each jbod. Have you tried with just 1 connection to 1 jbod?

Tried so many different scenarios I can't recall truly, but will for sure do a couple more and post the tests and results.

What model jbods are you using?

They are Dell C2100s/FS12-TYs (acting as a JBOD). ;)

I think part of the issue might be the way they are cabled

More than likely the way I am cabling them.

 

[Mirfster]

Just a quickly tossed together diagram of what I am trying to accomplish:

Currently trying to use a single SFF-8088 cable for each connection. On JBOD #1, the connection to the SSDs is simply a Mini-SAS Breakout internally. While the actual connections to the drives are a single SFF-8087 to SFF-8087 on the backplane.

Still no love as of yet with just trying to leverage a single LSI 9201-16e...

 

[Ericloewe]

On JBOD #1, the connection to the SSDs is simply a Mini-SAS Breakout internally.

Oh, you can't do that with SATA drives. SATA is limited to a maximum cable length of 1m and it's rather sensitive.

SAS devices use higher voltage signalling, so you can either swap out the SSDs or add an SAS expander.

 

[Mirfster]

Oh, you can't do that with SATA drives.

Okay, I will disconnect the SSDs on JBOD#1 for now and give it a try. If it works, then I will move them to the actual FreeNAS box itself. Will try it in a bit and post back.