IBM diskshelf and Internal to external sas adapter?

otpi

Contributor
Joined
Feb 23, 2017
Messages
117
I'd like some HW expertise before I purchase an adapter:

I got two diskshelfs (24x2.5in): IBM DS3524 and EXP3524. And I'm trying to put them to good use.

The DS has redundant controllers, which are 2011 era HW and I'd rather not power those 24/7. AFAIK the EXP3524 is just a sas-expander-thingy? Which talks to its 24 hdds through a midplane. Since I have two (redundant) expander thingies, and the sas-expander-thingy is interchangeable between the DS-"controller" and EXP-"expansion" unit I could use both shelves as EXPanders, but connected directly to my TrueNAS server through external sas?

My server (Lenovo x3650) has onboard HBA and two internal sas-8087 connectors. AFAIK: N2215 HBA, 12 Gb SAS/SATA non-RAID

If I purchase a "Dual Mini SAS SFF-8088 to SAS36P SFF-8087 Adapter PCI Bracket". Will it work? Could I daisy chain the EXP units "indefinitely" using a single cable?
 

klu

Dabbler
Joined
Nov 19, 2021
Messages
16
As far as I can see, it should be possible - in theorie. Since I am trying to get something similar to run right now, have look over here. I know it is not the same setup because I am running two NETAPP DS4246, but in theorie they also can be chained indefinatly by one/two connections.

I tested my setup today with an SAS 9300-8e, which has the same sas3008 chip as your N2215 HBA, it might prevent you some headache. I cannot get more than 24 disk from two shelf running at the same time, on any SAS3 controller I tried so far. So if something like this happens, It could be a simiilar problem.

Anyway, I hope your setup will work just fine. If it is not too much trouble, please give me a PM or post if you get it to work.
 

otpi

Contributor
Joined
Feb 23, 2017
Messages
117
After double checking i found that what I first believed to be the "onboard" sas is actually just 8x6G sata (2xsff-8087). And, I have an M5210 raid controller (LSI SAS3108), not the N2215 HBA (LSI SAS3008), which is installed horizontally in PCIE. No cache on mine (controllers-for-internal-storage).

Anyway. All is not lost. The M5210, while advertised as raid, behave and present the disks properly as a HBA. Supports up to 240 disks, so should be good for my needs.

I'll order som sas cables and see if it works.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
If I purchase a "Dual Mini SAS SFF-8088 to SAS36P SFF-8087 Adapter PCI Bracket". Will it work? Could I daisy chain the EXP units "indefinitely" using a single cable?

Yes. Please feel free to head on over to the SAS Primer.


The M5210, while advertised as raid, behave and present the disks properly as a HBA.

It does not. It is still a RAID controller. If it is showing under the MFI driver, it is absolutely unsafe to use. If it is coming up under MRSAS (my expectation), it is merely a very bad idea -- the following article will explain this.


Note that I am the author of that article and I am *telling* you that your card is a RAID card for the purposes described in that article.

We absolutely have had people with MFI-attached pools with irreversible damage (pool lost, data irretrievable). There haven't been such clear-cut examples with MRSAS, and the MRSAS stuff is generally better than MFI, but it really comes down to the same sorts of issues, and whether or not you'd prefer to go with a true HBA (replacing the 3108) or be a guinea pig for MRSAS reliability testing.

It's worth noting that I really do like the 3108 controller and use them heavily for actual RAID controllers under ESXi etc.
 

otpi

Contributor
Joined
Feb 23, 2017
Messages
117
Oh... bummer. Thanks for the info Mr. Grinch. I remember reading your sas'sy primer a while back, but reading and understanding... well it's different. My naive interpretation is that a sas-cable is like 4 budled network cables, HBAs are like routers, and an expander is a "switch". I'm still using FreeNAS (sig) for my daily needs. Testing FreeNAS Scale on this new one. Will be "moving" once I feel every kink is ironed out.

How do I figure out if Scale is using MFI or MRSAS? Are there any stresstest I can run to see if the raid controller chokes?

I ordered a pass-through 8087/8088 bracket, two 8643-8087 and two 8087-sata breakout cables. Maybe I'll also source a proper HBA.

Code:
root@truenas[~]# lspci | grep RAID
0b:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS-3 3108 [Invader] (rev 02)

root@truenas[~]# dmesg | grep mega
[   10.212320] megasas: 07.714.04.00-rc1
[   10.259676] megaraid_sas 0000:0b:00.0: BAR:0x1  BAR's base_addr(phys):0x00000000913f0000  mapped virt_addr:0x00000000cd6f3449
[   10.275674] megaraid_sas 0000:0b:00.0: FW now in Ready state
[   10.290956] megaraid_sas 0000:0b:00.0: 63 bit DMA mask and 32 bit consistent mask
[   10.310558] megaraid_sas 0000:0b:00.0: firmware supports msix        : (96)
[   10.327198] megaraid_sas 0000:0b:00.0: requested/available msix 33/33
[   10.343159] megaraid_sas 0000:0b:00.0: current msix/online cpus      : (33/32)
[   10.359480] megaraid_sas 0000:0b:00.0: RDPQ mode     : (disabled)
[   10.374561] megaraid_sas 0000:0b:00.0: Current firmware supports maximum commands: 272        LDIO threshold: 237
[   10.394720] megaraid_sas 0000:0b:00.0: Configured max firmware commands: 271
[   10.411978] megaraid_sas 0000:0b:00.0: Performance mode :Latency
[   10.427303] megaraid_sas 0000:0b:00.0: FW supports sync cache        : Yes
[   10.442741] megaraid_sas 0000:0b:00.0: megasas_disable_intr_fusion is called outbound_intr_mask:0x40000009
[   10.597221] megaraid_sas 0000:0b:00.0: FW provided supportMaxExtLDs: 0       max_lds: 32
[   10.613733] megaraid_sas 0000:0b:00.0: controller type       : iMR(0MB)
[   10.628521] megaraid_sas 0000:0b:00.0: Online Controller Reset(OCR)  : Enabled
[   10.644325] megaraid_sas 0000:0b:00.0: Secure JBOD support   : Yes
[   10.658673] megaraid_sas 0000:0b:00.0: NVMe passthru support : No
[   10.673083] megaraid_sas 0000:0b:00.0: FW provided TM TaskAbort/Reset timeout        : 0 secs/0 secs
[   10.690348] megaraid_sas 0000:0b:00.0: JBOD sequence map support     : Yes
[   10.705267] megaraid_sas 0000:0b:00.0: PCI Lane Margining support    : No
[   11.272502] megaraid_sas 0000:0b:00.0: megasas_enable_intr_fusion is called outbound_intr_mask:0x40000000
[   11.291112] megaraid_sas 0000:0b:00.0: INIT adapter done
[   11.364599] megaraid_sas 0000:0b:00.0: pci id                : (0x1000)/(0x005d)/(0x1014)/(0x0454)
[   11.381174] megaraid_sas 0000:0b:00.0: unevenspan support    : no
[   11.395534] megaraid_sas 0000:0b:00.0: firmware crash dump   : no
[   11.409936] megaraid_sas 0000:0b:00.0: JBOD sequence map     : enabled
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
My naive interpretation is that a sas-cable is like 4 budled network cables, HBAs are like routers, and an expander is a "switch".

That's a healthy-enough interpretation. The switch doesn't have any sort of spanning tree protocol, so don't make any topology loops.

My guess is that what you're seeing there is MRSAS. I don't have a Linux box and card handy to try it on. It's plausible that it could work fine, but my preference would be to swap in an HBA.
 
Top