Can someone check my conclusion here please

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,943
My build (in the sig below, ignore the QNAS unit) has an LSI9211-8i, plugged into a PCIe3*8 slot connected to an IBM 46M0997 SAS Expander which only uses PCIe for power. Motherboard is a Supermicro X10SRI-F

The case is 24 Slots (18 used), 16 of which connect to the SAS Expander, and 8 connect to the motherboard SATA connectors - done in a logical manner so I know what is what

I have a bunch of HDD's - not important for this (too slow to matter), and a bunch of SSD's. The SSD's have been plugged in some on SATA and some on SAS (although all are SATA)

The main pool is 6 SSD's (mirrored pairs) of which 3 are on SATA 3 Ports and 3 on SAS Ports (they are mirrored 1 SAS, 1 SATA in case the SAS infrastructure goes down - I still have a pool. (I have done the same with the 3 mirrored vdev HDD Pool)

I use 10Gb networking * 2 (1 for storage, 1 for LAN)

The 9211 supports up to 6Gbps SAS on all 8 lanes. One cable can carry 6Gbps. The IBM is also SAS-2 (6Gbps)
The 9211 is PCIe2 * 8 Lanes = 4GB/s = (in round numbers) 32Gb/s

There are two connectors between the SAS HBA and the Expander = 12Gb/s, or more probably 2*6 Gb/s as I have no idea if the lanes will add together across different cables. But this is worst case.

1 SSD is, say, 500Mb/s – so at most, and worst case I have 3*500=1,500Mb/s down one cable (up to 3 SSD’s) between the Expander and HBA. The link between the HBA and the rest of the system is high enough that I don’t care.

1,500Mb/s = 1.5Gb/s < 6Gb/s (of the SAS Cables). So I have no bottlenecks here – other than the speed of the drives overall. Even all 6 SSD’s on SAS ports would not be a problem here

Therefore there is no purpose in me upgrading to a more modern LSI board. An upgrade to a -16i would enable the removal of the IBM card but at significant cost. The current setup will go considerably faster assuming I have the slots / finance / disks to do so.

Have I got this right?

Note 1 : I cannot use an additional HBA card due to lack of useful PCIe slots (2 of 4 useful used by Optanes, 1 by dual 10Gb NIC and 1 by HBA)
Note 2 : I could use a single cable to the IBM and breakout the other “input” to 4 drive slots but there is no point
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
You might see benefit by upgrading to a SAS2308-based HBA (SAS9207/SAS9217 or equivalent). These are PCIe 3.0 vs the SAS2008's PCIe 2.0; so they deliver a little more bandwidth. And they support multiple MSI-X interrupts, which ought to help with your SSDs. Some good details here:

 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,943
I have found a 9305-16i which means I could tidy the cabling up a bit and get rid of the IBM - costs a bit though
Are these (in IT mode) OK in TrueNAS?
Of course it would be simpler to substitute a 2308 which I think uses the same cabling as what I currently use.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
I have found a 9305-16i which means I could tidy the cabling up a bit and get rid of the IBM - costs a bit though
Are these (in IT mode) OK in TrueNAS?
Of course it would be simpler to substitute a 2308 which I think uses the same cabling as what I currently use.
Those 9305's cost the Earth!

Yes, they can be flashed to IT mode. But they use a different connector than your 9211: the card has MiniSAS HD SFF-8643 ports. Don't know what type of connector you need for your expander, but you can probably find a cable to make it work.

Here's a link to the specs:
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,943
Yeah - not entirely cheap - but wondering if its a worthwhile long term purchase. Might be able to sell some of my other cards to cover the cost of the bracket for it. :cool:

Decisions decisions.

I know the cables will be different - which is nuisance but the cost of cables is small in comparison to the cost of the card.
 
Top