- Joined
- Apr 16, 2020
- Messages
- 2,943
My build (in the sig below, ignore the QNAS unit) has an LSI9211-8i, plugged into a PCIe3*8 slot connected to an IBM 46M0997 SAS Expander which only uses PCIe for power. Motherboard is a Supermicro X10SRI-F
The case is 24 Slots (18 used), 16 of which connect to the SAS Expander, and 8 connect to the motherboard SATA connectors - done in a logical manner so I know what is what
I have a bunch of HDD's - not important for this (too slow to matter), and a bunch of SSD's. The SSD's have been plugged in some on SATA and some on SAS (although all are SATA)
The main pool is 6 SSD's (mirrored pairs) of which 3 are on SATA 3 Ports and 3 on SAS Ports (they are mirrored 1 SAS, 1 SATA in case the SAS infrastructure goes down - I still have a pool. (I have done the same with the 3 mirrored vdev HDD Pool)
I use 10Gb networking * 2 (1 for storage, 1 for LAN)
The 9211 supports up to 6Gbps SAS on all 8 lanes. One cable can carry 6Gbps. The IBM is also SAS-2 (6Gbps)
The 9211 is PCIe2 * 8 Lanes = 4GB/s = (in round numbers) 32Gb/s
There are two connectors between the SAS HBA and the Expander = 12Gb/s, or more probably 2*6 Gb/s as I have no idea if the lanes will add together across different cables. But this is worst case.
1 SSD is, say, 500Mb/s – so at most, and worst case I have 3*500=1,500Mb/s down one cable (up to 3 SSD’s) between the Expander and HBA. The link between the HBA and the rest of the system is high enough that I don’t care.
1,500Mb/s = 1.5Gb/s < 6Gb/s (of the SAS Cables). So I have no bottlenecks here – other than the speed of the drives overall. Even all 6 SSD’s on SAS ports would not be a problem here
Therefore there is no purpose in me upgrading to a more modern LSI board. An upgrade to a -16i would enable the removal of the IBM card but at significant cost. The current setup will go considerably faster assuming I have the slots / finance / disks to do so.
Have I got this right?
Note 1 : I cannot use an additional HBA card due to lack of useful PCIe slots (2 of 4 useful used by Optanes, 1 by dual 10Gb NIC and 1 by HBA)
Note 2 : I could use a single cable to the IBM and breakout the other “input” to 4 drive slots but there is no point
The case is 24 Slots (18 used), 16 of which connect to the SAS Expander, and 8 connect to the motherboard SATA connectors - done in a logical manner so I know what is what
I have a bunch of HDD's - not important for this (too slow to matter), and a bunch of SSD's. The SSD's have been plugged in some on SATA and some on SAS (although all are SATA)
The main pool is 6 SSD's (mirrored pairs) of which 3 are on SATA 3 Ports and 3 on SAS Ports (they are mirrored 1 SAS, 1 SATA in case the SAS infrastructure goes down - I still have a pool. (I have done the same with the 3 mirrored vdev HDD Pool)
I use 10Gb networking * 2 (1 for storage, 1 for LAN)
The 9211 supports up to 6Gbps SAS on all 8 lanes. One cable can carry 6Gbps. The IBM is also SAS-2 (6Gbps)
The 9211 is PCIe2 * 8 Lanes = 4GB/s = (in round numbers) 32Gb/s
There are two connectors between the SAS HBA and the Expander = 12Gb/s, or more probably 2*6 Gb/s as I have no idea if the lanes will add together across different cables. But this is worst case.
1 SSD is, say, 500Mb/s – so at most, and worst case I have 3*500=1,500Mb/s down one cable (up to 3 SSD’s) between the Expander and HBA. The link between the HBA and the rest of the system is high enough that I don’t care.
1,500Mb/s = 1.5Gb/s < 6Gb/s (of the SAS Cables). So I have no bottlenecks here – other than the speed of the drives overall. Even all 6 SSD’s on SAS ports would not be a problem here
Therefore there is no purpose in me upgrading to a more modern LSI board. An upgrade to a -16i would enable the removal of the IBM card but at significant cost. The current setup will go considerably faster assuming I have the slots / finance / disks to do so.
Have I got this right?
Note 1 : I cannot use an additional HBA card due to lack of useful PCIe slots (2 of 4 useful used by Optanes, 1 by dual 10Gb NIC and 1 by HBA)
Note 2 : I could use a single cable to the IBM and breakout the other “input” to 4 drive slots but there is no point