LSI9400-16i not all drives on 6gb

[swissroot]

to the storage gurus

I've searched through the forum and also other ones including all the guides from Broadcom, but was not finding an answer that explains me the behaviour which happens in my fresh TrueNAS.

So short explain about my hardware:
- MB: X11SCZ-F
- CPU: left over i3-8300 until XEONS are available again
- RAM: 64GB ECC
- 10GB AOC-STGN-I2S
- HBA: LSI9400-16i (sitting in x16 slot, SAS/SATA firmware p16 from 08/24/2020)
- Case: Supermicro 743TQ-865B-SQ
- additional 2.5" cage CSE-M28SACB
- boot supermicro sata dom
- onboard nvme samsung 860 evo (not used, still in there from esx)
- TrueNAS 12.0 RELEASE

disks:
- main cage case
- 6x seagate exos x16 16tb sata
- 2x ssd samsung evo/wd blue 512gb (leftovers, just for testings)

- addon cage
- 4x wdblue ssd 256gb moved from syno for esx storage

The SSDs actually used are leftovers and not datacenter hw so I had long time ago the plan to build a freenas/truenas but was to lacy to swap. If everything works I will grab something from a DC move or from ebay to replace the consumer ssd with dc ready stuff. But actually the problem is way far before I will plan to buy other hw.

I've setup everything the backplanes are both cabled with SFF-8643 to SFF-8643 or SFF-8643 to 4x sata for the main backplane in the case. Cables are supermicro or broadcom no no-name ebay chinese buys. I tried to get SGPIO working but with less success, during this I checked with storcli some parameters of the 9400 HBA. When I list all disks (storcli /c0 show all) I see that not all are running on 6gb sata. Half of them are on 3gb. I checked with the cabling from the controller and it's reproducible when I swap the cables on the controller. So due to each have 8 slots I use always a pair for connecting the cages.

When I move the connections from c0/1 to c2/3 then this drives will get 6gb and vice versa. I not mixed it up so to keep the cages together on the same SAS core.

Printscreens are from the Broadcom manual (https://docs.broadcom.com/docs/pub-005851)


Output from storcli on the truenas server:

Show : storcli: controller show

root@truenas[~]# storcli /c0 show
CLI Version = 007.1207.0000.0000 Sep 25, 2019
Operating system = FreeBSD 12.2-RC3
Controller = 0
Status = Success
Description = None
Product Name = HBA 9400-16i
Serial Number = SP84203272
SAS Address = 500605b00e84ece0
PCI Address = 00:01:00:00
System Time = 11/01/2020 13:00:20
FW Package Build = 16.00.00.00
FW Version = 16.00.00.00
BIOS Version = 09.31.00.00_16.00.00.00
NVDATA Version = 15.12.00.11
PSOC Version = 05689001
Driver Name = mpr
Driver Version = 23.00.00.00
Bus Number = 1
Device Number = 0
Function Number = 0
Vendor Id = 0x1000
Device Id = 0xAC
SubVendor Id = 0x1000
SubDevice Id = 0x3020
Board Name = HBA 9400-16i
Board Assembly = 03-50008-15005
Board Tracer Number = SP84203272
Physical Drives = 12
PD LIST :
=======
----------------------------------------------------------------------------------
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
----------------------------------------------------------------------------------
0:0 2 JBOD - 232.885 GB SATA SSD N N 512B WDC WDS250G2B0A-00SM50 U
0:3 5 JBOD - 232.885 GB SATA SSD N N 512B WDC WDS250G2B0A-00SM50 U
0:4 4 JBOD - 232.885 GB SATA SSD N N 512B WDC WDS250G2B0A-00SM50 U
0:7 6 JBOD - 232.885 GB SATA SSD N N 512B WDC WDS250G2B0A-00SM50 U
0:8 11 JBOD - 14.551 TB SATA HDD N N 4 KB ST16000NM001G-2KK103 U
0:9 1 JBOD - 465.761 GB SATA SSD N N 512B Samsung SSD 860 EVO 500GB U
0:10 3 JBOD - 465.761 GB SATA SSD N N 512B WDC WDS500G2B0A-00SM50 U
0:11 12 JBOD - 14.551 TB SATA HDD N N 4 KB ST16000NM001G-2KK103 U
0:12 7 JBOD - 14.551 TB SATA HDD N N 4 KB ST16000NM001G-2KK103 U
0:13 8 JBOD - 14.551 TB SATA HDD N N 4 KB ST16000NM001G-2KK103 U
0:14 10 JBOD - 14.551 TB SATA HDD N N 4 KB ST16000NM001G-2KK103 U
0:15 9 JBOD - 14.551 TB SATA HDD N N 4 KB ST16000NM001G-2KK103 U
----------------------------------------------------------------------------------

Show : storcli: disk speed output

root@truenas[~]# storcli /c0 show all | grep "Drive\|Link Speed"
Driver Name = mpr
Driver Version = 23.00.00.00
Support Set Link Speed = No
Support Emulated Drives = No
Drive Coercion Mode = none
Max Drives to Spinup at One Time = 0
Un-Certified Hard Disk Drives = Block
LED Show Drive Activity = No
Dirty LED Shows Drive Activity = No
Supported Drives = SAS, SATA
Drive /c0/e0/s0 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s0 - Detailed Information :
Drive /c0/e0/s0 State :
Drive /c0/e0/s0 Device attributes :
Link Speed = 3.0Gb/s
Drive /c0/e0/s0 Policies/Settings :
Drive /c0/e0/s3 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s3 - Detailed Information :
Drive /c0/e0/s3 State :
Drive /c0/e0/s3 Device attributes :
Link Speed = 3.0Gb/s
Drive /c0/e0/s3 Policies/Settings :
Drive /c0/e0/s4 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s4 - Detailed Information :
Drive /c0/e0/s4 State :
Drive /c0/e0/s4 Device attributes :
Link Speed = 3.0Gb/s
Drive /c0/e0/s4 Policies/Settings :
Drive /c0/e0/s7 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s7 - Detailed Information :
Drive /c0/e0/s7 State :
Drive /c0/e0/s7 Device attributes :
Link Speed = 3.0Gb/s
Drive /c0/e0/s7 Policies/Settings :
Drive /c0/e0/s8 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s8 - Detailed Information :
Drive /c0/e0/s8 State :
Drive /c0/e0/s8 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s8 Policies/Settings :
Drive /c0/e0/s9 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s9 - Detailed Information :
Drive /c0/e0/s9 State :
Drive /c0/e0/s9 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s9 Policies/Settings :
Drive /c0/e0/s10 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s10 - Detailed Information :
Drive /c0/e0/s10 State :
Drive /c0/e0/s10 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s10 Policies/Settings :
Drive /c0/e0/s11 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s11 - Detailed Information :
Drive /c0/e0/s11 State :
Drive /c0/e0/s11 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s11 Policies/Settings :
Drive /c0/e0/s12 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s12 - Detailed Information :
Drive /c0/e0/s12 State :
Drive /c0/e0/s12 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s12 Policies/Settings :
Drive /c0/e0/s13 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s13 - Detailed Information :
Drive /c0/e0/s13 State :
Drive /c0/e0/s13 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s13 Policies/Settings :
Drive /c0/e0/s14 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s14 - Detailed Information :
Drive /c0/e0/s14 State :
Drive /c0/e0/s14 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s14 Policies/Settings :
Drive /c0/e0/s15 :
EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup
Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info
Drive /c0/e0/s15 - Detailed Information :
Drive /c0/e0/s15 State :
Drive /c0/e0/s15 Device attributes :
Link Speed = 6.0Gb/s
Drive /c0/e0/s15 Policies/Settings :

Is this by design when using a 16i adapter fully patched that they will step down to 3gb for sata? In the spec sheet they tell it can handle 1024 drives with no NVME connected, so I wonder what speed they get when I already get a step down with only 12 drives in use.

May someone can point me in the right direction that may it's a hardware fault or just a fault of my mind...;-) so I not be ultradeep in all this block sizes and this stuff but I know how to design HA storage solutions for ESX in datacenters (ok we buy full boxes and not building it from scratch as for home labs... ). But this thing is pulling me my hairs off since last night . I just wanted a nice storage box which hosts my fileshare for home (6x16tb) and 2x4 ssd for two esx storges to replace ma plain old synos.

I started with an LSI 9207-8i to just see how it performs in an old pc case with the same mainboard and some ssds and four old wdred 4tb. In this I had the 8i fully loaded and also some of the disks on the onboard sata. Always the volumes together not stripped over the controllers. And there I didn't saw such behaviour.

This was the only reason for buying the 9400-16i that I can handle all disks on one controller because of missing more pcie slots on the mainboard. There is actually just one 16x, the other 8x are just 4x in 8x slots.

If I missed some information which is needed to sort me out I can provide it.

Thanks in advance for any help to safe my hairs and nights!

 

[sretalla]

What's the temperature on the board? maybe the core that backs off is too hot?

Also, what is dmesg | grep transfers showing you?

 

[swissroot]

What's the temperature on the board? maybe the core that backs off is too hot?

It sits around 42 until 46, also checked the warning light of the controller which should stay on orange if a temperature issue is there and this is not lightning. The behaviour is also there when I shutdown the machine wait for cooling and right after fresh boot. Then the temp is around 35 going up to the mentioned ones.

 

[sretalla]

OK, you replied before I updated my post with :
Also, what is dmesg | grep transfers showing you?

 

[swissroot]

was right testing something and just saw the reply, else I will not be so fast...;-) thx for your reply!

just filled the second cage with some ssd's that's why there are more disks now.

but unfortunately also this output shows clear 300 instead of 600 :-(

Code:

root@truenas[~]# dmesg | grep transfers
ses0: 300.000MB/s transfers
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
da5: 600.000MB/s transfers
da0: 300.000MB/s transfers
da1: 300.000MB/s transfers
da2: 300.000MB/s transfers
da3: 300.000MB/s transfers
da6: 600.000MB/s transfers
da7: 600.000MB/s transfers
da4: 600.000MB/s transfers
da8: 600.000MB/s transfers
da11: 600.000MB/s transfers
da9: 600.000MB/s transfers
da10: 600.000MB/s transfers
da12: 300.000MB/s transfers
da13: 300.000MB/s transfers
da14: 300.000MB/s transfers
da15: 300.000MB/s transfers

 

[sretalla]

ses0: 300.000MB/s transfers

So that's your controller ses0 with SATAII speeds... strange it hasn't also shown ses1...

Perhaps try dmesg | grep ses

 

[swissroot]

so I tried with some SAS drives in both enclosures just to be sure it's not a SATA related issue. But with the same effect unfortunately...

Show : dmesg | grep ses

ses0 at mpr0 bus 0 scbus0 target 16 lun 0
ses0: <BROADCOM VirtualSES 03> Fixed Enclosure Services SPC-5 SCSI device
ses0: Serial Number 300605B00E11ECE0
ses0: 300.000MB/s transfers
ses0: SES Device
ses1 at ahciem0 bus 0 scbus7 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses1: ada0 in 'Slot 00', SATA Slot: scbus1 target 0
ses0: (none) in Array Device Slot 1, SAS Slot: 1 phys at slot 13
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece0 addr 5000cca23bb357c1
ses0: (none) in Array Device Slot 2, SAS Slot: 1 phys at slot 12
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece1 addr 5000cca2668999ad
ses0: (none) in Array Device Slot 3, SAS Slot: 1 phys at slot 14
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece2 addr 5000cca2667a3359
ses0: (none) in Array Device Slot 4, SAS Slot: 1 phys at slot 15
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece3 addr 5000cca23b577b61
ses0: da3 in Array Device Slot 5, SAS Slot: 1 phys at slot 9
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece4 addr 5000cca23bb32935
ses0: da2 in Array Device Slot 6, SAS Slot: 1 phys at slot 8
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece5 addr 5000cca23b588ca5
ses0: da4 in Array Device Slot 7, SAS Slot: 1 phys at slot 10
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece6 addr 5000cca251bad2dd
ses0: (none) in Array Device Slot 8, SAS Slot: 1 phys at slot 11
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece7 addr 5000cca2669296f5
ses0: da0 in Array Device Slot 14, SAS Slot: 1 phys at slot 0
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84eced addr 5000c50016eb64c9
ses0: da1 in Array Device Slot 16, SAS Slot: 1 phys at slot 3
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ecef addr 5000c500169bee11

Show : dmesg | grep transfers

ses0: 300.000MB/s transfers
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
da0: 300.000MB/s transfers
da2: 1200.000MB/s transfers
da3: 1200.000MB/s transfers
da1: 300.000MB/s transfers
da4: 1200.000MB/s transfers
da5: 1200.000MB/s transfers
da6: 1200.000MB/s transfers
da8: 1200.000MB/s transfers
da7: 1200.000MB/s transfers
da9: 1200.000MB/s transfers

The two 300 are also SAS disks with 6gb/s. I should build a volume with my sata dom this has 600mb

Do you agree that is something controller related? Because when I swap just the connections on the controller the other cage gets the full speed. So in that case I don't have to organize new cables or cages to dig further. Just to be prepared...;-)

all cages have the sideband connected. In the addon one the sideband is inside the sff-8643 with no extra connector on the cage end. On the main one its a pinout plug which is connected. Due to the fact that the cages are equipped with two ports each 4 drives, I have both sidebands connected on the main cage as in the manual from supermicro. Also jumpered to sgpio and not i2c. On the addon cage there are no big options for settings, just fan enable/disable and the temp monitor.

When I compare to a HP/NETAPP or so, I can clearly see cage X, slot X and so on. When I compare to here it's not clearly separated. In the manual of the broadcom I read something about a virtual enclosure which they create or so. Never hered about it. May this could be the reason that everything is just listed on the same hirarchy. Just guessing... First time building it truly from scratch...;-)

 

[sretalla]

Do you agree that is something controller related?

Yes, it seems to be the controller as far as I can see.

I'm far from an expert on that chassis (nor on the specific card model) so hopefully someone with direct experience with both can chip in.

 

[swissroot]

short update...

my brain said just check everything again, starting from pci config to firmware usw...

I downloaded the newest p16 firmware from broadcom, reflashed the adapter and rebooted the machine. And what the hell????

Show : dmesg output

root@truenas[~]# dmesg | grep transfers
ses0: 300.000MB/s transfers
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
da2: 1200.000MB/s transfers
da5: 1200.000MB/s transfers
da7: 1200.000MB/s transfers
da9: 1200.000MB/s transfers
da4: 1200.000MB/s transfers
da3: 1200.000MB/s transfers
da6: 1200.000MB/s transfers
da8: 1200.000MB/s transfers
da0: 600.000MB/s transfers
da1: 600.000MB/s transfers
da10: 600.000MB/s transfers
da11: 600.000MB/s transfers
da12: 600.000MB/s transfers
da13: 600.000MB/s transfers
da14: 600.000MB/s transfers
da15: 600.000MB/s transfers



root@truenas[~]# dmesg | grep ses
ses0 at mpr0 bus 0 scbus0 target 16 lun 0
ses0: <BROADCOM VirtualSES 03> Fixed Enclosure Services SPC-5 SCSI device
ses0: Serial Number 300605B00E11ECE0
ses0: 300.000MB/s transfers
ses0: SES Device
ses1 at ahciem0 bus 0 scbus7 target 0 lun 0
ses1: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses1: SEMB SES Device
ses1: ada0,pass11 in 'Slot 00', SATA Slot: scbus1 target 0
ses0: da6,pass6 in Array Device Slot 1, SAS Slot: 1 phys at slot 12
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece0 addr 5000cca23bb357c1
ses0: da2,pass2 in Array Device Slot 2, SAS Slot: 1 phys at slot 8
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece1 addr 5000cca2668999ad
ses0: da7,pass7 in Array Device Slot 3, SAS Slot: 1 phys at slot 13
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece2 addr 5000cca2667a3359
ses0: da3,pass3 in Array Device Slot 4, SAS Slot: 1 phys at slot 9
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece3 addr 5000cca23b577b61
ses0: da8,pass8 in Array Device Slot 5, SAS Slot: 1 phys at slot 14
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece4 addr 5000cca23bb32935
ses0: da4,pass4 in Array Device Slot 6, SAS Slot: 1 phys at slot 10
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece5 addr 5000cca23b588ca5
ses0: da9,pass9 in Array Device Slot 7, SAS Slot: 1 phys at slot 15
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece6 addr 5000cca251bad2dd
ses0: da5,pass5 in Array Device Slot 8, SAS Slot: 1 phys at slot 11
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ece7 addr 5000cca2669296f5
ses0: da0,pass0 in Array Device Slot 14, SAS Slot: 1 phys at slot 2
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84eced addr 5000c50016eb64c9
ses0: da1,pass1 in Array Device Slot 16, SAS Slot: 1 phys at slot 3
ses0: phy 0: SAS device type 1 phy 0 Target ( SSP )
ses0: phy 0: parent 500605b00e84ecef addr 5000c500169bee11
ses0: da10,pass13 in Array Device Slot 12, SAS Slot: 1 phys at slot 1
ses0: phy 0: SATA device
ses0: phy 0: parent 500605b00e84eceb addr 300605b00e84eceb
ses0: da11,pass14 in Array Device Slot 10, SAS Slot: 1 phys at slot 0
ses0: phy 0: SATA device
ses0: phy 0: parent 500605b00e84ece9 addr 300605b00e84ece9
ses0: da12,pass15 in Array Device Slot 15, SAS Slot: 1 phys at slot 7
ses0: phy 0: SATA device
ses0: phy 0: parent 500605b00e84ecee addr 300605b00e84ecee
ses0: da13,pass16 in Array Device Slot 13, SAS Slot: 1 phys at slot 6
ses0: phy 0: SATA device
ses0: phy 0: parent 500605b00e84ecec addr 300605b00e84ecec
ses0: da14,pass17 in Array Device Slot 11, SAS Slot: 1 phys at slot 5
ses0: phy 0: SATA device
ses0: phy 0: parent 500605b00e84ecea addr 300605b00e84ecea
ses0: da15,pass18 in Array Device Slot 9, SAS Slot: 1 phys at slot 4
ses0: phy 0: SATA device
ses0: phy 0: parent 500605b00e84ece8 addr 300605b00e84ece8

The transfers now are all up to that what they should. The only thing what I actually not understand, ok two things, what was wrong before???? and why ses0 stays at 300mb.

Now I have to do some benchmarks to see if those 300mb are correct or just a wrong bit in the data-heaven.

so thanks for your help and support!

If anyone can explain me the ses0 behaviour would be great. will post an update when I've done the dd's and copy tests.

thx!