I have a very strange problem with my HBA. I'm doing a parallel read test from all of the drives using:
foreach i ( da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23 )
dd if=/dev/${i} of=/dev/null bs=1M &
end
iostat -x da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23 5
My result under FreeNas 9.3 is very low - around 110MB/s per drive when my HBA is in any of the 4 gen3 PCIe slots connected to CPU2 while if I plug the HBA in the only slot connected to CPU1 (also gen3) the result is around 210MB/s per drive which is the maximum speed of those drives. The card is gen3, using 8 pcie lanes and all those slots are gen3 with 8 or 16 lanes. There aren't any other cards plugged in during this test. Under Linux (CentOS 6.7) this problem does not exists - I'm always getting around 210MB/s average read speed per drive no matter which slot I'm using.
I know it looks crazy that's why I asked in another thread how do I check how many lanes my card is currently using but it turns out that lspci -vv under FreeBSD doesn't show that the way Linux shows it - you can only see info about motherboard devices and not connected cards. Anyway under Linux everything seems normal:
81:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
....
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
I've reset my bios to defaults and made sure that all PCIe slots settings are set to gen3 and maximum lanes. Any ideas how I can check what's going on? Maybe it has nothing to do with PCIe - then I really don't know how moving the HBA to the only PCIe slot connected to CPU1 fixes the issue.
My system:
Chasis: Supermicro SC846BE1C-R1K28B
MB: Supermicro X10DRi-T4+
HBA: AOC-S3008L-L8e
CPU: 2x Intel Xeon E5-2623 v3
RAM: 256GB - 8x 32Gb ECC DDR4
Boot: 2x Supermicro SSD-DM032-PHI SATA DOM
HDD: 24x6TB Hitachi SAS3
foreach i ( da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23 )
dd if=/dev/${i} of=/dev/null bs=1M &
end
iostat -x da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23 5
My result under FreeNas 9.3 is very low - around 110MB/s per drive when my HBA is in any of the 4 gen3 PCIe slots connected to CPU2 while if I plug the HBA in the only slot connected to CPU1 (also gen3) the result is around 210MB/s per drive which is the maximum speed of those drives. The card is gen3, using 8 pcie lanes and all those slots are gen3 with 8 or 16 lanes. There aren't any other cards plugged in during this test. Under Linux (CentOS 6.7) this problem does not exists - I'm always getting around 210MB/s average read speed per drive no matter which slot I'm using.
I know it looks crazy that's why I asked in another thread how do I check how many lanes my card is currently using but it turns out that lspci -vv under FreeBSD doesn't show that the way Linux shows it - you can only see info about motherboard devices and not connected cards. Anyway under Linux everything seems normal:
81:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
....
LnkSta: Speed 8GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
I've reset my bios to defaults and made sure that all PCIe slots settings are set to gen3 and maximum lanes. Any ideas how I can check what's going on? Maybe it has nothing to do with PCIe - then I really don't know how moving the HBA to the only PCIe slot connected to CPU1 fixes the issue.
My system:
Chasis: Supermicro SC846BE1C-R1K28B
MB: Supermicro X10DRi-T4+
HBA: AOC-S3008L-L8e
CPU: 2x Intel Xeon E5-2623 v3
RAM: 256GB - 8x 32Gb ECC DDR4
Boot: 2x Supermicro SSD-DM032-PHI SATA DOM
HDD: 24x6TB Hitachi SAS3
