-
Important Announcement for the TrueNAS Community.
The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums
LSI 9200-8e, DS4246, SSD Speeds and ses: transfer speeds
- Thread starter xnaron
- Start date
Chris Moore
Hall of Famer
- Joined
- May 2, 2015
- Messages
- 10,080
What warning message?
The one you have highlighted in yellow on your last screenshot.
This warning about not using this release for production was never present until RC2.
I received the 2 HB-1235 controllers and am trying them in the DS424X enclosures connected to the LSI 9200-8e (p20). Everything worked as expected on first boot into FreeNAS. I have been rebooting freenas to try and duplicate the issue with the negotiation of 300 vs 600MB/s. So far every reboot all the 6gb/s drives have negotiated properly to 600MB/s. With the Netapp IOM6 controllers installed some of the drives would negotiate to 300MB/s after the first FreeNAS reboot after a power cycle. I'd need to power cycle the DS4246's to get them to all negotiate at 600MB/s. I'm not seeing this behavior with the HB-1235 controllers. They see my 4TB sata drives fine as well. I don't have anything bigger to try.
update: After a few reboots I can duplicate the issue with the negotiations at 300 instead of 600MB/s with the HB-1235 controllers. I need to power cycle the controller and restart FreeNAS to get the negotiation of 600MB/s
Last edited:
I added more disks to the enclosures. I have 29 disks across the 2 enclosures. 23 of the drives are 6gbps and should negotiate to 600MB/s. On a clean power off start I can no longer get all of the drives to negotiate to 600MB/s (previously had 12 drives over 2 enclosures). I have completed 2 tests and I'm getting 19 of the 23 disks at 600MB and 4 at 300MB/s. In the 2 tests only 1 drive out of the 4 overlapped and were spread between the 2 enclosures. I'm not sure why sometimes some of the drives don't negotiate properly. Perhaps something with the backplane. I have tried different controllers (HB-1235) and different cables with the same results. The only common denominator is the drives and the enclosure backplane (not sure if it is a dumb or smart backplane). I'm pretty sure I had the same results with the PMC Sierra PM8003 netapp controller. I an reinstall and try it though to confirm. Maybe there is a bug with the 9200-8e firmware (p20) or some other Freenas piece in play here.
From what I have read the M1015 uses the LSI SAS2008 which is the same chipset as my 9200-8e card and uses the mps driver in FreeNAS. I have a 9207-8e sas2308 card coming that I will test with to see if it makes any difference. Could be a bug in the mps driver maybe or some parameter to tweak.
I booted into 18.10 Ubuntu server and checked the drive speeds with smartcl to see if it was operating at 3 or 6gbps. It had the same results as in freenas where some 6g drives (4) were at 3 instead of 6gbps.
I tried the 9207-8e (sas 2308) and it was even worse with only 10 of 22 elgible drives negotiating at 6gbps.
It definitely seems to be something with the lsi card. I remember now why I didn't use the pmc-sierra card. It had issues with smart.
Open to any suggestions on non 2008 and 2308 cards I could try.
I tried the 9207-8e (sas 2308) and it was even worse with only 10 of 22 elgible drives negotiating at 6gbps.
It definitely seems to be something with the lsi card. I remember now why I didn't use the pmc-sierra card. It had issues with smart.
Open to any suggestions on non 2008 and 2308 cards I could try.
Chris Moore
Hall of Famer
- Joined
- May 2, 2015
- Messages
- 10,080
It is not the LSI card. It might be those NetApp enclosures but I have hundreds of drives running from LSI cards at work, 124 drives in one server alone, and they just work. You are barking up the wrong tree blaming the controller. The SAS expander in the drive shelf has got to be where the problem is, but even then, it isn't really a problem since the lower speed is still faster than the mechanical speed of the drive.
I don't think there is anything wrong with the LSI. The only reason I wanted to try a different lsi chipset was because of the difference I saw between the 2008 and the 2308. The 2008 would negotiate 19 out of 23 eligible drives to 600MB/s where as the 2308 would negotiate 8 out 20 eligible drives to 600MB/s. Swapping out the NetAPP IOM6 controllers in the shelf with the Dell HB-1235 controllers made no difference in the results. The only thing that hasn't changed in the tests are the physical backplanes that the IOM6 or HB-1235 controllers plug into.
I agree 300MB/s is fine for the spinning disks I am just trying to determine root cause. I will have an opportunity next week to update the firmware in the NetAPP IOM 6 controllers to the latest. I don't think that will make a difference but it is worth a try.
I agree 300MB/s is fine for the spinning disks I am just trying to determine root cause. I will have an opportunity next week to update the firmware in the NetAPP IOM 6 controllers to the latest. I don't think that will make a difference but it is worth a try.
Chris Moore
Hall of Famer
- Joined
- May 2, 2015
- Messages
- 10,080
This is the chassis I use for my primary home NAS:
http://www.chenbro.com/en-global/products/RackmountChassis/4U_Chassis/NR40700
It has two 24 bay backplanes by Chenbro:
I have all sorts of other hardware at work from different vendors. The only thing I can an think of that would cause the kind of problems you are having is an outdated firmware or some native incompatibility.
What version of the firmware is in the SAS controller? The command
sas2flash -c 0 -list should produce a listing like the one below. If you will share that with us, it might help with the troubleshooting.Code:
root@Emily-NAS:~/scripts # sas2flash -c 0 -list LSI Corporation SAS2 Flash Utility Version 16.00.00.00 (2013.03.01) Copyright (c) 2008-2013 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2308_2(D1) Controller Number : 0 Controller : SAS2308_2(D1) PCI Address : 00:03:00:00 SAS Address : 500605b-0-09ef-7220 NVDATA Version (Default) : 14.01.00.06 NVDATA Version (Persistent) : 14.01.00.06 Firmware Product ID : 0x2214 (IT) Firmware Version : 20.00.07.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9207-8i BIOS Version : N/A UEFI BSD Version : N/A FCODE Version : N/A Board Name : SAS9207-8i Board Assembly : H3-25412-00J Board Tracer Number : SV45308383 Finished Processing Commands Successfully. Exiting SAS2Flash. root@Emily-NAS:~/scripts #
Chris Moore
Hall of Famer
- Joined
- May 2, 2015
- Messages
- 10,080
PS. The zero in the command specifies the number of the controller, so if you have more than one controller, that number might need to be one instead.
OR I just tried using the shorter command
OR I just tried using the shorter command
sas2flash -list and still got the information that is needed.Hi,
Thanks and nice setup. Here is the info looks inline with yours:
Thanks and nice setup. Here is the info looks inline with yours:
Code:
LSI Corporation SAS2 Flash Utility Version 16.00.00.00 (2013.03.01) Copyright (c) 2008-2013 LSI Corporation. All rights reserved Adapter Selected is a LSI SAS: SAS2308_2(D1) Controller Number : 0 Controller : SAS2308_2(D1) PCI Address : 00:15:00:00 SAS Address : 500605b-0-0a9a-16a0 NVDATA Version (Default) : 14.01.00.06 NVDATA Version (Persistent) : 14.01.00.06 Firmware Product ID : 0x2214 (IT) Firmware Version : 20.00.07.00 NVDATA Vendor : LSI NVDATA Product ID : SAS9207-8e BIOS Version : 07.35.00.00 UEFI BSD Version : 07.25.01.00 FCODE Version : N/A Board Name : SAS9207-8e Board Assembly : 03-25601-00B Board Tracer Number : SV53047322 Finished Processing Commands Successfully. Exiting SAS2Flash.
Chris Moore
Hall of Famer
- Joined
- May 2, 2015
- Messages
- 10,080
Did you try this with a single link to each disk shelf?
Yes it is connected with 2 cables one running to each shelf. I did that intentionally so I could split the load. I have half the drives in a pool on each shelf. The other card I was testing with is not in the machine (9200-8e). I recently flashed the firmware on it to IT mode. It was a dell branded 9200-8e. That was the card that negotiate 19 out of 23 drives right. This 9207-8e is like 8/20.
Last edited:
I want to add some experience relating to this topic:
I am having some strange issue with shucked 10TB easystore and my DS4243. The DAS got 2 IOM3 and connected to R620 with LSI 9207-8e and a Mini-SAS to QSFP cable.
If I plug the drive in when system was running, the link speed is limited to 1.5Gb/s. After rebooting the system the drive would show up with 3Gb/s (the limitation of IOM3), but if I re-seat (unplug and re-plug) the drive then it's back to 1.5Gb/s. I can reproduce this behavior on both FreeNAS and Ubuntu so most likely this is a hardware problem. Only 3 easystore 10TB was plugged into the shelf during testing.
I am not totally buying that it's NetAPP causing the problem. Across reboot the shelf kept running, so there might be something different in SAS negotiate during boot and OS running.
Another problem about the HB-1225 controller, it works but will trigger the fault LED at left front ear. It also cannot keep the temperature in check when running 7.2K SAS disks.
I am having some strange issue with shucked 10TB easystore and my DS4243. The DAS got 2 IOM3 and connected to R620 with LSI 9207-8e and a Mini-SAS to QSFP cable.
If I plug the drive in when system was running, the link speed is limited to 1.5Gb/s. After rebooting the system the drive would show up with 3Gb/s (the limitation of IOM3), but if I re-seat (unplug and re-plug) the drive then it's back to 1.5Gb/s. I can reproduce this behavior on both FreeNAS and Ubuntu so most likely this is a hardware problem. Only 3 easystore 10TB was plugged into the shelf during testing.
I am not totally buying that it's NetAPP causing the problem. Across reboot the shelf kept running, so there might be something different in SAS negotiate during boot and OS running.
Another problem about the HB-1225 controller, it works but will trigger the fault LED at left front ear. It also cannot keep the temperature in check when running 7.2K SAS disks.
Basically your experiencing the same issue as I am but with the IOM3. I shucked 10 x8TB Elements drives yesterday and plugging them in most of them were at 6Gbps. One I plugged in negotiated to 3Gbps and I reseated it and it went to 6Gbps. Another would only stay at 3Gbps. It's not related to specific slots either as on reboot a different set would go to 3Gbps. Results with the 9207-8e card were significantly worse than with the 9200-8e. The 9200-8e (sas2008) would negotiate about 19/23 drives correctly on reboot. The 9207-8e(sas2308) would only negotiate 8/20 at full speed on reboot.
I think there is something with lsi card/driver in combination with sata drives and this enclosure causing an issue. I have a 9300-8e and a 9305-16e coming to test with and will see if things change with either of those cards. In my case it's not really an issue as the lowest speed (IOM6) is 3Gbps and that is more than enough for the drives I am running. However in your case it is an issue as the drives are capable of faster than 1.5Gbps. An option for you would be to the the IOM6 controllers. They work with the DS4243 chassis and can usually be had on ebay for cheap.
Will keep this thread up to date.
I think there is something with lsi card/driver in combination with sata drives and this enclosure causing an issue. I have a 9300-8e and a 9305-16e coming to test with and will see if things change with either of those cards. In my case it's not really an issue as the lowest speed (IOM6) is 3Gbps and that is more than enough for the drives I am running. However in your case it is an issue as the drives are capable of faster than 1.5Gbps. An option for you would be to the the IOM6 controllers. They work with the DS4243 chassis and can usually be had on ebay for cheap.
Will keep this thread up to date.
Mine case is a little bit different. If drives were hot plugged they are 1.5Gb/s, if they are there on boot they will be 3Gb/s. There is no randomness like what you have seen. Granted that I only got 3 drives for now though.
What's interesting is that it is always halving the max speed. I wonder if it was because it tried to multi-path the SATA or something like that. It would be interesting to see if you put 4 IOM6 into one shelf and speed goes down to 1.5Gb/s...
Mine were also all negotiating all fine with a small number of drives like 3. Put in 20 drives and I bet your results will be more like mine.
You can't multi-path the sata without using sata interposers in the trays. I don't have any of those (yet) to test with.
I have 24 of them, but they have their own problems. The LSI chip ones would not work with SSDs for some reason, you can see the drive, SN and so forth, but it shows up as 0B capacity. The other one just plainly don't work. I spent weeks trying to troubleshoot the problem. The seller even send me a Dell H200E and a SSD they tried and worked, but somehow it didn't on mine. It was not even NetAPP because I got a 9211-8i in my desktop with SAS breakout cable and if I put the interposer between the SSD would become 0B in size.
At last the seller gave me a partial refund for the interposer, but yesterday I found out that it actually work with HDDs (at least easystore 10 TB ones). Strange
