SOLVED SAS Expander only showing 3gb

[needo]

Greetings,
I recently purchased one of these... https://www.supermicro.com/products/chassis/4U/847/SC847E16-R1400LP.cfm I was overjoyed to read that it supported 6GB.

I threw in my trusty M1015 flashed to IT and my bundle of WD Red 6TB drives but the expander seems to only be set to 3GB.

ses0 at mps0 bus 0 scbus0 target 23 lun 0
ses0: <LSILOGIC SASX36 A.1 7015> Fixed Enclosure Services SCSI-3 device
ses0: 300.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device

Is there a way to fix this? Scrubs are going to take almost a week at 3GB

 

[Nick2253]

Something else is going on. First off, a spinning platter disk isn't going to be able to hit 3Gbps, so it really shouldn't matter if you're at 3Gbps or 6Gbps. Second, even if it was the problem, going from 3Gbps to 6Gbps would at best halve the scrub time, and even 72 hours is an excessive time to do a scrub.

Step one: post your system specs.

Step two: run SMART tests and check for a failing drive. Failing drives could be part of why your scrub takes so long.

NB: there's a difference between 3GB, which is a unit of space, from 3GB/s, which means three gigabytes per second, from 3Gbps, which means 3 gigabits per second.

 

[needo]

Thank you. Below are my system specs:

Chassis: Supermicro CS-847
Motherboard: Supermicro X10SRL-F
CPU: Intel Xeon E5-1630v3
CPU Cooler: Supermicro SNK-P0048AP4
RAM: HMA42GR7MFR4N-TF 64GB (4 x 16GB) 288-Pin DDR4 2133
HMA84GR7MFR4N-TF 32GB (1 x 32GB) 288-Pin DDR4 2133
Boot Devices: 2xSanDisk Cruzer Fit 16GB USB 2.0 Low-Profile Flash Drive SDCZ33-016G-B35
Hard Drives: 18xWD 6TB Red (WD60EFRX)
SAS Cards: 1xIBM M1015 LSI SAS9220-8i

 

[Nick2253]

How are your pool(s) configured? Shares? Jails?

 

[needo]

Three vdevs. Each vdev contains 6 drives in RAIDz2.

 

[Spearfoot]

@needo, can you verify the backplane part number actually installed in your system? The 847 chassis is sold with several different backplanes, not all of them support SAS2 (6Gb/s).

 

[needo]

Is there a way to do so remotely? I do not have physical access to the machine at this moment.

Thank you.

 

[Spearfoot]

Is there a way to do so remotely? I do not have physical access to the machine at this moment.

Thank you.

Do you have IPMI access to the server? If so, you can boot into the BIOS and check there to see what's installed.

 

[Spearfoot]

Barring IPMI access... did you buy the server new? From eBay? Can you confirm with the seller which part number you purchased? Or give us the eBay item link?

 

[needo]

It is not listed in the BIOS, I purchased this... http://www.ebay.com/itm/401177145910?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT

 

[Sakuru]

Those cheap eBay chassis frequently come with SAS1 backplanes, unfortunately.

 

[Spearfoot]

It is not listed in the BIOS, I purchased this... http://www.ebay.com/itm/401177145910?_trksid=p2057872.m2749.l2649&ssPageName=STRK:MEBIDX:IT

I was afraid of something like this... the seller didn't list the full part number of the server or backplane and was cagey about providing much information. I'm pretty sure you bought a system with a SAS (3Gb/s) backplane.

EDIT: As pointed out by @Sakuru above. Boy! You guys are fast around here!

EDIT 2: @needo, you purchased this box on Aug 29th and it came with a 30 day money back return policy... so you have the option of shipping it back to the seller, albeit you'll have to pay the freight.

 

[needo]

Yeah, I did some googling and found this page: http://www.natecarlson.com/2010/05/07/review-supermicros-sc847a-4u-chassis-with-36-drive-bays/

So, it looks like I got one of these:

• ‘E1’ style (IE – SC847E1) – This chassis includes backplanes with integrated 3gbit/s SAS expander, without multipath support. Each backplane has one SFF-8087 connector, so you only need two SFF-8087 ports in a SC847E1 system. The downside is that you are limited to 3gbit/s per channel – so you’d have a total of 6 drives on each 3gbit/s channel for the front backplane, and 3 drives on each channel for the rear backplane. SuperMicro also has a ‘E16’ option (IE – SC847E16) which is upcoming, and supports SATA3/SAS2, for a total of 6gbit/s per channel.

Which is this... https://www.supermicro.com/products/chassis/4U/847/SC847E1-R1400LP.cfm

Grrr... so the question I have for you all. Is it worth returning, since as it was pointed out I will really never see full 3bit/s speeds out of mechanical hard drives. (I currently have 18 drives on the single front backplane.) Another option is replacing the backplane.

Is it worth it?

 

[danb35]

Is it worth it?

Up to you, but I'd lean toward "no"--if your drives are all recognized at their full capacity (there's been some suggestion that the SAS1 backplanes have a size limit of about 2 TB). I don't think it should slow down your scrubs as much as you think--mine (also with 18 drives, but on a SAS2 backplane) usually run about 24 hours for around 20 TB of data.

 

[needo]

Yes, they are all seen at 6TB. The only reason I noticed was my scrub was not breaking 200MB/s and was reported to take 5+ days with 40TB of data.

 

[Mirfster]

Maybe it was just me, but when I connected my SLOG to the MB SATA Port (which is 3G on the C2100/FS12-TY), it was not as fast as on 6G (to the Perc H200). Keep in mind that is an Intel DC S3710 SSD and I may have been imagining it... ;)

 

[Spearfoot]

If it were me, I'd take my lumps, ship the thing back, and look for a more suitable server with a SAS2 backplane. There are plenty of these available on eBay...

 

[depasseg]

Greetings,
I recently purchased one of these... https://www.supermicro.com/products/chassis/4U/847/SC847E16-R1400LP.cfm I was overjoyed to read that it supported 6GB.

I threw in my trusty M1015 flashed to IT and my bundle of WD Red 6TB drives but the expander seems to only be set to 3GB.



Is there a way to fix this? Scrubs are going to take almost a week at 3GB

It would be interesting to see the full output (or at least the output of a known 6G drive). SES0 is the enclosure management, and might only support 3G, when the backplane can do 6G. IOW I don't think SES0 is always a good indicator of backplane speed.

 

[danb35]

Here's what it says on my chassis:

Code:

[root@freenas2] ~# dmesg | grep ses0
ses0 at mps0 bus 0 scbus0 target 26 lun 0
ses0: <LSI SAS2X36 0e12> Fixed Enclosure Services SPC-3 SCSI device
ses0: Serial Number
ses0: 600.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device

Edit: followed by listings for the different drives.

 

[Stux]

Modern 5200rpm HDs have now broken 1.5gbps. They're heading for 3gbps in 7200rpm drives.

The issue isn't 3gbps per drive. It's 3x4, Ie 12gbps per expander.

You're expander will bottleneck your drives at less than 8 drives.

I see up to 180MB/s per drive on my simple 4TB NAS drives. That means any more than 6 drives on a sas2 expander and the sas expander would be the bottleneck. But after that you have 32gbps pcie2 (8x) bottleneck from the SAS2 HBA anyway.

Depends on your max attainable drive speeds.