Using SATA in place of SAS drives, Cable length and SCSI Sense issues

[KempelofDoom]

So I'm adding an external case of HD's to my setup. I'm using this LSI card with this Intel expander card. Inside the server, using SFF-8087 to SFF-8087 cables to connect the expander and drive controller to SFF-8087 to SFF-8088 adapters. Then use as short as I can get, SFF-8088 to SFF-8088 cables to connect to the expansion case. The expansion case comes with a 1000W PS and what appear to be quality SFF-8087 to SFF-8087 cables to the backplane. The expansion case is SATA/SAS compatible so I went with WD Reds which are SATA.

So I connect and power everything up. I notice issues with creating pools on the expansion case immediately. I figure that I might have some bad drives. I eventually pull the cards and plug them into a different server with similar specs to the main server. I eventually pull 6 drives that were flagged by FreeNAS as faulting. The other drives were moved to different ports when I pulled the "bad" ones to see if it might have been that port on the backplane. The remaining 18 handled the stress testing I did with creating different Z pools and copying over about a TB of data and running scrubs and enabling compression and dedupe over an 8 hour period.

Then I pulled everything and plugged in the 6 "bad" drives and did the same stress testing on them. This time around, only 1 drive failed and the others have worked well except that the current scrub I'm doing is only running at 3MBps instead of the 300MBps I had seen before. Otherwise, they have been holding up well over 24 hours with an occasional drop of SMB which I'm not sure if it's the pool or network adapter.

What I noticed was that I saw errors flying when I transferred data to the pools regardless of the set of drives that were being tested. I would get SCSI Sense errors "SCSI sense: UNIT ATTENTION asc:29,0 (Power on, reset, or bus device reset occurred)". I did a little searching and found that my cables might be suspect.

Since I am using SATA drives, am I limited to the SATA cable length specifications? I think I managed to stay below the 3 feet spec when I combine the cables from the backplane to the controllers.
Should I try to update the firmware on the drives?
Does anyone have recommendations on a quality SFF-8087 and SFF-8088 cable maker?
Any other suggestions on why I'm seeing these errors?

I'm planning on removing all of the cabling and connecting the backplane's to the controllers directly and add till I see errors. However, if I still see errors when it's simplified, how should I proceed?

Edited to fix the wrong case link. Now it points to the exact item.

 

[mjws00]

I'd look at the firmware on the controller. I'm wondering if you are hitting the miss match driver error? There have been problems with newer firmware, not sensing drives correctly etc. Run LSI p16 with 9.2.1.7.

For troubleshooting and on the bench, I'd throw a different OS on there as well... if everything is flawless. Then at least you have limited your issue to drivers and FreeNAS and you aren't wasting time and money chasing physical cables, drives, and hardware.

You've made reasonable choices. I would also keep the backplanes in the norco case as a potential problem. Definitely a few issues with those online. I would go after drives last as a bad drive shows itself, and cables, if bad, should reproduce their error when you change their position.

Two bits.

 

[anodos]

I'd be wary of using SATA drives in an expansion shelf. I've read that they can be flaky. For reference see here:
https://github.com/zfsonlinux/zfs/issues/840#issuecomment-7134386
http://serverfault.com/questions/33...array-bring-the-entire-array-to/331504#331504
http://garrett.damore.org/2010/08/why-sas-sata-is-not-such-great-idea.html

I know that according to specs you should be able to do it, but sometime spec doesn't match reality.

 

[KempelofDoom]

That's what I'm worried about. I'm having a hard time finding whether or not the cable length specs are responsible with my troubles or if it's something else. No one seems to have real data that I can test or come to a conclusion. What I do find are approximates that leave me with too large a margin to say with certainty that I have to obey SATA cable length specs even if everything else is SAS quality.

 

[mjws00]

To my way of thinking there is only ONE physical layer, and that is SAS. Your controller and expander send sas level signals. The controller is simply smart enough to tunnel SATA (STP) so we can use sata drives. You should get all of the benefits of the sas physical layer.

When we encounter sata vs sas issues, it is typically due to the way that sata handles errors vs sas handling them gracefully. Personally I wouldn't sweat sas vs. sata on a good expander and hardware. But with a gun to my head, I suppose nearline sas is a maybe.

Also on the first link Anodos posted... note the guy swapped out the norco... and the problem went away ;)

That doesn't mean that something in your chassis wiring isn't messing with you. But it probably isn't a length issue, as the majority of users with that case are likely running sata. All would be in the same boat as you. I'd probably mod that case and stuff the expander in with two 8088 external cable runs. You would be ensured a short point to point sas link and the shortest possible sata run. Essentially that turns it into a 24E with an intel controller vs chenbro.

Seriously, rule out the software and driver issues. If linux, illumos, or windows is flawless on your cables and chassis.... you know exactly where to look. Good luck.

P.S Monoprice has good deals on 8087 to 8088 cables.

 

[KempelofDoom]

So I took out the middle man and connected the backplanes to the expander using the cables that came with the Intel expander and it's working flawlessly. It's doing dedupe and keeping a 60MBps transfer steady on a 16GB Intel server. I did notice that the cables in the expansion case are missing the grey cable that these supermicro ones have. The intel ones seem to have the same grey cable but not in a ribbon, just separate inside the snag sheath.

http://www.newegg.com/Product/Produ...6101355&cm_re=sff-8087-_-16-101-355-_-Product

What does that grey cable do? Is it part of the SAS spec or something that's related to the controller?

 

[cyberjock]

The grey is OOB. Out-of-Band management.

 

[Ericloewe]

Regarding cable lengths:

Only the last link in the chain uses SATA signalling, which means it must not exceed 1m.
Everything that does not end in a SATA device will use SAS signalling, encapsulating SATA in SCSI, and as such is only subject to SAS limitations, not SATA cable length restrictions.

 

[KempelofDoom]

Regarding cable lengths:

Only the last link in the chain uses SATA signalling, which means it must not exceed 1m.
Everything that does not end in a SATA device will use SAS signalling, encapsulating SATA in SCSI, and as such is only subject to SAS limitations, not SATA cable length restrictions.

So with that said, what part of my chain is considered the last link? It's currently hooked up from the drive controller and expander to the SFF-8087 to SFF-8088 adapters using dual end SFF-8087 cables. Then I use SFF-8088 to SFF-8088 to connect the server to the expansion case. It in turn uses SFF-8088 to SFF-8087 adapters then uses SFF-8087 to SFF-8087 cables to connect that to the backplanes which have SATA drives plugged in. I would think that since I'm using SAS level equipment all the way through, that only the backplane could be considered the SATA part.

I hooked up all 6 backplanes to the controller and expander directly and have been transferring several TB's of data over with ZERO errors. So now it's time to test the adapters and external cables. I'll need to get some SFF-8088 to SFF-8087 cables to test the adapters and see if the added length adds errors.

 

[Ericloewe]

So with that said, what part of my chain is considered the last link? It's currently hooked up from the drive controller and expander to the SFF-8087 to SFF-8088 adapters using dual end SFF-8087 cables. Then I use SFF-8088 to SFF-8088 to connect the server to the expansion case. It in turn uses SFF-8088 to SFF-8087 adapters then uses SFF-8087 to SFF-8087 cables to connect that to the backplanes which have SATA drives plugged in. I would think that since I'm using SAS level equipment all the way through, that only the backplane could be considered the SATA part.

I hooked up all 6 backplanes to the controller and expander directly and have been transferring several TB's of data over with ZERO errors. So now it's time to test the adapters and external cables. I'll need to get some SFF-8088 to SFF-8087 cables to test the adapters and see if the added length adds errors.

Unless the backplane has an expander, it's most likely electrically transparent, at least for the drive connections (minus the activity LEDs). This means it would be using SATA signaling.

The last point still using SAS signaling is the last expander (or the controller if no expanders are present).

The solution would be to place the expander as close to the backplane as possible (same chassis, basically), which would mean that the difficult part of the route (the external part) uses SAS signaling, which is far more robust than SATA.

 

[Z300M]

Unless the backplane has an expander, it's most likely electrically transparent, at least for the drive connections (minus the activity LEDs). This means it would be using SATA signaling.

The last point still using SAS signaling is the last expander (or the controller if no expanders are present).

The solution would be to place the expander as close to the backplane as possible (same chassis, basically), which would mean that the difficult part of the route (the external part) uses SAS signaling, which is far more robust than SATA.

No expander in use here, but with the original Monoprice 8087 cables (0.75m, IIRC) I was using I was getting errors. I got new cables from a different source and shorter (only 0.5m -- not for any reason other than price and being long enough), and the errors went away.

 

[KempelofDoom]

Unless the backplane has an expander, it's most likely electrically transparent, at least for the drive connections (minus the activity LEDs). This means it would be using SATA signaling.

The last point still using SAS signaling is the last expander (or the controller if no expanders are present).

The solution would be to place the expander as close to the backplane as possible (same chassis, basically), which would mean that the difficult part of the route (the external part) uses SAS signaling, which is far more robust than SATA.

Sorry if I'm repeating your words but I want to make sure I get what you're saying. The expansion case has 6 backplanes each combines 4 drives into 1 SFF-8087 port. That means that the SATA spec would only apply to the traces on the backplane till it hits the SFF-8087 port? If so, then the rest of the path to the controller and expander are considered SAS?

OR, are you saying that it stops being SAS spec from the drive controller and expander which means the cabling to the drives are bound to SATA cable length spec?

 

[Ericloewe]

Sorry if I'm repeating your words but I want to make sure I get what you're saying. The expansion case has 6 backplanes each combines 4 drives into 1 SFF-8087 port. That means that the SATA spec would only apply to the traces on the backplane till it hits the SFF-8087 port? If so, then the rest of the path to the controller and expander are considered SAS?

OR, are you saying that it stops being SAS spec from the drive controller and expander which means the cabling to the drives are bound to SATA cable length spec?

Let me try some crude ASCII art.

The only devices capable of interfacing SATA and SAS are Expanders or Controllers. The backplane does nothing more to the data link than provide an electrical path for it.

 

[KempelofDoom]

Let me try some crude ASCII art.

The only devices capable of interfacing SATA and SAS are Expanders or Controllers. The backplane does nothing more to the data link than provide an electrical path for it.

So it looks like the lengths shouldn't be adding to the problem per se. So then maybe the quality of the cables could be playing into the equation.

I'm at the point where I think I'll have to replace my current volume with a properly setup one since my current isn't setup with 4K sectors, improper number vdevs and raidz1 on each vdev. Then use SAS drives on the expansion case. Just crazy how expensive SAS drives and HD's in general are even though prices were supposed to have gone down by now after the flooding had caused a shortage.