External SAS(8088) Enclosure drives timing out on boot depending on how many drives are in the enclosure

[cabyio]

Okay, I don't have any experience with external sas enclosures so this might be something easy, but here are the details. I'll give a summary of the issue then as much detail and logs as I can.

Summary of the Issue:
----------------------------
I am setting up a new TrueNAS Scale server. I have a Dell T7810 Workstation as the hardware to run TrueNAS and have installed TrueNAS Scale on two mirrored SSDs. That part all seems to work fine. However just yesterday I received my external sas enclosure, an 8 bay enclosure with 2 8088 ports to run to the LSI SAS9207-8E card I have in the dell. If I put one drive in the enclosure it show up exactly like I would expect. However when I start to put more than that in it gets weird and only some drives show up. It looks like they are timing out in DMESG which I will post a little farther down the post. The drives are inconsistant about loading, and that changes when other drives are put in. See what I mean below.

Experiment to see if I could Identify a pattern:
---------------------------------------------------
I filled the bottom 4 bays of the enclose with 4 - 4TB drives (in bays 4,5,6,7). When booted with those 4 drives I can see 2 of the drives, specifically the ones in bays 5 and 7.

sas2ircu with all 4 drives
dmesg with 4 drives (around line 1325 you can see some errors when its trying to startup all the sas drives)

Then I took the drive out of bay 7, and left the other three in. When booted, now I can still see 2 drives but this time its the drives in bay 5 and 6, where bay 6 was one of the ones that didn't load in the previous boot. Here are the same 2 log files for the three drive boot.

sas2ircu with 3 drives
dmesg with 3 drives (~line 1325 for errors)

When I removed the drive from bay 6 and just left 2 drives in I am only able to see 1 drive, the one in bay 5. Here are the same logs.

sas2ircu with 2 drives
dmesg with 2 drives (~line 1325 for errors)

When I drop it down to the single drive in bay 4 (which so far hasn't shown up at all), I can see that one drive in bay 4 just fine. Logs:

sas2ircu with 1 drive
dmesg with 1 drive (no errors on this one, just it loading the 1 drive on ~ line 1325)


So do I have a bad piece of hardware? I have tried 2 different sas cables and also tried a similar setup in the top bays (0-3) and I get very similar results. I just need a little push in the right direction here. What do you think is wrong?

Thanks,

 

[Arwen]

It appears you are using SATA drives in your 8 bay external enclosure. This is fine by the controller. BUT, you MUST keep the cable lengths SHORT.

When a SAS controller's disk port is wired to a SATA drive, the signaling voltage levels drop to SATA's lower distance & voltage mode. You MUST take in to account both the external cable, and any distance internal to the external disk enclosure. The maximum is 1 meter, not including additional line losses due to connectors, which would likely be another 2 in your case.

With more drives, you might be getting cross talk on your cables. Making the extra drives un-recognizable.

You don't describe all your hardware. Please do so with make, model, and length of cables.


All that said, I don't know excessive cable lengths are your problem... I simply don't have enough information.


Now if your external disk enclosure has a SAS Expander, that is a whole different situation. Then, the length of cable from SAS controller to the SAS Expander is limited to 2 meters.

 

[cabyio]

Thanks for responding, here is some more details.

I am using SATA Hard Drives. Specifically these: https://www.amazon.com/dp/B09NHV3CK9?psc=1&ref=ppx_yo2ov_dt_b_product_details

Here is the enclosure I am using: https://www.pc-pitstop.com/scsat84xt

And here are the 0.5 M cables I am using: https://www.amazon.com/dp/B09B26TW7W?psc=1&ref=ppx_yo2ov_dt_b_product_details

I was trying to use the shortest external cables I could find. I have opened up the enclosure and looked at the internal cables running to the backplane in the enclosure and they look like they are decently long, at least 1M, maybe more (I didn't want to unbundle them just to measure). If I were to replace those internal cables with shorter ones, say 0.5 M. That would be about 1M for the whole run, do you think that would work? Or is this kind of setup just a bad idea?

Thanks again for your input.

 

[Arwen]

Well, if the internal data cables on that enclosure are really long, as you say, 1 meter, that could cause problems.

If you replaced the internal data cables on the disk enclosure with as short as possible, that might work.

Keep in mind, it's 1 meter total, and the data connector on the external disk enclosure reduces the overall length available. How much reduced length, I don't know...

 

[cabyio]

Okay, the shortest internal cable I could find was 0.5M. So technically with both cables at 0.5M and the extra length of the connector I am still over 1M. The behavior seems the same with this setup. Dang, was hoping that would fix it.

If I get SAS drives instead of SATA ones would that make the cable length not an issue?

 

[Arwen]

Well, SAS drives may work... SAS is designed for longer cables.

It almost seems like a cross talk issue on the external cable or the external enclosures' connectors. If one disk is talking, it may create enough noise on another line to prevent that other disk port from working.

Did you buy good cables?

Cheap or off-brand cables could also be a cause.

 

[WI_Hedgehog]

What's the SMART log for each drive show???

At this point I'd guess what @Arwen states plus possibly power supply noise. I bought a 4-channel oscilloscope specifically to diagnose noise problems. Something about Noctua fans (besides being quiet due to low RPM and less air moved) is they don't generate much line noise compared to inexpensive Chinese fans, but even those are fine as long as the PSU is filtering noise properly.

 

[cabyio]

These are the external cables I am using:

https://www.amazon.com/dp/B09B26TW7W?psc=1&ref=ppx_yo2ov_dt_b_product_details&ozlp-20

I have no experience with SAS stuff so I have no idea if the cables I bought are bad or not. Can you point me toward a reliable brand?

 

[WI_Hedgehog]

These are the external cables I am using:

https://www.amazon.com/dp/B09B26TW7W?psc=1&ref=ppx_yo2ov_dt_b_product_details&ozlp-20

I have no experience with SAS stuff so I have no idea if the cables I bought are bad or not. Can you point me toward a reliable brand?

AMAZON REVIEW: As an electronics technician from the NAVY and a NET+ technician, I can assure you these CHEAP cable are just that. The Chinese company that makes these cables has cut the most important corner in high speed data transmission, they left out the shielding and opted for a cheaper cable. I tested and tested very thoroughly and using these SAS cable will lead to data corruption. I will be getting my money back and I recommend you buy from either Tripp-lite as they are shielded, or from Taiwanese company Star-Tac as they are also shielded. I am going by the printed information on the cable, and if it is shielded it is supposed to say shielded, plus these cable feel too soft to have rigid shielding in them. 3m for length is pushing your luck in the first place with shielded cable, but un-shielded is a disaster waiting to happen as my SAS Drive-bay locked up many many times during testing.

 

[cabyio]

Well, shit. Thank you for pointing that out. I usually read the 1 star reviews on things I buy. Guess I skipped that step this time.

 

[WI_Hedgehog]

Can you point me toward a reliable brand?

I think other members could point you in the right direction, because I like my cables like I like my women.
(and you can expect both to let you down when it comes to anything important)

Personally, I'd ask here on the TrueNAS forums,
and otherwise go to:

https://www.ArtOfServer.com/#about​

Go to my eBay store​

Search for: SFF-8088 to SFF-8088​

and confirm with him what I pick would work for what I intend before purchasing.

But first I'd ask here.

---
Realistically I like high-quality cables and women at budget prices, and For A Limited Time Only, but if "cheap" gets the job done...

 

[Arwen]

Yes, if the review is correct about the lack of shielding, then you need a different cable.

Plus, that cable is listed as;

Data Transfer Rate

3 Gigabits Per Second

The HBA is more than likely trying 6Gbps for any modern SATA III disk. You could try and limit the speed to 3Gbps, which won't affect hard drive access because they all have direct connections to the HBA. How to limit the speed, I don't know.

As for what brand / manufacturer to use, I can't recommend any specific one. But, sticking with more well known brands is generally what is suggested in cases like this. (Even if the brand name cables cost more...)

 

[WI_Hedgehog]

You could try and limit the speed to 3Gbps, which won't affect hard drive access because they all have direct connections to the HBA. How to limit the speed, I don't know.

LSI Logic MPT Configuration Utility, Version 1.71, Sep 18, 2013
lsiutil_1.71_Linux.x86_64 -i
Link Max Rate: 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0

Change each port to have a maximum speed of 1.5 Gbps and see if it works, if so then try 3.0 Gbps.