TrueNas not detecting SATA drives through LSI 9201-16E card

[BRIO]

Title is pretty self explanatory. I'm currently trying to connect SATA disk outside of my server through a LSI 9201-16E card but for some reason the disks aren't showing up in TrueNas.

Here is the diagram of how it is build:

Everything works together, the only items that I can't test separately are:

1. LSI 9201-16E card
2. CABLEDECONN Mini Sas cable
3. Cablecc Dual Ports Mini Sas

My wild guess it that the problem might be the LSI card since I purchased it on Ebay from a Chinese seller but I was wondering if there is a procedure I should do with TrueNas to make it work or if there is any item incompatibility that you guys could spot in my build? Also, if any of you have another LSI card or reseller to suggest me that you are sure that will work with my system, I would be glad to hear more about it!

----------------------------------

SPECS:

SERVER
Motherboard: ASUS Pro WS WRX80E-SAGE SE WIFI II AMD WRX80
CPU: AMD Ryzen Threadripper PRO 5955WX
RAM: Team T-Create Classic 10 LAYERS 64GB (2 x 32GB) 288-Pin DDR4 SDRAM DDR4 3200 (PC4 25600) Desktop Memory Model
External SAS Card: LSI 9201-16E 6G 16-lane external SAS HBA IT Mode ZFS FreeNAS unRAID NoROM

CABLE IN BETWEEN
Cable: CABLEDECONN Mini SAS26P SFF-8088 to SFF-8088 1M External Cable Attached SCSI

DISK RACK:
Port: Cablecc Dual Ports Mini SAS SFF-8088 to SAS 36Pin SFF-8087 PCBA Female Adapter with PCI Bracket
Cable: YIWENTEC Mini SAS 36P SFF-8087 to 4 SATA Target Hard Disk Connector Data Cable 1m
Hard Drive: SATA 16TB Ultrastar DC HC550

 

[jgreco]

You have far too much cable. The maximum allowable SATA pathway is 1 meter. You have at least two.

 

[BRIO]

You have far too much cable. The maximum allowable SATA pathway is 1 meter. You have at least two.

Thanks a lot for the quick reply, hopefully it will be that simple, will check into shorter cable!

 

[Arwen]

You have far too much cable. The maximum allowable SATA pathway is 1 meter. You have at least two.

Thanks a lot for the quick reply, hopefully it will be that simple, will check into shorter cable!

Do you have a SAS Expander chip in the disk rack?

You almost certainly do, because it appears you have 24 disks, but only 8 lanes wired.

If so, your external cable length is less important. The SAS Expander would perform SATA to SAS Tunneling protocol. Then SAS protocol from the SAS Expander to the SAS controller chip. As long as you don't exceed SAS cable lengths, then that is likely not the problem.

 

[jgreco]

Do you have a SAS Expander chip in the disk rack?

You almost certainly do, because it appears you have 24 disks, but only 8 lanes wired.

Facts not in evidence. A TQ chassis or A chassis, which are very common, will not have an SAS expander. The model number has to say "E1" or "E2" to be equipped with an expander. At least for Supermicro. The use of a breakout cable with discrete SATA data connectors strongly suggests a TQ backplane, which is why I didn't consider the possibility of an expander backplane. However, you could certainly install an SAS expander card in the chassis, run SAS to it, and then break out SATA from inside the chassis. Length limit not violated that way due to the SAS-to-SAS jump from the server to the chassis.

 

[BRIO]

Do you have a SAS Expander chip in the disk rack?

You almost certainly do, because it appears you have 24 disks, but only 8 lanes wired.

If so, your external cable length is less important. The SAS Expander would perform SATA to SAS Tunneling protocol. Then SAS protocol from the SAS Expander to the SAS controller chip. As long as you don't exceed SAS cable lengths, then that is likely not the problem.

Facts not in evidence. A TQ chassis or A chassis, which are very common, will not have an SAS expander. The model number has to say "E1" or "E2" to be equipped with an expander. At least for Supermicro. The use of a breakout cable with discrete SATA data connectors strongly suggests a TQ backplane, which is why I didn't consider the possibility of an expander backplane. However, you could certainly install an SAS expander card in the chassis, run SAS to it, and then break out SATA from inside the chassis. Length limit not violated that way due to the SAS-to-SAS jump from the server to the chassis.

Hmm, I don't think there is a SAS expander integrated in the disk shelf. I actually purchased it on AliExpress, it was the only disk shelf that met all the criteria I needed. The backplane actually have 24 SATA slot so I doubt that's a SAS expander as you said. I only have 8 disk in the shelf yet so that's why I only have 8 lines.

Here is the link: https://www.aliexpress.com/item/100...o.order_list.order_list_main.4.7bdc1802DtZeMX


Though, what is a SAS expander exactly? Might solve my problem here if the shorter cables doesn't work. I made some search but all I can find is some cards similar to the LSI one I have. Would you have some link of what you mean by "SAS expander"?

 

[jgreco]

Though, what is a SAS expander exactly? Might solve my problem here if the shorter cables doesn't work. I made some search but all I can find is some cards similar to the LSI one I have. Would you have some link of what you mean by "SAS expander"?

See the SAS Primer. An SAS expander is like an ethernet switch but for SAS.

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

 

[WI_Hedgehog]

Also note the LSI-based SAS cards are like full-blown computers scaled down to embedded system size; buying used sometimes means you need to do the equivalent of a wipe and full reinstall. Admittedly that's rare and it's more often there's a data settings issue (like 'persistent' settings that refuse to go away), but underestimating what these cards are capable of and the uses some companies found for them can cause lots of headaches.

 

[Arwen]

Facts not in evidence. A TQ chassis or A chassis, which are very common, will not have an SAS expander. The model number has to say "E1" or "E2" to be equipped with an expander. At least for Supermicro. The use of a breakout cable with discrete SATA data connectors strongly suggests a TQ backplane, which is why I didn't consider the possibility of an expander backplane. However, you could certainly install an SAS expander card in the chassis, run SAS to it, and then break out SATA from inside the chassis. Length limit not violated that way due to the SAS-to-SAS jump from the server to the chassis.

I could not find the information on the "disk rack", so I asked the question...

Where did you find the model of the "disk rack"?
(Before the OP listed the web site...)

 

[jgreco]

Where did you find the model of the "disk rack"?

I inferred it. First off, the model of the disk rack is irrelevant. It's the length limit that's causing the fault here. Second off, BECAUSE the cable being used inside the disk chassis is a standard break-out cable, this means that the OP had to have been able to attach it SOMEHOW. SATA data connectors are not going to fit into any backplane with an SFF-8087 or SFF-8643 backplane; hence it is probably not either a direct wire-thru like a Supermicro -A backplane, and similar logic eliminates -E1/-E2 expander backplane options. If you have something like a -TQ with discrete wire-thru capability, that would absolutely work with the OP's described situation. But this does not force it to be a SUPERMICRO -TQ backplane. Still, for all the other various SATA and SAS backplanes out there, most of the ones that take discrete SATA data connectors are going to be SATA, and it is generally unusual to find someone having bodged together something that includes a SAS expander in their disk rack without having mentioned it.

Could it possibly have been something else? Maybe, but I'm not picturing it, and I'm pretty mentally flexible about dreaming up possibilities.

 

[Arwen]

Thank you for the reasoning. I had overlooked both the length and breakout nature of the internal cable.

And you are right, most people buying lower end disk shelves would not know if they should buy one with a SAS Expander. In fact, given cost, they might purposefully reduce the cost and not get a SAS Expander.

 

[BRIO]

I received the shorter cables and tested everything. It doesn't work as expected + the fact that the cable are short make it impossible to reach the far right disks of the shelf.

That being said, from what I read from you guys, I can forget about a backplane SAS expander + to be honest, I feel like it will be kind of a headache to replace the actual backplane.

So option are left? Is there any PCIe card that can "work as SAS expander"? Or any other idea of how I can make my setup work?

 

[samarium]

You can get sas expanders in many formats. Backplane is just one. You can get an expander in a PCIe format which just uses PCIe for power, and some even have molex connecters that could be used if you want to mount the card some where arbitrary in the case. If you were to get one of these expanders, and mount it in the external case, and power it there, and then use 8088/8087 cables to bridge the case gap as SAS, then the expander splits out to SATA, that might work out for you. IIRC Intel has a 2x 8087 IN, 4x 8087 out card, there were plenty of other when I was looking around aliexpress recently, so there are probably some on amazon and ebay too. You will have to look around to find something that works for you, and you might have to read for background.

 

[smcclos]

So option are left? Is there any PCIe card that can "work as SAS expander"? Or any other idea of how I can make my setup work?

I too have a configuration similar to yours, and from what I have researched, here are options that could be considered:

1. Go with a complete JBOD from Ebay, here is an example
2. Lenovo 03X3834 SASExpander (Ebay), it comes with 2 SFF-8087 connectors for input and 4 SFF-8087 for output, Detailed video on Youtube
   1. and Since it is a PCI-e Card, you need a PCI-e riser card with power (Amazon),
3. Dell 6TDVN 5R10N SAS Expander (Ebay), it comes with 3 external SFF-8088 connectors, and 6 internal SFF-8087 connectors, Detailed video on YouTube

 

[WI_Hedgehog]

@smcclos : That Lenovo SAS Expander looks like a knock-off (for many reasons); others have had reasonable luck with them, I personally don't trust my data storage systems to use knock-off equipment.

 

[samarium]

So what do you regard as a knock off, and what is acceptable?
ie buy genuine LSI HBA/Expander card, or SMC / Dell / HP / Lenovo / Fujitsu / Intel / ... version?
Are they all knock offs and not worth considering?
Or is A reliable, and B not reliable?
Is there a why? Or just accumulated anecdotes which would be difficult to be generally accessible.
Similar for NICs I expect, although the players may change.
Or is this already codified in a Resource page?
Probably could say the same for VGA cards, most of them use reference designs, but some actually seem to do better than the reference design.

 

[WI_Hedgehog]

@samarium : I would say a brand-name card like LSI/Dell/IBM/etc. which use genuine LSI chips are almost always highly reliable (I have one half-dead LSI card, so they're not "invincible"). Reverse-engineered in China and made to chabunduo standards, while affordable, is not something I would use. Note in some very specific situations I would use a Chinese mainboard, HBA, video card, case, fans, although for a power supply and hard drives I'd stay name-brand.

Personally, I use 16-port LSI HBAs with SAS drives instead of expanders, but that's probably overkill for most applications--highly reliable though.

I've lost data in the past and for the price of used server equipment vs. the value of data retention it's worth going the route I'm going. The hardware resource is really valuable. If you look at my signature (or profile) there's a list (resource) of resources in logical order.

 

[BRIO]

Alright everyone, just an update on how it turned out, I managed to make it work!

1. I purchased a LSI 9300-8e card off Ebay: https://www.ebay.com/itm/166214474471
2. I purchased these SFF-8644 to 4 Sata cable off Amazon: https://www.amazon.ca/dp/B08NT76CD8?psc=1&ref=ppx_yo2ov_dt_b_product_details

I plugged the card to the server then the cables from the card that goes directly to the HDD in the disk shelf. I didn't even needed to flash the card, it worked out of the box. I currently bought a LSI 9300-16e card to test it as well to get more of a single PCIe port. I plan to test longer cables than 1m as well to see if it works too. Will keep you updated!

(btw, how can I mark this topic as "solved"?)

 

[WI_Hedgehog]

I plan to test longer cables than 1m as well to see if it works too.

Unless you have a cable tester ($8,000 USD and up) it's more like seeing if the error rate is low enough for the auto-recover routines to fix the problems. Note some checksums will fix some errant data and your data will corrupt eventually, which ZFS will probably fix.

Manufacturers are always trying for greater speeds at given cable lengths, if "faster" were an option they already used it.

Conversely cable manufacturers are always looking to cut costs and still meet the specs, so you're tempting fate by exceeding specs.

 

[smcclos]

purchased these SFF-8644 to 4 Sata cable off Amazon: https://www.amazon.ca/dp/B08NT76CD8?psc=1&ref=ppx_yo2ov_dt_b_product_details

I didn't know that this cable was made, now I'm curious and will have to try it out.