My TrueNAS server can't recognize all SAS devices

[Hyunseok_Kim]

My HBA card can connect 1024 SAS devices.
and My NAS chassis has three backplanes and each backplane has an expander chip and 24 bays.
so I put one HBA card on my server and connect three backplanes to the HBA card.
However, TrueNAS only recognize SAS devices that are mounted on two backplanes.

for example, there are A, B, and C backplanes. each backplane has SAS devices such as A-1, A-2, B-1, B-2, C-1, C-2
TrueNAS recognize A-1, A-2, C-1, C-2 but not B-1, B-2.
If I switch cable A and B then, TrueNAS recognize B-1, B-2, C-1, C-2 but not A-1, A-2.

HBA card only recognize up to two expander chip?
Do I need one more HBA card?

Please let me have your valuable advice.

Thank you for your time.

 

[nKk]

Your HBA have 2 x8 SFF-8654 connectors. What type of cables do you use to connect the backplanes and how exactly you connect the cables?

 

[Hyunseok_Kim]

Your HBA have 2 x8 SFF-8654 connectors. What type of cables do you use to connect the backplanes and how exactly you connect the cables?

I connected two SFF-8654 to SFF-8643 Cables.
so backplane side, there are a total of four SFF-8643 connectors.
one SFF-8654 to two SFF-8643, (backplane A and B)
one SFF-8654 to one SFF-8643 (backplane C), the other SFF-8643 is not connected anywhere.

 

[HoneyBadger]

According to the documentation on the 9500-16i, each of the SFF-8654 ports is supposed to be able to automatically detect the SAS wide port configuration, and supports a 2x4 setup - so I'm not sure why it wouldn't properly detect that it's been connected to two separate backplanes. You aren't allowed to build wideports across the ports (eg: can't connect port-0/sff8643-0 and port-1/sff8643-0 to the same backplane) but you should be able to use two different backplanes on a single port.

Can you post the details of your system, including the NAS chassis model number (3x 24-bay chassis? Sounds big) - it might require some internal SAS cascading if that's possible.

 

[Hyunseok_Kim]

According to the documentation on the 9500-16i, each of the SFF-8654 ports is supposed to be able to automatically detect the SAS wide port configuration, and supports a 2x4 setup - so I'm not sure why it wouldn't properly detect that it's been connected to two separate backplanes. You aren't allowed to build wideports across the ports (eg: can't connect port-0/sff8643-0 and port-1/sff8643-0 to the same backplane) but you should be able to use two different backplanes on a single port.

Can you post the details of your system, including the NAS chassis model number (3x 24-bay chassis? Sounds big) - it might require some internal SAS cascading if that's possible.

Hi, My chassis is Innovision S86572-E12.
which has three backplanes and each backplane has an expander chip.
Total of 72 SAS devices can be connected.

Do I need one more HBA card?

 

[Hyunseok_Kim]

 

[Ericloewe]

From your description, the cable might just be defective. Have you switched the SFF-8643 connectors around to see if the problem follows one of them?

 

[Hyunseok_Kim]

From your description, the cable might just be defective. Have you switched the SFF-8643 connectors around to see if the problem follows one of them?

Hi, Yes. I did change cable to new one. but same.
and If I switch blue cable backplane side, backplane A is disabled instead.

I assume HBA card each port recognize single expander chip.

 

[nKk]

On pictures from aliexpress the backplane have 3 SFF-8643 ports. Do you have a documentation how to connect the backplanes?
What happen if you connect the free red cable to A or B backplane?

 

[Ericloewe]

Since they're expander backplanes, presumably with LSI/Broadcom expanders, the ports should all be equivalent.

Hi, Yes. I did change cable to new one. but same.
and If I switch blue cable backplane side, backplane A is disabled instead.
View attachment 62667

I assume HBA card each port recognize single expander chip.

I would hope that's not the problem, but you never know. Could be a firmware or even hardware bug.
It would be very interesting to see if the behavior changes with a different environment:

• (If you have the card's UEFI extension ROM setup) You should be able to access the card's configuration menus through the system firmware setup application. One of the options should be to examine attached devices and even a tree view of the SAS topology.
• You haven't specified TrueNAS Scale or Core - try the other option.

Also, I see no mention of having updated the firmware on the controller - please try that as well, as the 95xx cards are pretty new still and bugfixes are likely.

 

[HoneyBadger]

Hi, Yes. I did change cable to new one. but same.
and If I switch blue cable backplane side, backplane A is disabled instead.
View attachment 62667

I assume HBA card each port recognize single expander chip.

Question - what happens if you switch the red and blue cables at the HBA itself (so that port0 on the HBA is going to A+B, and port1 is going to C+null)?

I wonder if perhaps there's some oddity that made it detect only an x4 wide port on the first cable, and make some (incorrect) choice to just use an x4 on the second cable in the thought that it's going to a dual-path backplane.

Checking the SAS topology/tree in the BIOS/UEFI may provide some insight as @Ericloewe mentioned. Also, which of the ports on the backplane are you using? Look for any labels or printing that will indicate which of the two ports close together is the primary/first port, and connect there. The "single port" at the lower section of each backplane is likely intended to connect an additional/downstream device.

In a worst-case scenario, you can likely chain the two front planes together.

 

[Zyrox]

I had issue like that in the past. One time there were 10 pcs of 4 Pin Molex power plugs and I didn't had enough power, so one cable 4-5 SAS drives. Another one had two backplanes, according to manufacturer they worked individually, but they didn't. The power was coupled to the other backplane. And the third storage case had two backplanes as well and the second one was broken, but not on the first look, after a while there came r/w errors.