SOLVED Did I just max out my HBA? 23 drives will boot, 24 will not.

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
Server: SuperMicro CSE-846 24 Bay SAS2 BP Server w/ X9DRi-F/2x 6 Core E5-2620
HBA: HP H220 6Gbps SAS PCI-E 3.0 HBA LSI 9205-8i P20 IT Mode https://www.ebay.com/itm/HP-H220-6G...9205-8i-P20-IT-Mode-From-US-Ship/192639052923
MPT2BIOS-7.25.05.00

Up until this point I've run 16 drives on this 24 bay server. Recently I decided to buy 8 more drives, so that I can move pools, then re-arrange some drives.

When I run the new 8 drives for a total of 24 slots the server gets stuck on the HP200 Series step and just shows"Initializing.." indefinitely without HDD activity lights. If I power-off, remove the new drives and only run the original 16, then the server boots properly.

Configuration that boots:
8x4tb
8x2tb

Configuration that will not boot:
16x4tb
8x2tb

Did I hit a limit of the HBA and it is time to upgrade it? Maybe the backplane?

EDIT: I just tried booting with 4 out of 8 new drives. It didn't get stuck on initializing. Making it 20 total drives and it booted to TrueNAS.

EDIT2: Moving drives around and testing with different combinations it will boot with up to 23 of 24 drives. Running a 24th drive (any of the new drives, so not isolated to a single one) in any slot causes that stuck behavior!

EDIT3: Replacing the backplane with another one resolved the issue. All 24 drives are recognized.
 
Last edited:

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I tried leaving different slots open on the backplane as well. Didn't matter which slots used it could initialize 23 drives, but not 24.
Right, but it shouldn't be the HBA chip that's the limiting factor. It's not impossible that the HP implementation of it is somehow adding a limit, but I think you could probably look at other factors like backplane first.

Maybe think about power supply... too much load on one rail? or overall?
 
Last edited:

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
I'm on supermicro 1200w psu's. Sounds like HDD can eat around 30W, so talking 720watt if all hdds are running at 30w. This server is dual socket, so maybe that's not quite enough. Seems possible.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
I'm on supermicro 1200w psu's. Sounds like HDD can eat around 30W, so talking 720watt if all hdds are running at 30w. This server is dual socket, so maybe that's not quite enough. Seems possible.
Check how your power is connected to the backplane... is it all the one rail or are you mixing rails? You may have enough, but you're not exposing all the capacity from all the rails available to the backplane.

Maybe one of the guys with more hardware experience with Supermicro chassis can chime in on that.
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
Check how your power is connected to the backplane... is it all the one rail or are you mixing rails? You may have enough, but you're not exposing all the capacity from all the rails available to the backplane.

Maybe one of the guys with more hardware experience with Supermicro chassis can chime in on that.
I will take a look and upload a photo. I'm not familiar with the term rails, but there are multiple molex plugs for power on the backplane and all are connected.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702

no_connection

Patron
Joined
Dec 15, 2013
Messages
480
Could it be something with staggered spin-up where it deadlocks itself ether waiting for a signal to continue or just freaks out?
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
Could it be something with staggered spin-up where it deadlocks itself ether waiting for a signal to continue or just freaks out?
There's some logic to that... staggered spin up may also be causing you to reach some kind of timeout value with the 24th disk.

You could try it with that setting off...
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
Could it be something with staggered spin-up where it deadlocks itself ether waiting for a signal to continue or just freaks out?
Unfortunately staggered spin-up didn't change the behavior. Both enabled and disabled had the same outcome. Was worth a shot though!
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
Check how your power is connected to the backplane... is it all the one rail or are you mixing rails? You may have enough, but you're not exposing all the capacity from all the rails available to the backplane.

Maybe one of the guys with more hardware experience with Supermicro chassis can chime in on that.
1614642458989.png


It looks ok to me, there are two different runs for power.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
When I run the new 8 drives for a total of 24 slots the server gets stuck on the HP200 Series step and just shows"Initializing.." indefinitely without HDD activity lights. If I power-off, remove the new drives and only run the original 16, then the server boots properly.

Not positive that this is your problem, but it sometimes happens that a card or mainboard BIOS was written in such a way that it craps out at or after Z:

C: + 24 is Z:

Quick way to test this is to go into the mainboard BIOS and disable the option ROM load on the card.
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
It looks ok to me, there are two different runs for power.
Although it does seem that one of the rails is covering 2 "sections" and the other covers 4... any chance the last disk you're adding is on the rail covering 4?
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
Although it does seem that one of the rails is covering 2 "sections" and the other covers 4... any chance the last disk you're adding is on the rail covering 4?
Yes, it is on the top right of the backplane.
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
1614816512842.png


I moved things around and was able to unziptie a molex and run additional. There are now 4 runs to the backplane. Should be plenty I think.

Also I moved all the drives around, placing the 2.5" 2TB drives at the top of the backplane in case they would need less power.

Same behavior. 24 drives is a no go. I'm going to look into the ROM option that @jgreco mentioned and dig around the BIOS.
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
1614817087482.png


Now I don't know what to think. Apparently this might be positional. During my test of re-arranging all the drives. I moved the existing pool drives to the top. 2.5", then 3.5" on lower sections. Even after removing the 8 new drives, this will not initialize using 16 drives when in this order.
 

CarlB

Dabbler
Joined
Jan 30, 2018
Messages
40
Just wanted to revisit this now that I have some more time to troubleshoot. After playing the drive shuffle game I have found that there are some particular spots that cause the issue.

I'm not really sure what component to blame here. Would this be the backplane?

Example of the configurations that won't work and does work. I tested with staggered spin-up on and off, no change in behavior.

1620866346713.png
 
Last edited:
Top