Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SOLVED 11 HDDs is the limit!

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

{HD}

Member
Joined
Oct 30, 2016
Messages
47
Solution: Will be written here once I reboot into the HBA Setup Utility to see what the settings were that I changed to make it work. See the other posts, there are a few different solutions.

IRC UName: {HD}

MOBO: SuperMicro X8DTE-F (with up to date bios and ipmi firmware)
Backplane: SuperMicro SAS-846EL-1
PSU x 2: SuperMicro PWS-902-1R Redundant 900W
CPU x 2: X5650 6-core 2.66Ghz
96GB ECC DDR3
HBA: LSI9211-8i(IT)

Boot Pool: Mirrored SanDisk 16GB Cruzer Fits
OS: FreeNAS 9.10.1-U2

6 x 2TB WD REDS in Z2
1 x 2TB WD RED as spare

Ok...Starting there everything works as expected. I have everything setup the way I like it.

The issue came around when I try to reboot after adding 6 x 4TB WD Reds to the box.

Adding the drives went fine, no issues. I made a separate Z2 pool to test how it would work and what kind of additional storage capacity I will achieve. Then I went to reboot and got an error "Reboot and select proper Boot device or Insert Boot Media in selected Boot device and press a key". When I saw this I double checked the BIOS boot order and it looked as it should. I then tried removing the BOOT USBs and tried to boot from the INSTALL USB and it did not work. So, I unpluged all 6 4TB drives I had just added and it booted into the INSTALL USB with no issues...I quickly rebooted to the BOOT USBs and it worked as well. Then I added HDDs one at a time rebooting between each one. When it finally failed(after adding 4 HDDs) I tired several different Backplane slots and/or hard drives with no different results.

The BIOS and HBA config look good.

I can run with all drives once I am booted into system. But, in order to reboot or startup I need to undock 2 HDDs...Very Inconvenient...

If you need any additional data I will be happy to augment my original post(specs) just let me know what you need.

PS: Should this be in hardware? Or is it in the correct location in storage?
 
Last edited:

nojohnny101

Neophyte Sage
Joined
Dec 3, 2015
Messages
1,474
What brand and size is your PSU?
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
The box has 2 900W SuperMicro PWS-902-1R
*The original post has been updated to reflect this information
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
2016.10.31 Update: I have the machine running with all 13 drives installed and dedicated to a pool. I was going to make the additional 6 drives a vdev of my main pool but since I cannot figure out what the problem with booting is I am going to leave it separate(for now).
I officially cannot reboot without removing drives so only 11 are connected during startup. But, once it boots and is running I can add the remaining drives to un-degrade my pool and everything works as expected.
I would like to solve this inconvenience because my goal was to add another 6 drives in the next month or so.
I am willing to try any diagnostic steps (as long as it doesn't destroy my data on the first pool)

I have tried:
A different SAS cable from HBA to Backplane
Different Backplane slots
Removing different Hard Drives on boot
Different USB ports for boot drives
Switched from MirrorBoot1 to MirrorBoot2
Tried other boot media

Nothing I have tried has led me to the cause of the problem: If Backplane > 11 drives then USB Boot Disable Else Boot Normally...

I have not tried to boot to media installed into backplane. I think I have a small SATA Drive I could attempt to install freenas onto...If you think that is worth a shot let me know.
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862
Then I went to reboot and got an error "Reboot and select proper Boot device or Insert Boot Media in selected Boot device and press a key".
This sounds like a boot device selection error. I would try disabling all HBA drives so they aren't visible during the BIOS bootup (enter the HBA setup screen during BIOS boot).
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
This sounds like a boot device selection error. I would try disabling all HBA drives so they aren't visible during the BIOS bootup (enter the HBA setup screen during BIOS boot).
I have essentially he done that by unplugging the Sas cable from the HBA during start up. When I did that I was able to boot from the USB devices. I then went further by plugging my backplane back in. Then one by one I added the drives until I received the "select proper boot device "error. I receive that error between adding the 11th and 12th hard drive. Are there some settings in the HBA I should document here?
 

depasseg

FreeNAS Replicant
Joined
Sep 16, 2014
Messages
2,862
you can either
1. Disable any of the disks attached to the hba in BIOS as possible boot devices (check out the boot options in bios)
2. disable the HBA boot function by erasing the firmware and not installing the boot rom. (something like: sas2flsh.exe -o -e 6, sas2flsh.exe -o -f 2118it.bin (leaving off the -b mpt2sas.rom option removes the ROM based HBA BIOS. Highly desirable for quick boots on SW raid systems in IT mode)).
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
1. Disable any of the disks attached to the hba in BIOS as possible boot devices (check out the boot options in bios)
I have done this. All boot options are disabled with the exception of USB.

2. disable the HBA boot function by erasing the firmware and not installing the boot rom. (something like: sas2flsh.exe -o -e 6, sas2flsh.exe -o -f 2118it.bin (leaving off the -b mpt2sas.rom option removes the ROM based HBA BIOS. Highly desirable for quick boots on SW raid systems in IT mode)).
I will definitely try this. I have two extra 9211–8i sitting here. I will post a status update when I get back to the site and have an opportunity to replace it.
 

philhu

Senior Member
Joined
May 17, 2016
Messages
258
Actually had same problem. I assume you are using a super micro box.

The list of allowed boot devices gets full or corrup. After 11 devices, the normal boot device scrolls off the list. Go to the boot dev ices screen, and remove the devices you will never boot, ie, the data disks, reboot into bios, go to same screen agan, your ssd/boot usb should show again

I have a full boat sc847 chassis with a dtn8f+ morther board and am running 3 drives on the motherboard ports and 36 12x4tb and 24x6 tb drives as 3 vdevs.
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
Actually had same problem. I assume you are using a super micro box.

The list of allowed boot devices gets full or corrup. After 11 devices, the normal boot device scrolls off the list. Go to the boot dev ices screen, and remove the devices you will never boot, ie, the data disks, reboot into bios, go to same screen agan, your ssd/boot usb should show again
I am indeed using a SuperMicro Box.
I've listed my BIOS 'Boot' Menu....I already had disabled everything but USB.

Code:
BIOS SETUP UTILITY

Main Advanced Security (Boot) Exit

Boot Device Priority
-1st Boot Device = USB: SanDisk Cruzer
-2nd Boot Device = Disabled
....
-6th Boot Device = Disabled

Hard Disk Drives
-1st Drive = Disabled

Removable Drives
-1st Drive = Not Installed

CD/DVD Drives
-1st Drive = Not Installed

USB Drives
-1st Drive = USB:SanDisk Cruzer Fit
-2nd Drive = USB:SanDisk

Network Drives
-1st Drive = Disabled

Other Drives
-1st Drive = Not Installed

Retry Boot Device = Disabled
 

Chris Moore

Wizened Sage
Joined
May 2, 2015
Messages
10,062
Solution: Will be written here once I reboot into the HBA Setup Utility to see what the settings were that I changed to make it work.

IRC UName: {HD}

MOBO: SuperMicro X8DTE-F (with up to date bios and ipmi firmware)
Backplane: SuperMicro SAS-846EL-1
PSU x 2: SuperMicro PWS-902-1R Redundant 900W
CPU x 2: X5650 6-core 2.66Ghz
96GB ECC DDR3
HBA: LSI9211-8i(IT)

Boot Pool: Mirrored SanDisk 16GB Cruzer Fits
OS: FreeNAS 9.10.1-U2

6 x 2TB WD REDS in Z2
1 x 2TB WD RED as spare

Ok...Starting there everything works as expected. I have everything setup the way I like it.

The issue came around when I try to reboot after adding 6 x 4TB WD Reds to the box.

Adding the drives went fine, no issues. I made a separate Z2 pool to test how it would work and what kind of additional storage capacity I will achieve. Then I went to reboot and got an error "Reboot and select proper Boot device or Insert Boot Media in selected Boot device and press a key". When I saw this I double checked the BIOS boot order and it looked as it should. I then tried removing the BOOT USBs and tried to boot from the INSTALL USB and it did not work. So, I unpluged all 6 4TB drives I had just added and it booted into the INSTALL USB with no issues...I quickly rebooted to the BOOT USBs and it worked as well. Then I added HDDs one at a time rebooting between each one. When it finally failed(after adding 4 HDDs) I tired several different Backplane slots and/or hard drives with no different results.

The BIOS and HBA config look good.

I can run with all drives once I am booted into system. But, in order to reboot or startup I need to undock 2 HDDs...Very Inconvenient...

If you need any additional data I will be happy to augment my original post(specs) just let me know what you need.

PS: Should this be in hardware? Or is it in the correct location in storage?
I bet that this is a SAS-1 Backplane and that it doesn't properly support drives over 2TB.

I have spent all the time looking for the answer that I can tonight, I have to get some sleep.

Would someone research that ( Backplane: SuperMicro SAS-846EL-1 ) and see if you can find out?
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
I bet that this is a SAS-1 Backplane and that it doesn't properly support drives over 2TB.
You are correct in thinking it is a SAS1 backplane. However it is my understanding that the backplane only limits the xfer speeds (3Gbit vs 6Gbit) and that the HBA is what determines max HDD size. I have the 4TB drives being recognized and working. But, correct me if I am wrong.

and BTW I marked it as solved because I am able to boot after tinkering in some settings in the HBA config. But, thanks for taking a look and trying to help.

Solution: Will be written here once I reboot into the HBA Setup Utility to see what the settings were that I changed to make it work.
 

Chris Moore

Wizened Sage
Joined
May 2, 2015
Messages
10,062
You are correct in thinking it is a SAS1 backplane. However it is my understanding that the backplane only limits the xfer speeds (3Gbit vs 6Gbit) and that the HBA is what determines max HDD size. I have the 4TB drives being recognized and working. But, correct me if I am wrong.

and BTW I marked it as solved because I am able to boot after tinkering in some settings in the HBA config. But, thanks for taking a look and trying to help.
If it is booting, that is great and if you are able to use all the drives, even better.

I was under the impression, from the reading I have done, that the SAS1 backplane would create some problems if you tried to fill it all the way up with drives larger than 2TB. I am only using 2TB drives, so I have no experience with the potential problems.

The SAS2 backplane, that is a direct 1-for-1 replacement, doesn't create any problems.
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
...if you tried to fill it all the way up with drives larger than 2TB.
It will be a few more months before I fill the last 11 slots with drives. If I run into a problems Ill track down a backplane upgrade. I went SAS1 because it was much cheaper and the 3Gb limit is fine for my backup.
 

digity

Member
Joined
Apr 24, 2016
Messages
126
I just had the same problem too. I solved it by going into the LSI 9211-8i config utility and setting the adapter's Boot Support option to OS only (originally set to BIOS & OS). I can now boot from USB with 12 or more HDDs and/or SSDs installed.

Here's my setup:
Norco RPC-4224 24 bay 3.5"/2.5" HDD/SSD drive bay 4U rackmount server case
Intel RES2SV240NC SAS Expander
LSI 9211-8i controller (in IT mode)
Supermicro X8DTL-i dual LGA1366 mobo
 

Chris Moore

Wizened Sage
Joined
May 2, 2015
Messages
10,062
You just take the BIOS out of the card. I have no BIOS in any of my LSI cards. All you need is the firmware, the BIOS is only if you're going to boot from it.

Sent from my SAMSUNG-SGH-I537 using Tapatalk
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
Interesting,
You just take the BIOS out of the card
I have an few extra LSI 9211-81's laying around I will try to remove the bios... Then swap it out this weekend when I have to open the box anyway for a degraded boot drive.
 

Chris Moore

Wizened Sage
Joined
May 2, 2015
Messages
10,062
Interesting,

I have an few extra LSI 9211-81's laying around I will try to remove the bios... Then swap it out this weekend when I have to open the box anyway for a degraded boot drive.
It has been a couple years since I dealt with flashing the firmware on the cards I am using, but if I recall, you have to erase all (boot BIOS and firmware) and then just flash the firmware. It doesn't let you do a selective erase. I tried it both ways on my system when I initially set it up and found that it booted faster without the boot BIOS because it wasn't trying to check the drives before the boot of the system continued.
I use the built-in SATA on the system board for my boot drives and run all my data drives from the SAS controller. One system has 12 data drives and the other has 16.
 

{HD}

Member
Joined
Oct 30, 2016
Messages
47
Alright, I have a card flashing with no bios. I hope it doesn't brick my card, I did read it might.

I am taking the opportunity to upgrade to U1 while I replace the boot drive and HBA. Nothing like a good scientific process of changing tons of variables at once. WCGW?
 

Chris Moore

Wizened Sage
Joined
May 2, 2015
Messages
10,062
Alright, I have a card flashing with no bios. I hope it doesn't brick my card, I did read it might.

I am taking the opportunity to upgrade to U1 while I replace the boot drive and HBA. Nothing like a good scientific process of changing tons of variables at once. WCGW?
If it makes you feel better, I was afraid that I had bricked my card the first time I flashed one but I just started over with the erase process and cleared it out, flashed the firmware and it all worked out.
I have flashed six of them now and it is a smooth operation at this point, but I have not done it in so long now that I would have to find my directions again.
 
Status
Not open for further replies.
Top