BUILD My Freenas can’t boot if using >11 hard drives.

[Mex]

Hi guys,
I’m reusing my old mobo Asus P5K-E for my freenas file storage. It’s been working fine except every time it reboots I have to take 1 Hdd out and wait for booting progress then reinsert that hdd in (because the boot drive can only be recognized in bios when using 11drives). My case supermicro 846 EL has 24 slots but so far I can only use 12 due to this limitation. Is there anyway I can solve this issue? Is there a way to make it boot when using >11hdds? Must I upgrade? I have a supermicro X9DAI-o with dual Xeon 2560 V2 & 64GB to be used for Freenas but only if I must upgrade. Is this configuration any good? My hdd controller is Adaptec 24 ports use as JBOD pass through (I know LSI HBA is better but I try to reuse as much as I can). Please advise. Thank you all.

My current configuration:
- Mobo: Asus P5K-E bios 2.58
- CPU: intel core 2 Quad Q6600 2.4GHz
- RAM: 8153MB (maxed out I know it’s freenas minimum...)
- Hdd controller: ARC-1880 24hdd
- Case: supermicro 846EL
- Freenas 11.1 - U5
- PSU: Supermicro 1200W

 

[Jailer]

What do you have for a power supply?

My hdd controller is Adaptec 24 ports use as JBOD pass through

Really bad idea.

 

[Mex]

What do you have for a power supply?


Really bad idea.

Jailer, I use Supermicro 1200W PSU. I’m aware that arc-1880 is bad for freenas so I’m thinking of changing it to LSI 9305-24i. Is it good?

 

[Chris Moore]

I’m aware that arc-1880 is bad for freenas

Certainly get that out of the system, it may not be the entire and sole source of the problem, but it can't help causing trouble.

so I’m thinking of changing it to LSI 9305-24i. Is it good?

This is a perfectly good controller:
https://www.ebay.com/itm/NEW-HP-H22...-Adapter-card-2PCS-SFF-8087-SATA/192770232992
There is no need for a 9305 because you are not going to see any benifit from the 12Gb SAS. It is just a waste of money.
If you need more ports than the 8 on the card I suggested, simply use a SAS expander. If you need guidance, please ask.

Something I want to point out, your thread title:

Freenas can’t boot if using >11 hard drives.

That is completely not true of a properly functioning FreeNAS system. My primary FreeNAS at home has THIRTY drives in it at the moment and one of the systems I manage at work has 124 drives. If there is a limit to the number of drives that FreeNAS is able to handle, it is more (many more) than 11.
Your system is not working properly due to some hardware problem. That said, please give us as many details about your hardware as you can. Here is some guidance to help you with what we might need to know to help you find your problem:

Updated Forum Rules 12/5/18
https://forums.freenas.org/index.php?threads/updated-forum-rules-12-5-18.45124/

 

[Mex]

Certainly get that out of the system, it may not be the entire and sole source of the problem, but it can't help causing trouble.

This is a perfectly good controller:
https://www.ebay.com/itm/NEW-HP-H22...-Adapter-card-2PCS-SFF-8087-SATA/192770232992
There is no need for a 9305 because you are not going to see any benifit from the 12Gb SAS. It is just a waste of money.
If you need more ports than the 8 on the card I suggested, simply use a SAS expander. If you need guidance, please ask.

Something I want to point out, your thread title:

That is completely not true of a properly functioning FreeNAS system. My primary FreeNAS at home has THIRTY drives in it at the moment and one of the systems I manage at work has 124 drives. If there is a limit to the number of drives that FreeNAS is able to handle, it is more (many more) than 11.
Your system is not working properly due to some hardware problem. That said, please give us as many details about your hardware as you can. Here is some guidance to help you with what we might need to know to help you find your problem:

Updated Forum Rules 12/5/18
https://forums.freenas.org/index.php?threads/updated-forum-rules-12-5-18.45124/

Chris, I will eventually switch that raid card. I need a 24 ports with less PCI. I eventually upgrade parts so I intend to upgrade to LSI 9305-24i then mobo+cpu+ ram... What should I do to take advantage of 12Gb/s or just can’t at all?

My case can handle 24hdds so is it a waste using sas expander? I updated my original post with configuration. I meant that my Freenas can’t handle >11 hdd.

My current configuration:
- Mobo: Asus P5K-E bios 2.58
- CPU: intel core 2 Quad Q6600 2.4GHz
- RAM: 8153MB (maxed out I know it’s freenas minimum...)
- Hdd controller: ARC-1880 24hdd
- Case: supermicro 846EL
- Freenas 11.1 - U5
- PSU: Supermicro 1200W

 

[CraigD]

Q6600 was a great chip 12 years ago, an ok chip 5 years ago, today it is good for web surfing

The high power usage, front side bus, and RAM limitations make it a poor choice for a FreeNAS server

I have a supermicro X9DAI-o with dual Xeon 2560 V2 & 64GB to be used for Freenas but only if I must upgrade. Is this configuration any good?

Nice FreeNAS server! (You mean E5 2650 v2?)

You can pick up a M1015 HBA really cheap, buy one


Have Fun

 

[Chris Moore]

- Mobo: Asus P5K-E bios 2.58
- CPU: intel core 2 Quad Q6600 2.4GHz
- RAM: 8153MB (maxed out I know it’s freenas minimum...)
- Hdd controller: ARC-1880 24hdd

Did you ever make any changes here to get the system working better?

Hardware Requirements
http://www.freenas.org/hardware-requirements/

Hardware Recommendations Guide Rev. 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Hardware Recommendations by @cyberjock - from 26 Aug 2014 - and still valid
https://forums.freenas.org/threads/hardware-recommendations-read-this-first.23069/

 

[sretalla]

I’m thinking of changing it to LSI 9305-24i. Is it good?

You can see in my sig that I run one of those in my primary system (picked it up cheap as an opened, but not used unit).

I confirm with almost a year connected to 128TB, 24 HDDs of storage, it runs well for me. (even if it is overkill)

 

[Mex]

You can see in my sig that I run one of those in my primary system (picked it up cheap as an opened, but not used unit).

I confirm with almost a year connected to 128TB, 24 HDDs of storage, it runs well for me. (even if it is overkill)

Chris & sretalla, I think the first component I should change is the raid card to LSI 9305-24i. In the future, I will eventually switch my current workstation of E5-2650 V2 to better one and use this old one to be freenas server. I need a faster storage for heavy media access in the end. What do you think?

 

[Chris Moore]

Chris & sretalla, I think the first component I should change is the raid card to LSI 9305-24i. In the future, I will eventually switch my current workstation of E5-2650 V2 to better one and use this old one to be freenas server.

There is no reason to buy such an expensive controller. Take a look at this video:

Explaining the IBM SAS-2 expander and how to do 24xHDD setup with only 2-port SAS controller
https://www.youtube.com/watch?v=qccpopxc_Uo

I need a faster storage for heavy media access in the end. What do you think?

While a newer system board is certainly a good idea. Mechanical drives can't take advantage of 12Gb SAS. A controller like that is wasted on a home system with only a few drives.

 

[Mex]

There is no reason to buy such an expensive controller. Take a look at this video:

Explaining the IBM SAS-2 expander and how to do 24xHDD setup with only 2-port SAS controller
https://www.youtube.com/watch?v=qccpopxc_Uo


While a newer system board is certainly a good idea. Mechanical drives can't take advantage of 12Gb SAS. A controller like that is wasted on a home system with only a few drives.

Thank you Chris. I'm upgrading my storage for home business. I do heavy graphics so I need fast and lots of storage. I agree currently 12Gb SAS is a waste but I think it will be good for the future. If I do upgrade to the IBM SAS-2, can I use just the one with 24 ports instead of having another of 8 port as in the video (my case has 24 ports direct backplane)? Also do I have to upgrade my backplane to SAS2 as well? I saw supermicro has a case with SAS3 backplane. I think SAS3 is coming.

 

[Chris Moore]

SAS3 is already here. What I am trying to tell you is that mechanical disk drives have no need for that speed, so it's a waste of money. The fastest mechanical drives are capable of just about 250MB/s where SAS3 is capable of 1200MB/s per lane. But mechanical drives rarely sustain that transfer rate because they are busy seeking for the data. I have not seen mechanical drives sustain transfer rates above 120MB/s so, you would need around 80 drives attached to the controller to fully utilize the capacity. Even with SAS2, you would need over 40 drives to saturate the controller.
How many drives are you planning on having?

Edit: I forgot to address this question:

If I do upgrade to the IBM SAS-2, can I use just the one with 24 ports instead of having another of 8 port as in the video (my case has 24 ports direct backplane)?

You could use a SAS Expander like this one:
https://www.ebay.com/itm/RES2CV360-INTEL-36-PORT-6G-SAS-EXPANDER-BOARD-FOR-INTEL-R1000-R2000-TYPE-SERVERS/132665195104
It would allow you to use two cables like this to connect back to the SAS Controller:
https://www.ebay.com/itm/2Pcs-Mini-SAS-36Pin-SFF-8087-to-36Pin-SFF-8087-SAS-Data-Internal-Server-Cable-1M/202094506489
You could use the rest of the ports on the expander to connect forward to the direct attach backplane, which does not care what the attached drives are, so it doesn't need to be changed.

I saw supermicro has a case with SAS3 backplane. I think SAS3 is coming.

I have a chassis at work that is running a SAS3 controller with two 30 bay expander backplanes, and it has 60 drives installed. There would be a potential bottleneck in accessing that many drives on a single SAS2 controller, so for this quantity of drives it makes some sense. It just depends on your use case.

 

[Mex]

...How many drives are you planning on having?

I see. I’m filling up 24HDDs for now. Just to make sure, should this one alone be good for my case? https://www.ebay.com/itm/IBM-ServeR...on-Adapter-46M0997-Firmware-634A/163321588238 then I plug directly to my current supermicro SC846TQ backplane via SFF8087?

You could use a SAS Expander like this one:
https://www.ebay.com/itm/RES2CV360-INTEL-36-PORT-6G-SAS-EXPANDER-BOARD-FOR-INTEL-R1000-R2000-TYPE-SERVERS/132665195104
It would allow you to use two cables like this to connect back to the SAS Controller:
https://www.ebay.com/itm/2Pcs-Mini-SAS-36Pin-SFF-8087-to-36Pin-SFF-8087-SAS-Data-Internal-Server-Cable-1M/202094506489
You could use the rest of the ports on the expander to connect forward to the direct attach backplane, which does not care what the attached drives are, so it doesn't need to be changed.

I don't quite understand this one. Where should I plug it to since it's not PCI express?

 

[Chris Moore]

g it to since it's not PCI express

on all the SAS expanders they get the data through the SAS cable not to the PCI Express interface. PCI Express is just there for the power

 

[pro lamer]

SAS Expander like this one:
https://www.ebay.com/itm/RES2CV360-INTEL-36-PORT-6G-SAS-EXPANDER-BOARD-FOR-INTEL-R1000-R2000-TYPE-SERVERS/132665195104

To me it resolves to RES2CV360 INTEL 36-PORT 6G SAS EXPANDER BOARD FOR INTEL R1000/R2000 TYPE SERVERS which seems to be a bare PCB... It has some mounting holes similar to motherboard mounting holes... Where to attach it? Or just tie the cables nicely and this way make the cables secure the PCB? The PCB seems lightweight so it seems ok in terms of preventing dangling...

Sent from my phone

 

[Chris Moore]

To me it resolves to RES2CV360 INTEL 36-PORT 6G SAS EXPANDER BOARD FOR INTEL R1000/R2000 TYPE SERVERS which seems to be a bare PCB... It has some mounting holes similar to motherboard mounting holes... Where to attach it? Or just tie the cables nicely and this way make the cables secure the PCB? The PCB seems lightweight so it seems ok in terms of preventing dangling...

Sent from my phone

Yes, there are options, but take a look at the photo in this post to get one possibility:
https://www.ixsystems.com/community...ro-x10-sri-f-xeon-e5-1650v4.46262/post-409299
There is some discussion around that. Basically, that card just needs power and SAS connections, AND some airflow over that heat-sync to keep it cool.
Some people have even added a fan to make sure:



I have a server chassis at work that uses two of these to run 60 drives, 30 drives on each expander, and each expander has a link back to a SAS controller. I manage several servers that have 60 drives, and this one was originally using HighPoint Rocket 750 cards but I changed it over to SAS controllers instead.

@Stux , did you ever take photos of your SAS expander during / after the installation? I don't recall seeing any more photos of it.

 

[Mex]

Yes, there are options, but take a look at the photo in this post to get one possibility:

My case is like this. It curently has ATX. If I upgrade to my supermicro X9DAI-O which is EATX I don’t think I have space for the intel board. I guess LSI sas2 controller will be good.

 

[Ericloewe]

There's a popular model that can fit in an expansion slot, optionally being powered from a PCIe slot.