Me, myself and a 24-bay build that needs (a lot of) guidance

[blaatje]

Hey everyone,

I've gotten my hands on a few things that I would like to make a FreeNAS storage server with, but I'm running into a few issues.

Hardware:
Supermicro 6016T-6RFT+ with 96GB of RAM:
Intel® Xeon® processor 5600/5500 series, with QPI up to 6.4 GT/s
288GB** DDR3 1333/ 1066/800MHz ECC Registered DIMM / 48GB Unbuffered DIMM
Left Slot (Full-height / Full-length): 2 (x8) PCI-E 2.0 slots
Right Slot (Low-Profile - 5.5" depth): 1 (x4) PCI-E 2.0 (in x8 slot)
Broadcom 2108 8-Port 6Gbps SAS Controller; RAID 0, 1, 5, 6, 10
Intel® 82599EB 10 Gigabit + 82576 Dual-Port Gigabit Ethernet Controller
4x 3.5" Hot-swap SAS/SATA Drives, For Customizable Storage
700W Gold-Level Redundant Power Supply
Chenbro RM519 case
Areca ARC-1280ML (which might be able to work according to Link)https://www.freebsd.org/cgi/man.cgi?query=arcmsr&sektion=4&manpath=FreeBSD+11.0-RELEASE+and+Ports
24x 4TB harddisks(TBD)

Because the Supermicro doesn't fit into the Chenbro case, I was contemplating putting the Raid controller in the Supermicro server and then have the SAS cables run between the 2 servers, but unfortunately the raid card has active cooling and doesn't fit in the 1U case. I could probably get my hands on a passive cooler somewhere, but then I would like to be more certain that the ARC-1280ML would work.

Another option I had in mind was to use a PCI extender cable between the two cases and put the ARC-1280ML in the Chenbro case, which would save me from having to buy longer SAS cables, but I was persuaded against this by people using a lot of sarcasm and no real explanation on why this is a terrible idea.

Third option and probably the "official option" would be to use external SAS like in this example.

I understand that this all might not be optimal but I would like to see what the downsides are of each option, because I've not been able to find any information about these cases(probably because other people aren't this idiotic).

The final option which I'm also considering is getting an extra motherboard and CPU that do fit in the Chenbro case and just steal a bunch of the memory from the supermicro (any recommendations are welcome, I am from Europe though, so what is cheap for you might be 3 times as expensive on this side of the pond).

I understand that this post might be a mess but I would ask you to bear with me, I'm trying to find all the information on my own but sometimes one just needs a bit of guidance.

Thanks for reading through this far and any help and tips are welcome!

TL;DR: Idiot got his hands on decent hardware and doesn't know what to do with it.

 

[Chris Moore]

There are some ways to extend the PCIe bus from one chassis to another, but the reliability of this might not be worth the price.
Let me do some research on the hardware you have. I will post back later today.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Chris Moore]

Areca ARC-1280ML (which might be able to work according to Link)
24x 4TB harddisks(TBD)

I couldn't follow the link you provided. It was blocked by firewall rules from where I work, but I did look that model card up and it is a caching, hardware RAID controller. Those are generally not good to use with ZFS. It would be my suggestion that you sell that on eBay and use any profit from that to invest in hardware that is known to be compatible with FreeNAS/ZFS. Just because there is a BSD driver for some hardware doesn't make it good to use in the way that FreeNAS works. FreeNAS is an apliance that was built using FreeBSD as the underlying operating system, but FreeNAS prevents access and modification of many of the underlying features because the FreeNAS application is in control. With very few exceptions, changes to the system must be done through the GUI so they can be properly recorded in the configuration database. As an example, most changes made at the command line (even edits saved in files like smb.conf) will be lost on reboot because FreeNAS configures the system according to the config.db during boot.

Intel® Xeon® processor 5600/5500 series, with QPI up to 6.4 GT/s
288GB** DDR3 1333/ 1066/800MHz ECC Registered DIMM / 48GB Unbuffered DIMM
Left Slot (Full-height / Full-length): 2 (x8) PCI-E 2.0 slots
Right Slot (Low-Profile - 5.5" depth): 1 (x4) PCI-E 2.0 (in x8 slot)
Broadcom 2108 8-Port 6Gbps SAS Controller; RAID 0, 1, 5, 6, 10
Intel® 82599EB 10 Gigabit + 82576 Dual-Port Gigabit Ethernet Controller
4x 3.5" Hot-swap SAS/SATA Drives, For Customizable Storage
700W Gold-Level Redundant Power Supply

This is an older system but it appears to be compatible with the exception that the Boadcom 2108 SAS controller may need to be flashed with a different/newer firmware to be sure it works properly. I hope someone else is more familiar with that specific hardware.
You will generate a lot of heat with this. Newer hardware would be better, if budget allowed.

Chenbro RM519 case

I got quite a variety of results when I searched for that model Chenbro case. Could you post some images of it so we can be sure what it is you are working with? Especially the back side (inside) of the drive bays, where the cables connect.

You can post photos directly to the forum.

For example:

 

[blaatje]

I couldn't follow the link you provided. It was blocked by firewall rules from where I work, but I did look that model card up and it is a caching, hardware RAID controller. Those are generally not good to use with ZFS. It would be my suggestion that you sell that on eBay and use any profit from that to invest in hardware that is known to be compatible with FreeNAS/ZFS. Just because there is a BSD driver for some hardware doesn't make it good to use in the way that FreeNAS works. FreeNAS is an apliance that was built using FreeBSD as the underlying operating system, but FreeNAS prevents access and modification of many of the underlying features because the FreeNAS application is in control. With very few exceptions, changes to the system must be done through the GUI so they can be properly recorded in the configuration database. As an example, most changes made at the command line (even edits saved in files like smb.conf) will be lost on reboot because FreeNAS configures the system according to the config.db during boot.

Yeh was thinking about doing this, only problem I have is that the backplate needs 6 SAS cables and the SAS controllers I've seen advised all seem to have 2 SAS ports(like the LSI SAS 9207-8i).

This is an older system but it appears to be compatible with the exception that the Boadcom 2108 SAS controller may need to be flashed with a different/newer firmware to be sure it works properly. I hope someone else is more familiar with that specific hardware.
You will generate a lot of heat with this. Newer hardware would be better, if budget allowed.

Will take a look at the firmware, hopefully it will work because that would possibly give me two out of 6 SAS ports needed. Heat shouldn't be much of a problem, especially since the drives will be in a different case and the supermicro case is cooled by jetcoolers(or at least they look like jet-engines).

I got quite a variety of results when I searched for that model Chenbro case. Could you post some images of it so we can be sure what it is you are working with? Especially the back side (inside) of the drive bays, where the cables connect.

You can post photos directly to the forum.

I hope these are clear enough. Thanks for your help so far!

 

[Chris Moore]

Yeh was thinking about doing this, only problem I have is that the backplate needs 6 SAS cables and the SAS controllers I've seen advised all seem to have 2 SAS ports(like the LSI SAS 9207-8i).

Simple solution to that is a SAS controller plus a SAS expander. I will provide the links to appropriate parts.

You would use this cable to connect the onboard SAS controller in the Supermicro chass is to the external SAS connector of the expander card
CableCreation External Mini SAS 26pin (SFF-8088) Male to Internal Mini SAS 36pin
https://www.ebay.com/itm/CableCreat...-Male-to-Internal-Mini-SAS-36pin/162933904074

This expander card works much like a network switch to allow all the drives to connect through the four lanes being passed by the external cable.
HP 24 BAY 3GB SAS EXPANDER CARD W/ 4x CABLE
https://www.ebay.com/itm/468406-B21...3GB-SAS-EXPANDER-CARD-W-4x-CABLE/131875011229

This is only needed to apply power to the SAS expander card because there is no system board in the Chenbro case.
4-Pin MOLEX PCIe PCI-E Express 1X to 16X 60cm USB Riser Adapter w/ SATA Power
https://www.ebay.com/itm/4-Pin-MOLE...m-USB-Riser-Adapter-w-SATA-Power/191919918770

You can use this to connect to the Chenbro case power supply and to the front pannel connectors so you can turn the power on and off more easily. It even has a couple fan connections.
SUPERMICRO JBOD POWER BOARD CSE-PTJBOD-CB2
https://www.ebay.com/itm/SUPERMICRO-JBOD-POWER-BOARD-CSE-PTJBOD-CB2/292467040069

 

[Chris Moore]

Here is an article where they did a similar thing using a junk system board to power the SAS expander:
https://www.servethehome.com/sas-expanders-diy-cheap-low-cost-jbod-enclosures-raid/

Here is one where they use the card I refferenced:
https://www.servethehome.com/supermicro-cse-ptjbod-cb1-jbod-power-board-diy-jbod-chassis-made-easy/

 

[nightshade00013]

Broadcom 2108 8-Port 6Gbps SAS Controller; RAID 0, 1, 5, 6, 10

http://www.supermicro.com/products/accessories/addon/AOC-USAS2LP-H8iR.cfm
And don't use this card. It is a problem waiting to happen. There is no firmware that can be changed out to make it a HBA.

Areca ARC-1280ML (which might be able to work according to Link)

This is another raid controller. Bad juju with software raid like ZFS.

Stick them in some windows computers and be happy they work.

As @Chris Moore said get a HBA and then put an expander into the other case if you must but honestly I would grab a board that will fit in the other case that either supports the CPU's and such that you have or get a newer board and CPU and wash your hands of the "proprietary" parts. Maybe throw a couple drives in and give it to a relative so they don't have to use your stuff.

 

[Chris Moore]

And don't use this card. It is a problem waiting to happen. There is no firmware that can be changed out to make it a HBA.

I wasn't familiar with it. Thanks for the info.

Thanks for your help so far!

If the cost of a system board is out of the question for the Chenbro, you can always put a SAS controller in the Supermicro chassis. I have been using these for several years with no problems.

https://www.ebay.com/itm/Dell-H310-...ZFS-FreeNAS-unRAID-High-Air-Flow/162834671120

You can use one of the port to run the four drives in the Supermicro chassis and the other port to run the SAS expander for the Chenbro chassis. That would allow you to use up to 28 storage drives in the system.

 

[Stux]

Wouldn’t recommend those HP SAS expanders, they’re full of issues.

 

[Stux]

Might work out cheaper finding an x8 board that will fit in the chenbro. But you’d still need/want a sas expander and probably a sas card anyway.

 

[Chris Moore]

I hope these are clear enough.

I see from the photos that the fans are missing from that Chenbro case...
Do you still have all the parts for that, like the fans?
I understand that the system board, processor and memory are gone, but do you have all the other bits?

 

[blaatje]

And don't use this card. It is a problem waiting to happen. There is no firmware that can be changed out to make it a HBA.

Ah bummer! Well guess I'll have to get myself an HBA card..

but honestly I would grab a board that will fit in the other case that either supports the CPU's and such that you have or get a newer board and CPU and wash your hands of the "proprietary" parts.

Well I can always use extra computing power, so I could go for a different board and CPU. Any recommendations that are cheap, would fit in the case and would word with the memory?

Would recommend those HP SAS expanders, they’re full of issues.

I'm guessing you mean wouldn't recommend, is there anything you would recommend in my case?

I see from the photos that the fans are missing from that Chenbro case...
Do you still have all the parts for that, like the fans?
I understand that the system board, processor and memory are gone, but do you have all the other bits?

Yes still have them, just have them stored in a box right now, the case is heavy enough without everything in it...

Thanks everyone for your replies, I really appreciate it.

 

[Chris Moore]

Any recommendations that are cheap <snip> and would word with the memory?

That is a big case, almost anything would fit. The problem is the cheap and working with the memory part.
A system board that will work with that memory is going to be on the expensive side. What kind of budget do you have?

 

[Stux]

I'm guessing you mean wouldn't recommend, is there anything you would recommend in my case?

Correct.

The intel ones are good.

https://www.intel.com.au/content/www/au/en/servers/raid/raid-controller-res2sv240.html

Note: the above does not require a pci slot to work.

24 ports means you’d only have one uplink and 20 bays and would need to use the other uplink for the last 4 bays.

There is also a 36 port version. The benefit is you can double uplink and then still have 28 bays, or even have another downlink to another chassis

 

[nightshade00013]

Agreed, you can probably fit any board that you can buy into it as long as it is standard sized. What you can find and afford is the major issue. Reusing the DDR3 RAM will limit what you are looking for so something in the x9 series supermicro along with a portion of the x10 series https://www.supermicro.com/products/motherboard/Xeon3000/#2011 just scroll down a little for the ones that support ddr3.

Some come with SAS controllers on board. As far as recommending one in particular I will leave that to someone who has one. I got lucky when I picked up an X8DT6-F as it has an onboard SAS controller. The boards are so expensive now I wouldn't be able to afford one, I think I paid about 150 for the one I am using and on ebay they are now running around 500 and up. It's odd to think that a used board actually appreciated in value.

 

[Stux]

Also, you can use reverse break out cables to connect 4 Sata ports to the Mini sas ports on your backplane, since it looks like a passive sas/Sata backplane,

Ie, if you put a motherboard in the case.

 

[blaatje]

What kind of budget do you have?

up to 1000 euro or something, but then it would need to include a motherboard, cpu('s) and a solution for connecting the 24 hdd bays.

Correct.

The intel ones are good.

https://www.intel.com.au/content/www/au/en/servers/raid/raid-controller-res2sv240.html

Note: the above does not require a pci slot to work.

24 ports means you’d only have one uplink and 20 bays and would need to use the other uplink for the last 4 bays.

There is also a 36 port version. The benefit is you can double uplink and then still have 28 bays, or even have another downlink to another chassis

Also, you can use reverse break out cables to connect 4 Sata ports to the Mini sas ports on your backplane, since it looks like a passive sas/Sata backplane

So I would be able to make it work by buying these components(?):
- A 24 port Intel sas expander
- A SAS SATA 9211-8i
- A Supermicro CSE-PTJBOD-CB2

Some come with SAS controllers on board. As far as recommending one in particular I will leave that to someone who has one. I got lucky when I picked up an X8DT6-F as it has an onboard SAS controller. The boards are so expensive now I wouldn't be able to afford one, I think I paid about 150 for the one I am using and on ebay they are now running around 500 and up. It's odd to think that a used board actually appreciated in value.

I think I see them for 100 dollars on ebay right now(refurbished)? Not that that is a real solution for me, unless i decide to buy everything in the US, but would like to stick to Europe if possible..

Thanks again guys, without everyone helping me I would've probably lost a bunch of money or data by making stupid mistakes!

 

[Chris Moore]

So I would be able to make it work by buying these components(?):
- A 24 port Intel sas expander
- A SAS SATA 9211-8i
- A Supermicro CSE-PTJBOD-CB2

There are a few cables and adapters that you would need, but that gets you the big things.
Did you want to look into that in more detail, or did you want to look at getting a system board, processor and such for the Chenbro case?

 

[blaatje]

Did you want to look into that in more detail, or did you want to look at getting a system board, processor and such for the Chenbro case?

Trying to look at both avenues, but I think it would be best most optimal if I could fit everything in the Chenbro case plus seeing that not going with a new motherboard/cpu would already cost me 400 euro's.. It wouldn't be that much of a stretch to look at other hardware as well. I can always use the other supermicro for other stuff(with a lot less memory).

 

[Chris Moore]

Trying to look at both avenues, but I think it would be best most optimal if I could fit everything in the Chenbro case plus seeing that not going with a new motherboard/cpu would already cost me 400 euro's.. It wouldn't be that much of a stretch to look at other hardware as well. I can always use the other supermicro for other stuff(with a lot less memory).

This is a good system board that isn't horribly expensive:
https://www.ebay.com/itm/NEW-Unopened-Super-Micro-Motherboard-LGA-2011-MBD-X9SRL-F-O/253300204890

Here is an economical CPU for that board and it has enough resources to do good things:
https://www.ebay.com/itm/Intel-Xeon...8-Core-LGA-2011-CPU-Processor-km/401506559263

Here is a CPU cooler that should fit in that chassis, it only has 3U over the system board:
https://www.ebay.com/itm/Dynatron-R...-Intel-Socket-LGA2011-Narrow-ILM/401284811045

You should be able to use the existing RAM, as much as will fit, until you can upgrade, if you decide you need more memory.

The SAS controller I pointed out earlier is still a good option. You can either use three of them or you can get a couple of SAS expander cards like this:
https://www.ebay.com/itm/IBM-46M0997-ServeRAID-Expansion-Adapter-16-Port-SAS-Expander/122281398896

I noticed that some of the existing SAS cables in your chassis are angled, you may be able to use some of those, but you may need to get some additional ones with straight ends to make these connections. Here is what I have used:
https://www.ebay.com/itm/Internal-M...in-SFF-8087-SHD-Raid-Data-Cable-/253334976071

Do you have any questions about how to connect these things?
We have a resource here that may help:

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/