Chassis/case upgrade for 24 x 3.5" SATA disks, using LSI 9201 type HBAs

[esamett]

I first did my build on the cheap but now want to upgrade cases to get rid of temperature warnings. I am running 11 x 4 TB and 11x 8TB SATA drives and two small SSD SATA boot drives. I have tried to do my research but am getting stuck on the tech. I found a couple server cases on ebay but I didn't see enough SAS cable plugs in the back (one plug for each 4 drives) suggesting an expander which I can't use for SATA drives. Hardware guide says I need a plain SAS backplane.

I found this item which I THINK may handle 12 of my drives and my motherboard:

XYRATEX IBM HS-1235T 2U 12 Bay Storage Server Barebone Chassis Case | eBay

Find many great new & used options and get the best deals for XYRATEX IBM HS-1235T 2U 12 Bay Storage Server Barebone Chassis Case at the best online prices at eBay! Free shipping for many products!

www.ebay.com

I would appreciate explanations, recommendations and links.

Thanks in advance.
/e

 

[jgreco]

Please feel free to proceed over to the SAS Primer.

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop topologies and this kind of sucked. SATA and Serial...

www.ixsystems.com

If you get a Supermicro "-A" style backplane, there is no expander and you will need to provide SAS/SATA connectivity for each individual drive. This is not great. You can stick in a discrete SAS expander but the cabling gets sorta ugly. For 24 drives this would be three 8-channel controllers (LSI 9211-8i or similar).

If you get a Supermicro "-TQ" style backplane, all the SAS are brought out to individual single-lane SAS connectors and the wiring is a nightmare. All the caveats of the -A backplane too.

If you get a Supermicro "-E16" (or "-E26") style backplane, this includes an SAS expander and you only need a single SFF-8087 HBA and cable to hook up your drives. This is great for people who aren't familiar with SAS and SATA. Your SATA drives will work in SAS slots.

 

[jgreco]

Also be aware that a server chassis is noisy. Do not follow "advice" to replace the fans with cheap gamer "quiet fans" (including Noctua). The industrial grade fans are there to do a brutal and difficult job, forcing air through millimeter-sized gaps between drives and chassis.

 

[Constantin]

We are in violent agreement... it's also why I put my server into a "gamer" enclosure with ~1cm gaps between the drives. Noctua fans quietly keep the drives within a few degrees of ambient temperature. I may like the Q26 Lian Li enclosure best but the thing is sadly not able to handle my current Flex ATX-sized board.

My only "industrial" issue is the LSI HBA chip on the motherboard that I cannot get below 40*C during scrubs even with active on-chip cooling. I have yet to find a larger passive copper heat sink that would fit the HBA chip on my motherboard. However, those chips are legendary re: heat so I guess I'm treating it nicely.

 

[jgreco]

We are in violent agreement... it's also why I put my server into a "gamer" enclosure with ~1cm gaps between the drives. Noctua fans quietly keep the drives within a few degrees of ambient temperature. I may like the Q26 Lian Li enclosure best but the thing is sadly not able to handle my current Flex ATX-sized board.

My only "industrial" issue is the LSI HBA chip on the motherboard that I cannot get below 40*C during scrubs even with active on-chip cooling. I have yet to find a larger passive copper heat sink that would fit the HBA chip on my motherboard. However, those chips are legendary re: heat so I guess I'm treating it nicely.

This sort of cooling strategy is actually quite a bit more efficient, quiet, etc., than using a server chassis, but of course you get to figure out all the hard parts yourself.

Every time some manufacturer creates a non-rackmount "NAS" enclosure, it seems like they design it to stack the drives with no real room for airflow. My ideal NAS chassis would be similar to what @Constantin describes. The Corsair Vengeance C70 came somewhat close --

On the bottom you will see two three-bay modules, holding the drives sideways. You can put a fan in the front (which you can't see) and add a fan in the back (as shown) for fully redundant cooling.

If someone made this into a full tower with four of these modules, that'd be fantastic. You don't need a backplane or hotswap or anything too complicated, although you could actually do some cool things without much extra effort.

 

[Constantin]

The Lian Li A76 I use now follows in those footsteps. What differentiates it from the Corsair though is having twelve slotted 3.5" cages (mounted side-saddle just like the Corsair) featuring backplanes for the SATA drives to allow them to be pulled easily on failure. I would not advise treating the backplanes as hot-swap, even if you retrofit the missing capacitors to the backplanes. It's not a good idea, just power the thing down.

FWIW, my drives are arranged somewhat differently now than shown in the build report to help with cooling. I only have 8 spinners and 12 slots, so I have arranged the drives to feature an "open" gap every other drive like this |o||o||o||o| (where | is a drive and o is a gap). Drives stay below 30*C. I also removed the useless front cover.

The Q26 from Lian Li is a pretty ideal home NAS server, as long as you can stick to Mini-ITX-sized motherboards. It has no 5.25" bays and only 10 3.5" drive bays but uses the same type backplane for the SATA connections and is a very well-built, compact, yet "airy" case. I tried my darnedest to get the Flex-ATX board to fit in there but my metal modification skills are just not there. Not only is the board to long (necessitating the use of a SFX power supply on a bracket) but it's also too deep, interfering with the sheet metal. It's just too tight.

 

[jgreco]

Oh see someone had to figure out such a case design at some point. Great job. Bonus point for the serial numbers. The gapping will help a bit, it also helps in rackmount cases (i.e. 8 drives in a 12-bay with the middle row empty, or 12 in a 24 with every other row). Thanks for sharing that.

 

[esamett]

Please feel free to proceed over to the SAS Primer.

https://www.ixsystems.com/community...-sas-sy-a-primer-on-basic-sas-and-sata.26145/

If you get a Supermicro "-A" style backplane, there is no expander and you will need to provide SAS/SATA connectivity for each individual drive. This is not great. You can stick in a discrete SAS expander but the cabling gets sorta ugly. For 24 drives this would be three 8-channel controllers (LSI 9211-8i or similar).

If you get a Supermicro "-TQ" style backplane, all the SAS are brought out to individual single-lane SAS connectors and the wiring is a nightmare. All the caveats of the -A backplane too.

If you get a Supermicro "-E16" (or "-E26") style backplane, this includes an SAS expander and you only need a single SFF-8087 HBA and cable to hook up your drives. This is great for people who aren't familiar with SAS and SATA. Your SATA drives will work in SAS slots.

Hello and thank you...

It seems I have been working under a misperception. I thought that expanders were not recommended for use with SATA drives. As such I currently use 4 in 1 8087/8088 SATA breakout cables.

If I am reading your post correctly I could I use this 24 bay case?:

Xtore XJ-SA13-224R SAS JBOD 2TB HARD DRIVE DISC 24-BAY STORAGE ARRAY CHASSIS | eBay

• (1) - XTORE XJ-SA13-224R SAS SBOD 24 BAY HARD DRIVE DISC ARRAY STORAGE CHASSIS ENCLOSURE. • (2) - XTORE SAS EXPANDER MODULE J2NA00-50020-A. HDD have been wiped. Unit has scratches on the top panel.

www.ebay.com

The rear of the case has "primary" and "secondary" input sections. Each has two inputs and one output to cascade to additional chassis'.

If so could I use my current HBA?

How many cables would I attach? 1, 2, 3, 4?

reference:

https://www.rackmountmart.com/dataSheet/ss2003_ss2005.pdf

https://www.rackmountmart.com/dataSheet/jbod-connection.pdf

SAS/SATA JBOD -Raid Storage - SS2005 / XJ-SA13-224R-B

/e

 

[jgreco]

Hello and thank you...

It seems I have been working under a misperception. I thought that expanders were not recommended for use with SATA drives. As such I currently use 4 in 1 8087/8088 SATA breakout cables.

If I am reading your post correctly I could I use this 24 bay case?:

Xtore XJ-SA13-224R SAS JBOD 2TB HARD DRIVE DISC 24-BAY STORAGE ARRAY CHASSIS | eBay

• (1) - XTORE XJ-SA13-224R SAS SBOD 24 BAY HARD DRIVE DISC ARRAY STORAGE CHASSIS ENCLOSURE. • (2) - XTORE SAS EXPANDER MODULE J2NA00-50020-A. HDD have been wiped. Unit has scratches on the top panel.

www.ebay.com

View attachment 34683

The rear of the case has "primary" and "secondary" input sections. Each has two inputs and one output to cascade to additional chassis'.

If so could I use my current HBA?

How many cables would I attach? 1, 2, 3, 4?

reference:

https://www.rackmountmart.com/dataSheet/ss2003_ss2005.pdf

https://www.rackmountmart.com/dataSheet/jbod-connection.pdf

SAS/SATA JBOD -Raid Storage - SS2005 / XJ-SA13-224R-B

/e

Having no experience with that product but assuming it to be a simple JBOD with an SAS expander.... DO YOUR HOMEWORK(!!!)...

If it's an older SAS1 (3Gbps) expander based array, it may be limited to drives of no more than 2.2TB capacity.

A SATA disk only has a single channel. A SAS disk typically has a primary and secondary channel (again see SAS Primer if q's). To use a JBOD with SATA disks you will need to use the primary channel. You can hook up the secondary but it won't be talking to any SATA drives.

The two inputs are intended to attach to different servers so that you can have a failover arrangement, but you may also connect both to the same server to theoretically double the speed of the connection between the HBA and the expander. This probably won't actually do much for you, and I've seen cases where that doesn't work, so my suggestion is "one cable".

"Output" and "input" on an expander are generally no such thing; expander ports are bidirectional unless specifically configured in a manner that I rarely see. They put the labels on the chassis to reduce confusion. It is SUPER IMPORTANT not to create a loop. Almost any tree topology is acceptable. This means you could have a stack of three or four of these all in a daisy chain and feed the whole thing with a single SFF8088 from your HBA.

If you have an internal HBA you can buy an AT slot paddleboard to bring the SFF8087 out to an SFF8088. See eBay for example.

 

[Johnnie Black]

They are SAS1 but *should* work with large disks, I have a few five disk SAS1 enclosures from Xtore and they support >2TB disks, expander is not from LSI, but can't say for sure they use the same expanders in that model, so don't get one unless you can confirm first or return it in case it doesn't support large disks, LSI SAS1 expanders are limited to 2.2TB.

 

[esamett]

Having no experience with that product but assuming it to be a simple JBOD with an SAS expander.... DO YOUR HOMEWORK(!!!)...

If it's an older SAS1 (3Gbps) expander based array, it may be limited to drives of no more than 2.2TB capacity.

A SATA disk only has a single channel. A SAS disk typically has a primary and secondary channel (again see SAS Primer if q's). To use a JBOD with SATA disks you will need to use the primary channel. You can hook up the secondary but it won't be talking to any SATA drives.

The two inputs are intended to attach to different servers so that you can have a failover arrangement, but you may also connect both to the same server to theoretically double the speed of the connection between the HBA and the expander. This probably won't actually do much for you, and I've seen cases where that doesn't work, so my suggestion is "one cable".

"Output" and "input" on an expander are generally no such thing; expander ports are bidirectional unless specifically configured in a manner that I rarely see. They put the labels on the chassis to reduce confusion. It is SUPER IMPORTANT not to create a loop. Almost any tree topology is acceptable. This means you could have a stack of three or four of these all in a daisy chain and feed the whole thing with a single SFF8088 from your HBA.

If you have an internal HBA you can buy an AT slot paddleboard to bring the SFF8087 out to an SFF8088. See eBay for example.

You helped me dodge a bullet!:
" 3Gb dual ported SAS; 1.5Gb/3Gb single ported SATA "

 

[jgreco]

Might or might not be a bullet.

6Gbps SATA is capable of, oh, roughly 550MBytes/sec. 3Gbps SATA is capable of half that, so let's say 275MBytes/sec, and 1.5 is maybe 130MBytes/sec.

Your very fastest modern HDD's can peak over 200MBytes/sec, but average transfer speeds to a HDD are well under 130MBytes/sec unless you happen to luck out on entirely sequential traffic. For typical HDD traffic, my guess is that you would never notice.

 

[esamett]

would this work? thanks.

Supermicro 4U 24 Bay Freenas ZFS Unraid Server Xeon 12 Cores 16GB 24x Trays RAIL | eBay

Find many great new & used options and get the best deals for Supermicro 4U 24 Bay Freenas ZFS Unraid Server Xeon 12 Cores 16GB 24x Trays RAIL at the best online prices at eBay! Free shipping for many products!

www.ebay.com

 

[esamett]

Might or might not be a bullet.

6Gbps SATA is capable of, oh, roughly 550MBytes/sec. 3Gbps SATA is capable of half that, so let's say 275MBytes/sec, and 1.5 is maybe 130MBytes/sec.

Your very fastest modern HDD's can peak over 200MBytes/sec, but average transfer speeds to a HDD are well under 130MBytes/sec unless you happen to luck out on entirely sequential traffic. For typical HDD traffic, my guess is that you would never notice.

I thought this was SAS1 and therefore not compatible with >2.2TB disks.

 

[jgreco]

I thought this was SAS1 and therefore not compatible with >2.2TB disks.

There is nothing about SAS1 that is inherently incompatible with larger disks. It is certain bits of LSI silicon that have an LBA limit of 2^32-1, and all of that silicon is SAS1. Therefore it is wise to avoid SAS1 silicon, *absolutely* including HBA's like the 1068 but also probably expanders. There is some debate as to whether or not this problem exists in LSI SAS1 expanders. I think it probably does, but it's been so long since I've seen one, ... well anyways.

 

[joeinaz]

For twelve 3.5" disks in a quiet tower arrangement I have used the two following configurations:

1. An Antec 900 mid-tower case with 3 iStarUSA BPN-DE340SS (4 disk) toolless enclosures installed. Each of the iStarUSA enclosures has a 120" fan and the case has a single 120mm fan in the rear and a huge 200mm" fan in the top of the case. I currently run this system with a SuperMicro ATX sized motherboard and twelve 7200 RPM disks. It is quiet enough to be used in my den. The iStarUSA enclosures will also support SAS disks if you are attached to a SAS controller. I have a spare Antec 900 I may put on eBay after the holidays.

2. Lian-Li A76 PC is my other choice. This is a full tower which supports 12 disks internally. It can also house up to a EATX sized motherboard. The cool part is the optional 3 disk backplanes which allow the removal of disks without touching any connectors. The case comes with four 120mm fans with the option to add 2 more. Cable management is another strong point of the system. I recently found another A76 I haven't decided whether I will keep or sell this case.

 

[styno]

I am using a Thermaltake core W100 case with 4x Icy Box IB-565SSK 5in3 hot swap sas/sata slots. It's perfect for my use case and has wheels that allow me to move it around without breaking my back. Plenty of room inside the case for boot & other ssd's if needed. Also a lot of options to add oversized fans (think 200mm) making it pretty quiet.
Next step is a sas expander so I can free up a slot to experiment with 10Gbit.