External SAS enclosure recommendation

[StorageCurious]

Hi,

I've written recently of my fun project running TrueNAS on a Dell R730XD with 768GB of RAM.

In short - this thing rocks. I have yet to put in my Optane drives (for SLOG - they are still in transit) but the reading is lightning fast. Doesn't hurt to have a 90%+ hit rate in ARC.

I've come to realize though that going with a 2.5" enclosure was a sub-part choice - let's call it a compromise. I am running a SSD pool for block storage, which makes 2.5" perfect for that, but I also need a (large) pool for simple files shared over Samba and general infrequently accessed archives. Turns out the 12 bays left for that purpose are tight. I can only find 2.4TiB drives, which with a "RAID10"-equivalent configuration means only 14.4TiB of storage space. I need something around 30TiB, at a minimum. And if I bought a LFF R730XD instead with 12 drive bays, it's also tight bay-wise considering the SSD pool, SLOG, boot drive, etc.

I would like to attach a JBOD SAS enclosures to my Dell server. I would like to avoid another Dell server/TrueNAS as I prefer to have only one to manage (in fact I'll have a second as a replication of the first, and I figured two total is enough).

Unfortunately my Google-fu and eBay-fu is letting me down. I am finding a lot of things, but they aren't quite what I am looking for. So I was hoping there was an easy answer that escaped me.

I want:
- a minimum of 12 LFF bays Rackmount
- As small as possible U-wise (2U is my expectation, 3U acceptable. 4U if I must, but I don't need 36-48 drives so it definitely would be wasting valuable rack space).
- attached via SAS3 and supports SAS3 and SATA3 drives. Ideally 2 redundant SAS3 controllers
- The enclosure has to be drive-agnostic (not stuck with HP or Dell only drives - bane of modern equipment)
- Dual PSU (because we all know the only thing more likely to give out than a PSU is a drive)

I can live with :
- SAS2 instead of SAS3
- Don't mind used / eBay - in fact, I've yet to regret buying used because I backup/make redundant as much as I can. (knocking very hard on wood)

The only thing I can find is a used JBODSYS-937R-E2JB. Would that be my only choice? Should I just suck it up and have a second R730XD for my bigger spinning pool?

 

[HoneyBadger]

Hello again!

Supermicro does have a complete 2U SAS3 option in the CSE-826BE2C-R609JBOD - however, that comes with the matching MSRP of $2200.

You can easily save some money by buying a used "CSE-826" system, but you'll need to be choosy and look closely at the backplanes, which are coded similar to:

BPN-SASX-826ELY

In this case, the X signifies the SAS generation, and the Y is the expander type. So a BPN-SAS2-826EL1 is a SAS2 single-port expander - it's SAS2 and won't give you the redundant SAS paths you're after. The model you ideally want is the BPN-SAS3-826EL2 which may be challenging to come by secondhand - the runner-up being the BPN-SAS2-826EL2 which will give you the redundancy you want at the cost of not supporting SAS3. But realistically, 8 lanes of SAS2 is 48Gbps (6GB/s) and you're unlikely to be coming anywhere close to that kind of bandwidth using spinning disks.

You can also look at OEM DAEs like the Dell MD1200/MD1400, HP D2600/D3600, and Lenovo SA120 - although those carry the risk of not wanting to update their backplane firmware without a matching branded HBA (or *shudders* RAID card) - there's also odd ducks like the Xyratex HB-1235. The Tegile J1100 is also a straight rebrand of the CSE-826 (and the J2100 its larger CSE-836 sibling) - DAEs from vendors such as those who were absorbed by a larger SAN provider are often available for very cheap.

Note that 2U 12-bay systems require a number of high-static-pressure fans to pull enough air over the HDDs to keep them cool - don't expect them to be quiet, and don't try to make them quiet by swapping in airflow-focused Noctuas, for example. 3U and 4U systems can be kept quieter through larger fans or more unique choices like making a "checkerboard" of filled vs open HDD slots.

Should I just suck it up and have a second R730XD for my bigger spinning pool?

That's certainly another option - it also lets you look at the potential of an R730XD LFF with the dual SFF rear bays, and would allow you to (assume you stay all-SSD in the first one) tune any "system-wide" ZFS parameters for the type of media in question. You could still set up replication between the systems for extra redundancy/backup of your SSD data as well. That would leave you with the option to buy SFF JBODs if you need to grow your "fast" storage, and LFF for your "slow" - but you do have the power and administrative overhead of maintaining two systems.

 

[firesyde424]

You should be fine with the MD1200 or 1400 series. They have been out long enough that you are unlikely, if ever, to need to perform any backplane updates. There are also tons of them so parts and support are fairly easy to come by and they can be had third part for very cheap. You can pick up an MD1200 under $100 on Ebay with a refurbished MD1400 coming in somewhere around $500 without drives.

MD1200 = 6GB SAS, MD1400 = 12GB SAS. Speaking from experience, the MD1200 fans can be annoyingly loud. If you go that route, there are a considerable number of fan mod threads you can find about modifying them with either different fans or to lower the stock fan speeds.

 

[StorageCurious]

You should be fine with the MD1200 or 1400 series.

Aren't those "locked in" to Dell drives only? I had that impression - because these were indeed my first choices.

 

[StorageCurious]

You can easily save some money by buying a used "CSE-826" system, but you'll need to be choosy and look closely at the backplanes, which are coded similar to:

I see those on eBay for sub-500$, but I don't quite understand - they have bays, they have a backplane (in most cases) but there are no SAS ports behind. I'm very comfortable building systems but these things aren't obvious to me when it comes to enclosures. It feels like they are means to put a motherboard in, with CPU, memory, and build a TrueNAS system - not as "dumb" JBOD enclosures.

The bays connect to the backplane - easy enough. But what does the backplane connect to? I need a specific motherboard equivalent, made for just JBOD'ing? Do I need the LSI HBA card (my guess is "of course"), and if so what the heck do they plug into?

I love learning new things so I'm not scared of going down that rabbit hole, but I don't know where to start.

 

[HoneyBadger]

Aren't those "locked in" to Dell drives only? I had that impression - because these were indeed my first choices.

I don't believe so. Dell has far less "vendor lock-in" than HP or Lenovo, but they aren't exactly innocent when it comes to PCIe devices. SAS/SATA though, historically they haven't cared.

I see those on eBay for sub-500$, but I don't quite understand - they have bays, they have a backplane (in most cases) but there are no SAS ports behind. I'm very comfortable building systems but these things aren't obvious to me when it comes to enclosures. It feels like they are means to put a motherboard in, with CPU, memory, and build a TrueNAS system - not as "dumb" JBOD enclosures.

The bays connect to the backplane - easy enough. But what does the backplane connect to? I need a specific motherboard equivalent, made for just JBOD'ing? Do I need the LSI HBA card (my guess is "of course"), and if so what the heck do they plug into?

I love learning new things so I'm not scared of going down that rabbit hole, but I don't know where to start.

Most of the CSE-826 units on eBay are barebones server units, and will come with the -EL1 single-expander backplane - they're designed to have a server motherboard. You can "convert" them into a JBOD by use of SFF-8087 to SFF-8088 brackets (or the equivalent in SAS3):



and the SuperMicro CSE-PTJBOD-CB2 or -CB1 boards



These boards basically just respond to power button input and act as fan controllers.

 

[StorageCurious]

I don't believe so. Dell has far less "vendor lock-in" than HP or Lenovo, but they aren't exactly innocent when it comes to PCIe devices. SAS/SATA though, historically they haven't cared.

Thank you for listing the missing pieces. "Brackets" was the word I was looking for, the possibilities just got bigger!

I might just suck it up and buy the 2200$ JBOD you mentioned. The MD1400, according to Dell forums, seem locked to Dell drives in some ways.

 

[HoneyBadger]

The MD1400, according to Dell forums, seem locked to Dell drives in some ways.

That's a shame if they're locking things out at the backplane/drive level. I know there was a period where the Dell RAID controllers were preventing non-Dell drives - but that was not only in the past but applied to RAID cards which shouldn't be used with TrueNAS.

All the more reason to stick with the SuperMicro hardware though if that's the case (or direct rebrands like the Tegile)

 

[StorageCurious]

eBay link what about this thing - it looks perfect, SAS2 aside. I just have a nagging feeling I'm overlooking something.

 

[firesyde424]

That's a shame if they're locking things out at the backplane/drive level. I know there was a period where the Dell RAID controllers were preventing non-Dell drives - but that was not only in the past but applied to RAID cards which shouldn't be used with TrueNAS.

All the more reason to stick with the SuperMicro hardware though if that's the case (or direct rebrands like the Tegile)

We have both an MD1400 and an MD1420 and do not see this behavior. Our MD1420 is currently full of non-Dell firmware Toshiba SAS SSDs and our MD1400 is currently full of non-Dell Seagate Exos 8TB SAS HDDs. We are using them via TrueNAS core installed on a PowerEdge R730xd and via a set of Dell 12GB SAS external HBAs. There's not really any logic in these shelves, or at least the ones we own. Dell calls them DAS for Direct Attach Storage, but they are just 12GB SAS JBODs.

It's possible there is a version of the MD1400 series shelves that contain actual raid controllers in the shelves themselves. We do see that kind of behavior in our MD3460 high density shelves which have redundant RAID controllers build into the shelves.

Dell's storage pricing is stupid and we stopped buying storage from Dell a long time ago.

 

[StorageCurious]

We have both an MD1400 and an MD1420 and do not see this behavior. Our MD1420 is currently full of non-Dell firmware Toshiba SAS SSDs and our MD1400 is currently full of non-Dell Seagate Exos 8TB SAS HDDs. We are using them via TrueNAS core installed on a PowerEdge R730xd and via a set of Dell 12GB SAS external HBAs. There's not really any logic in these shelves, or at least the ones we own. Dell calls them DAS for Direct Attach Storage, but they are just 12GB SAS JBODs.

It's possible there is a version of the MD1400 series shelves that contain actual raid controllers in the shelves themselves. We do see that kind of behavior in our MD3460 high density shelves which have redundant RAID controllers build into the shelves.

Dell's storage pricing is stupid and we stopped buying storage from Dell a long time ago.

Thank you - yes Dell hardware is fine, I like using the metal things they manufacture, but geez the locking down everything is an annoyance. I will consider the MD1400 then.

Their storage pricing is indeed ridiculous, and their drives too since we all know (it's barely hidden) that they are just rebranded disks.

Does your MD1400 throw any errors, even benign ones, from using third-party drives?

 

[firesyde424]

Does your MD1400 throw any errors, even benign ones, from using third-party drives?

The shelves themselves are just JBODs. The controllers on the JBODs have management ports but they are covered and we don't use them. The drives in the MD1400 and MD1420 are monitored by the TrueNAS server they are plugged into. To my knowledge, the biggest issue we had was actually finding the drive caddies. Turns out Dell has contracts with most of its official suppliers that don't allow blank drive caddies to be sold and we had issues finding third party caddies that would survive more than one or two latching cycles without breaking. It's less of an issue now as used genuine 14th gen Dell caddies are much easier to find.

 

[StorageCurious]

The shelves themselves are just JBODs. The controllers on the JBODs have management ports but they are covered and we don't use them. The drives in the MD1400 and MD1420 are monitored by the TrueNAS server they are plugged into. To my knowledge, the biggest issue we had was actually finding the drive caddies. Turns out Dell has contracts with most of its official suppliers that don't allow blank drive caddies to be sold and we had issues finding third party caddies that would survive more than one or two latching cycles without breaking. It's less of an issue now as used genuine 14th gen Dell caddies are much easier to find.

Thank you for your prompt response.

 

[HoneyBadger]

eBay link what about this thing - it looks perfect, SAS2 aside. I just have a nagging feeling I'm overlooking something.

Interesting that it's using the same chassis as the Storage Bay Bridge two-node systems - I suppose it makes it very clear that you're connecting to one side vs. the other - but yes, that's exactly what you're after in terms of a dual-pathed external SAS JBOD. SAS drives of course are required to leverage both paths.

It's possible there is a version of the MD1400 series shelves that contain actual raid controllers in the shelves themselves. We do see that kind of behavior in our MD3460 high density shelves which have redundant RAID controllers build into the shelves.

That's probably where people are getting confused; I hadn't seen the MD-series shelves themselves care about the drives in the past, and the drives being marked as "failed" would happen up on a RAID/controller level, with an LSI HBA obviously not caring what firmware is on the drive.

 

[firesyde424]

That's probably where people are getting confused; I hadn't seen the MD-series shelves themselves care about the drives in the past, and the drives being marked as "failed" would happen up on a RAID/controller level, with an LSI HBA obviously not caring what firmware is on the drive.

This has been our experience with Dell external storage as well. We have a considerable mix of older Dell standard and high density external storage shelves. The models with HBAs such as the MD3260, MD3060e, MD1200, and MD1400 do not care what drives we use in them. The models with full RAID controllers such as the MD3460 do care and we have to use Dell firmware drives. Unfortunately, it appears that all of Dell's newest external storage solutions are now vendor locked, even the JBODs. Because of that, we switched a while back from Dell's external storage solutions to the Ultrastar Data series of JBODs from Western Digital.