Interesting new option for all-flash build

[sretalla]

I was inspired by seeing this review for a soon-to-release product:

ICY DOCK MB872MP-B Review - 12 bay M.2 SATA

In our ICY DOCK MB872MP-B review we check out a 12-bay M.2 SATA SSD enclosure to see how this unique solution performs

www.servethehome.com

And then went looking for a matching case:

Core 500

www.fractal-design.com

I think with the two of those, somebody (maybe me one day), could build an excellent system with up to 12 of something like this: https://shop.westerndigital.com/products/internal-drives/wd-red-sata-m-2-ssd#WDS100T1R0B

It would take in addition:
a decent power supply
a Mini-ITX mobo (with Intel NIC(s)) and matching CPU/RAM
an HBA (I think SAS 12GB probably a good idea)
The SAS to Oculink cables with at least 3 oculink ends from the SAS ports on the HBA chosen
One or 2 SATA SSDs for boot media
Depending on Mobo, maybe an M.2 NVME SSD on-board for something like jails or whatever... possibly will perform better than the SATA m.2 you can do in the hot-swap even in various forms of pool layout.

I have no affiliation or profit sharing agreement with the linked products/vendors.

Happy NASing!

 

[HoneyBadger]

"A 12x M.2 into 5.25" bay sounds really interesting, I think I'll read the review."

$767 MSRP, cables extra, one lonely 40mm fan

100% DOA for me until they address at least the first and last point. Mostly the first though.

 

[sretalla]

100% DOA for me until they address at least the first and last point. Mostly the first though.

I hadn't yet looked at the price... maybe it's a little high, but compared in size, noise and cost with a server chassis that you will need to host 12 M.2 SSDs otherwise, maybe not so bad.

The single fan seems not to be an issue with the SATA M.2 disks, which don't really get anywhere near as hot as their NVME cousins (the review seems to agree with that).

The cables could be had for under $100 if you shop on Amazon

 

[HoneyBadger]

It's definitely a very niche product for sure, but I'm struggling to find a use-case where you can't work around the noise of a bigger rackmount for the cost difference. Right now I could probably buy a 24x2.5" Dell R720XD as a "complete unit minus drives" for not too much more. Apply some sound dampening and tuck it away in an HVAC'd closet or other secure climate-controlled area, and then I can use the 2.5" form factor.

With a significant price-cut it becomes a lot more appealing for those weird edge cases.

The single fan is more of a personal bugbear on redundancy - if I'm laying out $750+ for something like this, I want to have the secondary fan there for a backup in case it's needed.

 

[jgreco]

I hadn't yet looked at the price... maybe it's a little high, but compared in size, noise and cost with a server chassis that you will need to host 12 M.2 SSDs otherwise, maybe not so bad.

The price is basically a deal-breaker.

We have a lot of 2U gear with 12x 3.5" bays on it that are used for FreeNAS or other mass storage on the front under ESXi. In our environment, RAID1 and sparing is a business requirement, so finding a way to put a decent ESXi-compatible RAID controller into a system and finding a place for some SSD's to live is important.

I previously discussed using a carrier to hold four M.2 SATA SSD's --

https://extranet.www.sol.net/files/freenas/cool-hardware/upper-deck-storage.png

and we've actually been doing this with Silverstone SDP11's instead, but stacking a second SDP11 PC board on top with a 5/8" spacer. This gives you eight M.2 SATA SSD's in the space of a full height (1.6"H) 3.5" HDD.

This can be mounted in the front part of the PCIe expansion slot area by the fan bulkhead in a Supermicro chassis, and can hold up to 8x 2TB SSD's. Total cost is around $100 for the two SDP11's. Of course it isn't hot swap, but we maintain warm spares in the array, so that isn't a dealbreaker.

 

[jgreco]

This was rolling around in my brain for a bit. Back in the '90's we made our own disk shelves and 4U units that held 9x5.25" HH were pretty common... it would be fun to stuff nine of those IcyDocks (108 M.2 SSD's) into 4U with a nice fast SAS expander or four. I think we still have one around in inventory somewhere...

 

[Etorix]

I hadn't yet looked at the price... maybe it's a little high, but compared in size, noise and cost with a server chassis that you will need to host 12 M.2 SSDs otherwise, maybe not so bad.

Compared with the cost of any consumer chassis which could hold 12 2.5" SATA drives, it still looks bad. And with 2.5" drives you can use 4 TB QLC SSDs while M.2 is limited to 2 TB. Admittedly the price of 12 large capacity SSD would dwarf the price of this dock anyway.
For a home user, this is a dead end.

There should be some fringe use case for corporate use. But I struggle to come up with one. And I suspect that @jgreco will take a sizeable fraction of the worldwide sales if he buys nine docks in one go for his fun experiment. Please post pictures!

 

[jgreco]

Compared with the cost of any consumer chassis which could hold 12 2.5" SATA drives, it still looks bad. And with 2.5" drives you can use 4 TB QLC SSDs while M.2 is limited to 2 TB.

That's a fantastic point. Also, with 2.5" SSD's, you don't really even need a chassis, you could just put a big rubber band around your stack of 2.5"'s, haha.

Admittedly the price of 12 large capacity SSD would dwarf the price of this dock anyway.
For a home user, this is a dead end.

Perhaps. The best price I've hit on 860 Evo 2TB M.2 in the last few months was $229, so that's $2750 in SSD's. But if I look back just a little bit in time, a solution that provided 24TB of SSD for $3500 would have been very attractive, and I'd bet that's even price competitive with FreeNAS setups from the early to mid 2010's.

There should be some fringe use case for corporate use. But I struggle to come up with one.

I kinda doubt any significant corporate use. If you have the money to blow on a $700 tray, there are probably better prebuilt server options available, even if you have to go to 2.5" SSD's. Pricing on the M.2 SATA SSD's is a premium over 2.5" anyways, by about 20-25% in the last six months.

This is a really cool solution for someone who is building a gaming PC and wants to have primary NVMe with craptons of secondary SATA SSD storage, or possibly for people with non-rackmount servers or workstations who want to add a bunch of SATA flash but only have a single optical drive bay available. I promise you they are out there, but they're not that common, and the price will be somewhat off-putting.

And I suspect that @jgreco will take a sizeable fraction of the worldwide sales if he buys nine docks in one go for his fun experiment. Please post pictures!

I think I already locked up the market on 860 Evo 2TB M.2's. They're unobtainium at reasonable prices, although Samsung Direct finally got them back in stock at their normal $250.

 

[jgreco]

Also, not the picture you asked for, but still I like it, the pile of M.2 I'm working on ...

Yes, that really is 34 2TB M.2 drives, some SATA and some NVMe.

Current hypervisor design is favoring 4x 2TB NVMe (non-RAID) and 5x 2TB SATA (2x RAID1 with a spare) in each hypervisor. For awhile we were raiding Best Buys to get 970's (you can see two retail-tabbed ones in the stack). The 980's keep appearing and disappearing too.

 

[masru]

That's a fantastic point. Also, with 2.5" SSD's, you don't really even need a chassis, you could just put a big rubber band around your stack of 2.5"'s, haha.

And I thought I‘m the only one doing that …

 

[masru]

Afaik m.2 SATA drives are a dying breed, So I wounder why they came out with that product now.

But speaking of ICY DOCK, check this one:

ExpressCage MB326SP-B

www.icydock.de

6 2.5“ SATA drives in the same space, and reasonable price. And two fans!

With 6 8tb drives you get double the space as with the m.2 part above.

I have a 4 bay one, they are okay-ish with not very sturdy plastic. But they have an all-metal series which is quite nice.

 

[jgreco]

And I thought I‘m the only one doing that …

Doubtful. Plus there's other stuff you can do. For example, in the early 2000's, we bought some 1U AIC chassis and stuffed AMD Opteron boards in them, but by ~2010 that was being rapidly outstripped by Intel, so we used Supermicro X9SCi boards to refurb them, and made them into hypervisors at the same time. The AIC followed the general 1U design and had three spots up top above the 4x 3.5" bays, one for a laptop optical, one for a 2.5" HDD/SSD, and one for ????, but since AIC didn't get overly complex with their sheet metal work, it turns out that you could strip out the mounting studs and other hardware and gain access to a large area:

This one's actually in the process of being stripped down and is on the shop bench. You'll see three drives along each side and two in the middle, SSD's are held down with industrial velcro for serviceability.

 

[jgreco]

Afaik m.2 SATA drives are a dying breed,

Yup. I'm pretty exasperated with Samsung that they didn't release 870 Evo M.2's, or larger capacity SSD's overall.

The industry seems to be gung-ho on moving towards NVMe, and what's not to like about that? Oh, yeah, two things, lack of RAID (don't correct me, I know there are NVMe RAID controllers) and lack of density (also don't correct me, I know what a PLX switch is).

I came to FreeBSD because the costs for vendor (specifically: Sun) gear were killing me back in the early '90's. PC hardware has never had a totally clear stratification between desktop and server, even if Intel tried to make one with Xeon. The ability to pick from a wide variety of component options has always been awesome, and early on I was one of those who was abusing IDE/ATA drives in servers, etc.

The server market seems to be struggling to divorce itself from the general consumer PC component market now. We've watched the introduction of U.2 and other NVMe attachment/form factor introductions for "server-only" uses -- always at a significant price premium, on both the SSD and chassis side of the equation.

Overall, it's been made clear that SATA SSD's (not just M.2) are not a focus, which is really shitty due to the massive number of 2.5" bays out there where HDD run hours are edging up into the 50-100K hour range. Even as a business, I don't like to waste money on needless server refreshes (as evidenced by the picture above).

I do RAID1 for everything that's important, because I build "must-run" systems that form the networking infrastructure which allows them to run and be managed. For this, an LSI RAID controller and an eight-M.2 SSD carrier is an inexpensive retrofit into an existing server, and with controllers like Dell's H840P with its massive 8GB cache, SATA SSD's "slowness" is not particularly noticeable.

It is basically impossible to build something of similar capability with NVMe without actually buying a chassis designed for it.

So I wounder why they came out with that product now.

Proof of concept? Perhaps they are planning to do something radical. Sabrent's 8TB NVMe M.2 SSD might mean that there could be some interest in creating some sort of PCIe monster, where you had a 5.25" bay stuffed with 12x NVMe backed by a PLX switch and attached to the host via PCIe. The idea of putting 96TB into a 5.25" bay is crazy. But you would want to experiment with this before encouraging customers to stuff five figures worth of SSD's in there.

But speaking of ICY DOCK, check this one:

ExpressCage MB326SP-B

www.icydock.de

6 2.5“ SATA drives in the same space, and reasonable price. And two fans!

They also make one for 8 2.5" SATA.

With 6 8tb drives you get double the space as with the m.2 part above.

I have a 4 bay one, they are okay-ish with not very sturdy plastic. But they have an all-metal series which is quite nice.

The four bay all-metal ones are our go-to's for deskside.

 

[Arwen]

I ran across these 2 earlier today, which have 4 x M.2 NVMe to a 5.25" external disk bay;

ICY DOCK Full Metal 4 Bay M.2 NVMe SSD Mobile Rack for External 5.25" Drive Bay - MB720M2K-B

As well as this;

ICY DOCK ToughArmor Rugged Full Metal 4 Bay 2.5" NVMe U.2 SSD Mobile Rack for External 5.25" Bay - MB699VP-B
ICY DOCK (Tool-Less M.2 to U.2 Adapter) M.2 PCIe NVMe SSD to 2.5" U.2 (SFF-8639) SSD Converter - EZConvert MB705M2P-B

Neither are cheap. But, as a new owner of an AMD Epyc with the requisite PCIe lanes, it's attractive to avoid the "dead end" of SATA SSDs.

Icy Dock also has single and dual NVMe to a 3.5" external disk bay.


Side note. With the AMD Epyc board I bought, it was not enough to have 31 SATA and 4 x 16 lane PCIe, (@ version 4). They said, want more? How about 2, 4 lane M.2 NVMe slots? Still not enough? Want an 8 lane PCIe SlimSAS port? What about turning 1 or 2 of the SlimSAS with 8 SATA ports, into 8 lanes of PCIe?