DIY all flash/SSD NAS - not going for practicality

[Allan_M]

Received the backplanes on friday. Looked fine and all jumpers were as they should according to the (BPN-SAS2-216EL1/EL2 BACKPLANE - USER’S MANUAL)

I found some older hardware and soon realized that it was a 9260-8i I had not the 9211-8i. Bummer, but I guessed is should still work.

Today I haphazardly put the things together on the "project of the month"-table we have in the corner of our living room.

According to the manual, the SAS cable should go from HBA to the topmost port on the SAS expander.

The green post-it note was for jotting down power usage - just to see. The base system is a Core i7-2600S with 4 GB RAM on a ASUS P8H67-I Deluxe B3 motherboard. Pulled the bracket from the LSI 9260-8i so it wouldn't bump into the table and eject itself from the PCI Express slot.

The backplanes are connected with a single molex connector.... I know... I know. Wire gauge, power draw on the 5V line. Don't get a heart attack. It's just for checking out whether things work at all or not. Before connecting everything I put my multimeter into continuity mode and check the molex connectors on the backplane. To my surprise, they're all connected to each other - though with what appears to be some filtering on each, that had me think they were seperated into groups. They're not. Perfect! (disclaimer: for testing and toying around of course)

Anywho.

Everything lights up, POST seems to be fine but the disks are not detected by the controller. I suspect it's the cables that I had laying around - I seem to remember it bought another set of cable for my other server. Perhaps they're just bad / broken. Also, I'm unsure, if a RAID-controller supports the SAS expanders*. And according to some googling around, it seems to be a pain to flash the 9260-8i into IT-mode.

Preliminary power figures are not that exciting / surprising.

HBA adds around 10-12 W and the two backplanes (without drives) add around the same numbers. So, around 35-40 Watts without drives. That's what I'd expect anyway.

Even though the drives are not detected I'm still planning on trying out connecting them. They still draw power it seems which should be fine for estimating scaling.

That kinda leads nicely into the following two thoughts:

1. Considered stripping the SSD PCBs from the drives and running them without their enclosures - most of them look like mSATA drives inside anyway. Thought It'd look kinda cool and Star Trek'y and perhaps help with temperatures. I just have to find a way to keep track of the drives and their position. The problem is, if the label on the drive case itself is the only place they put the serial no. for instance.
2. Still haven't decided on a power supply. Initial calculation suggests a 350-450 W range unit. That puts it into SFX / FlexATX territory if I still wanna keep it somewhat compact. But then noise becomes a factor.

I have to admit that decent airflow was on my mind before, but after running the 9211-8i and two expanders without drive activity and experiencing how hot they got - even with the Noctua NF-A14 PWM on the table - I'm a little concerned. Need to do some more testing to see how a enclosure effects airflow and temps, but I think that makes more sense, when I'm actually able to detect and use the drives.

* Correction: According to this Avago Compatibility Report for MegaRAID SAS Gen2 Controllers - document, the MegaRAID 9260-8i should support the BPN-SAS2-216EL1. I guess it's down to the cables or perhaps the firmware. Have to check up on that.

 

[jgreco]

The high end LSI RAID controllers absolutely do support SAS expanders, and actually it's probably more unusual for them to NOT be connected to an SAS expander.

The labeling of "SAS IN" and "SAS OUT" on an expander can largely be ignored. It's like an ethernet switch -- you can have a port labeled "uplink" but what does that actually mean compared to the other ports?

The high end RAID cards might not show unconfigured drives in POST. The end goal of the LSI RAID environment is to provide RAID-protected devices to an OS that isn't smart enough to do it on its own (Windows, etc), so what you want to do is to go into the BIOS utility with the control-whatever keystroke and see if they show up as available drives there. There is an option for unconfigured drives to be shown as JBOD disks. If that's available from the BIOS utility, that'd be the best option, I think. I believe this is usually turned off in the default RAID configuration, which is probably why you're not seeing any drives.

The 9260 is a high end RAID controller and is the thing I warn against using in the "What's all the noise about HBA's" post, but as long as you're not planning to use this thing seriously, and are just playing around, it'll be fine.

 

[Allan_M]

The end goal of the LSI RAID environment is to provide RAID-protected devices to an OS that isn't smart enough to do it on its own (Windows, etc), so what you want to do is to go into the BIOS utility with the control-whatever keystroke and see if they show up as available drives there... I believe this is usually turned off in the default RAID configuration, which is probably why you're not seeing any drives.

I've been into the utility and went to the "devices" tab. No drives. I'll try to poke around but I'm fairly certain that it's the cables that are causing the problem. I think I recall they're the reason I bought another set of cables when I set up the FreeNAS box.

The 9260-8i was 100 % for playing around. Already ordered the 3008 with accompanying cables

 

[jgreco]

Okay, please report back as events unfold!

 

[Allan_M]

Well. Updated the firmware - went smooth - but still no drives. I've ordered some new SFF-8087 cables, should be here tomorrow.

I know this project kinda got more ambitious than originally planned - and this is just unnecessary, but it annoys me that I've had this 9260-8i on the shelf without ever using it and now I'm being side tracked by some cables.

 

[Allan_M]

Yay! New cables arrived today. Seems to work!

Guess I'll throw the other cables into the bin. No idea why they wouldn't work

 

[jgreco]

Guess I'll throw the other cables into the bin. No idea why they wouldn't work

You had me holding my breath there a bit when they didn't work. I was hoping I hadn't led you on a misadventure. Bad ${stuff} is just sort of an occupational hazard though.

Let me tell you about the Dell PERC H310's I've had to shave some PCB off of so that right angle connectors mate correctly ...

 

[Allan_M]

You had me holding my breath there a bit when they didn't work. I was hoping I hadn't led you on a misadventure. Bad ${stuff} is just sort of an occupational hazard though.

Let me tell you about the Dell PERC H310's I've had to shave some PCB off of so that right angle connectors mate correctly ...

No worries. Though I'd might been disappointed, if something was broken/DOA [knocks on wood], I'm also a grown up in charge of how I spend my own money and the risk / pitfalls of buying on Ebay.

I'd rather not. Modifying PCB's is the stuff of nightmares around here - just ask my girlfriend and her MSI laptop back in '07. The 'modifying the PCB'-part was unintentional, as those things tend to be.

I've updated the post with an image of 37 drives connected. According to the Watt-meter, the entire system draws around 185 W with all drives initializing.

As earlier mentioned, the motherboard, CPU, RAM and two backplanes draws around 78 W. So let's say something, something ~100 Watts for a potential 48 all SSD arrangement.

I'm anxious to test out the entire system when the other parts arrive.

 

[Allan_M]

128 GB of DDR4 REG ECC (MTA36ASF4G72PZ-2G3B1IK) arrived with the mail today. Yay!

 

[Allan_M]

Motherboard arrived today. Still waiting for the HBA, PCI riser cable and SFF-cables.

Correction: HBA and cables was waiting for me in the mailbox. Had almost given up on seeing them until december, but noticed, that they were marked as delivered on eBay yesterday [!]. Checked the shed - nothing, checked the hallway - nothing. Thinking I'd been scammed or perhaps someone else in the neighborhood had received it instead of me I asked my better half. She asked, if I had checked the mailbox, lo and behold. There they were!

Still waiting for the riser cable, but at least I can set things up now and see if everything works as it should.

I'm so excited right now. Love getting new hardware in general, but this is something else - thanks everyone

 

[Allan_M]

Well. It's up and running: Even posing for some late night photography

I have to admit. It seems somewhat opulent, to have 48 decommissioned SSDs running on two SAS2 backplanes connected to a 9340-8i, two surprisingly fast 120 GB PNY CS900s 240 GB Kingston A400 SSDs with TrueNas Core, 4 cores / 8 threads and 128 GB RAM.

Everything is kinda jank af, and it's currently connected to the rest of the network through an old 1 Gb switch that used to run so hot that I had to pull the lid and install a cooling fan (soldered to the input power, of course).

Current [wired] topology:

[Created with Visual Paradigm Online Free Edition]

Minor correction: The 1 Gb switch is connected to the AirPort Extreme in the living room, not the switch with the TV and gaming consoles as indicated.

Perhaps I'm able to illustrate, why some shared fast storage is desirable. I would be cool just to offload files from portable external storage (SSD/SD-cards) onto the CrazyNAS for editing and then transfer everything onto the FreeNAS box once the project has been delivered.

Anywho.

iPerf3 confirms 1 Gbps connectivity and preliminary file transfers are rock solid around +100 MB/s. Yay!

It's a shame, that nothing else on my network is 10 Gbps, so I guess I'll have to look into that.

Right now I'm just amazed by the fact, that everything seems to be running as it should. Power usage hovers around 95 - 100 Watts. I've recorded a peak at around 155 Watts, but I'll keep an eye on the meter to see if it should pass that at any time.

 

[jgreco]

That's hilarious and totally awesome

 

[Etorix]

Nice early shots from CrazyNAS!

It's a shame, that nothing else on my network is 10 Gbps, so I guess I'll have to look into that.

To just connect one PC to a NAS, an Asus XG-U2008 switch does the job. For more posts, or two NAS doing replication over 10 GbE in addition to serving one or two workstations, without breaking the bank there's the QNAP QMW-M408 series (different combinations of 10 Base-T/SFP+/shared). Or, just for the 10 GbE part, the Mikrotik CRS-305-1G-4S+IN or CRS-309-1G-8S+IN—with a 10 Base-T transceiver for CrazyNAS and the rest hopefully on SFP+.
@jgreco has made an excellent primer:

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

It seems that this fun project "not going for practicality" is going to transform your home network. Beware! It could end up having a real use…

 

[Allan_M]

That's hilarious and totally awesome

I hole heartily agree.

Nice early shots from CrazyNAS!

Thanks

To just connect one PC to a NAS, an Asus XG-U2008 switch does the job. For more posts, or two NAS doing replication over 10 GbE in addition to serving one or two workstations, without breaking the bank there's the QNAP QMW-M408 series (different combinations of 10 Base-T/SFP+/shared). Or, just for the 10 GbE part, the Mikrotik CRS-305-1G-4S+IN or CRS-309-1G-8S+IN—with a 10 Base-T transceiver for CrazyNAS and the rest hopefully on SFP+.

I’ve been looking into upgrading a part of the network into 10 gig for some time. I’ve already bought a MikroTik CRS112 and I’m considering a new docking station for the MacBook Pro with 10 gig functionality in the near future.

Sudden realization: The CRS112 is not SFP+ Guess I have to cancel my order then. Dang it!

The workstation is a little harder, because it’s an ITX board, so I’ll have to think about whether or not I’d get a new motherboard or upgrade the entire rig.

It seems that this fun project "not going for practicality" is going to transform your home network. Beware! It could end up having a real use…

Shush now. My wallet… I mean my wife might hear you

 

[Allan_M]

Well. Time for another update, I guess.

Status so far; got all the parts. And also - the PCI-e riser cable is stupid.

The cable itself works just fine, but as you probably can deduce from the picture alone, apparently it's not long enough. My plan was to have the HBA lay flat and being parallel to the motherboard. However, even though the riser cable by itself is long enough, the ends are very rigid near the print boards - meaning I have lesser cable length than anticipated. Well. You live, you learn.

My solution to the problem?

Throw some more money at it

Ordered one of these on eBay. Should arrive some time next year - yay!

On another note: After realizing that I was too eager once I got the MikroTik CRS112-8G-4S-IN, I cancelled the order and got a MikroTik CRS305-1G-4S+IN instead. It's been on the shelf since November 24th and will be until I decide what to do with the rest of my setup.

I'm still debating whether or not I should upgrade my desktop / workstation (new motherboard, just getting a USB-based ethernet adaptor), getting a new dock for my MacBook Pro or doing both. Also, I'll have to look into a NIC for the NAS in my sig.

Anywho.

Cheers. Hope you enjoy the holidays

 

[jgreco]

Yeah, PCIe risers can be hell.

 

[Allan_M]

The project has stalled somewhat, but that's unrelated to what I'm gonna ask about this time around.

The stream of flash drives continues and as such I almost have enough for another backplane.

At first I thought to myself that I'd have to invest in another HBA but then I remembered, that there's an expander.

Do I understand the concept correctly if I'm to assume, that I can 'just' get another backplane and attach it to one of the two original ones?

My thought then is to rearrange the vdevs. For instance:

• Pool
  • vdev#1 = backplane #1
  • vdev#2 = backplane #1
  • ...
  • vdev#n = backplane #n

For right now, one or two additional backplanes are all I'm considering. But what limitations are there when cascading them? According to the LSI 3008 documentation I see they say 'up to 1000' devices. I don't think I'll ever get that far down the rabbit hole.

For all I care, I just wanna add a bunch of disks, when it's time.

I hope the above made sense?

Btw.: The reason I'm asking this is, that it'd might have an impact on how I originally planned to hang it on the wall.

 

[Allan_M]

Well. It's been almost a year and except for some testing since, the system has been sitting idle (litterally) on top of some broken LCD-panels (for another projekt: DIY video light) while the backplanes and SSDs sits atop of a shelf (the system has since been put in antistatic bags for long term storage - see the second picture).

On a somewhat long list of #firstworldproblems, it doesn't help that 'my guy' keeps pushing me decommissioned SSDs. Not that I’m going to complain.

It has been useful for doing some mockups for also another projekt: Xbox S - The 'S' is silent and also stands for Sleeper inspired by Jeff from Craft Computing:

The last picture is the temporary setup in the living room.

Spec. for those who are interested:
Core i5-10400F
ASRock B560M-ITX/ac
32 GB Corsair Vengeance LPX DDR4-3200 C16
PNY NVIDIA RTX A2000 Black - 6GB GDDR6
Xbox controller (not pictured)

Local storage:
128 GB or 240/256 GB NVMe SSD for system and apps (currently using a 128 GB but I've got another NVMe just lying around)
500 GB / 1 TB 2.5" SSD for games (currently using a 500 GB Samsung EVO)
-> Have also been toying around with iSCSI - again, inspired by Jeff. Have to admit, if I could get 2.5/5 or even 10gig to the 'Xbox', that’d be cool.

Using GTA V as a 'power virus' representative for gaming (average - peak Watts):
4K:
- Benchmark: 120 - 130/140 Watt
- In game: 130 - 140/150 Watt

1440p:
- Benchmark: 120 - 140 Watt
- In game: 100 - 120 Watt

1080p Benchmark:
- 100 - 110/120 Watt
- In game: 90 - 100 Watt

Though the power consumption (1080p vs 4K) is around 50 % higher, I do prefer smoother gameplay and higher details.

It's an amazing setup that does 1080p@ultra/high just fine and even some 4K - but I regard that being kinda wasting energy. Just completed 'Mafia Definitive Edition' and 1080p seemed just fine at 'living room distance' to our 65" TV

Enough rambling about the Xbox-project.

I just wanted to give an 'update' for those interested. Have a nice day

 

[Allan_M]

Well. Almost there ...

The attached image shows the 80 Samsung-drives that have been prepared for the CrazyNAS. Along with some drives from other vendors the combined raw capacity has now reached 12,320 TB. The capacity for the drives shown is around 10 TB.

As fate will have it, it turns out that I'm almost at a point, where the CrazyNAS' raw capacity almost matches the current usage on the other ['old'] NAS (around 14 TB used out of 24 TB available).

So. I'm currently stuck because I haven't decided on a power supply. I've considered this one: HDPLEX 250W passive GaN AIO(All-In-One) ATX Power Supply

or, this one: HDPLEX 400W Hi-Fi DC-ATX Power Supply

- but I'm certainly open to suggestions

 

[NugentS]

I think a 250W has got to be too small