Upgrade advice needed including short depth chassis options

[tartan-prophesy]

Hi,

I've been running a FreeNAS system for a long time and it's due an upgrade. I mainly manage a lot of media with the system using a lot of jails. I also have a VM running for Roon and a few jails for home automation / Gitlab.

The current system is:

• 16GB of RAM
• Pentium G3220 @ 3.00GHz
• 6x Hitachi 4TB 5400RPM SATA (HDS5C4040ALE630) in RAIDZ2
• Fractal Design Node 304
• Seasonic 400W platinum PSU
• ASRock E3C224D2I motherboard
• USB thumb drive for booting

It's served me really, but is becoming slow. I'd like to:

• Keep the discs, but have an upgrade path to double the capacity (maybe more... who knows what the future will bring)
• Add a proper boot drive
• Add an l2arc
• Stick much more RAM in, I think 16GB is simply not enough
• Use my existing rack, this can only accomodate gear that is upto ~45cm deep. I was thinking a 3u case would be ideal
• Upgrade the CPU and motherboard to support all of the above

The case is causing me the most headache. I'm not fussed about hot swappable drives, but I do want to be able to get everything into a 3U case. It would be nice to have something with a backplane to handle wiring issues, but I guess this is a non starter as these don't seem to be available in a reduced depth.

Some advice for what to do would be really appreciated.

Thanks

 

[NugentS]

Why 3U and not 4U? (not that it helps)
The depth of a case isn't directly related to the U size of the case. Its really the size of the motherboard that sort of defines the depth of the case
Mind you fitting lots of drives in a rack mount of that depth is a bit difficult. I have had a quick look and can't find anything suitable.

You are right - the CPU is a bit on the low side as is memory and the PSU is probably light for adding more disks. Jails and VM's I bet are using most of the memory and CPU you have available.

Oh and cases are in short supply ATM, at least in the UK they are

 

[tartan-prophesy]

A 4u chassis would work as well, but as I don't need loads of drives I think it should be possible to do something smaller.

 

[tartan-prophesy]

Looking at server-case.co.uk, I think this SC-3380 might fit the bill (if only it were in stock!)

It's not a patch on the supermicro chassis, but I don't think one exists that will fit into my depth anyway. In terms of motherboard I guess it makes sense to stick to a mini-itx form factor as well. The ASRock Rack E3C246D4I-2T looks tempting as it takes a ton of RAM and has 8 sata ports (is there any value to me getting something that supports SAS?). Perhaps I could use the M.2 for the L2ARC device.

I've no experience with CPU coolers in these types of cases, whats the recommended setup?

 

[jgreco]

Seasonic 400W platinum PSU

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can find. DO NOT DO THIS. Your NAS lives or dies by...

www.truenas.com

In general, you are likely to have some difficulty with chassis options. Classic short-depths like the Antec IPC3480B are rare these days.

 

[tartan-prophesy]

jgreco, thanks for the info about power supply sizing. I'll make sure I get the right capacity so there's no issues. Apparently the SC-3380 is coming into stock this afternoon! It looks like the E3C246D4I-2T is difficult / very expensive to get hold of so I think I'm back to the drawing board on that one

 

[tartan-prophesy]

@NugentS I went with 4U Chenbro RM42300 in the end. This allows an ATX motherboard which should help with any possible expansion in the future. I'm thinking of going with:

• Supermicro X12STH-F
• Intel Xeon E-2386G (this is probably overkill)
• 2x 32GB of ECC Ram
• 6x existing Hitachi 4TB 5400RPM SATA (HDS5C4040ALE630) in RAIDZ2
• SFF-8643 to 4 SATA, I think this should allow me to plug all of my drives straight into the board
• Noctura cooler TBD when I see exactly how much room I have in the case
• PSU TBD

Is this a sensible setup? I've got an old SATA SSD knocking around that I was thinking of using as a boot drive. Should I take advantage of the M.2 interface and add an NVMe drive to act as an L2ARC. What size would make sense? Some of the guides refer to 'Enterprise' grade drives, what does this mean?

 

[NugentS]

The only problem I can see is that this is a workstation board taking UDIMMs
The logical upgrade is to a server board (more PCIe lanes etc) which will take RDIMMs

As to the specific questions.
L2ARC - this is only of use if ARC isn't big enough. Check your arc hit ratio (after you have been using the NAS for a few weeks) if its 90%+ then there is no point in L2ARC. Also be aware that L2ARC uses ARC, so it can slow things down although you are talking 64GB on 16TB of storage which seems a good starting point. Generally however max out ARC before thinking of L2ARC and you can add 2 more DIMMS.......

Use the old SATA SSD as a boot device. No point in wasting anything better.

Enterprise drives generally have:
1. Much better endurance than consumer drives
2. Generally have PLP (Power Loss Protection) which can be useful under some circumstances (not for L2ARC which is not pool critical, so doesn't even need resiliancy)
3. Much more expensive

If you have enough ARC and are desperate to use that M.2 slot (it would bug me not to use it) then the following suggestions may help:
1. L2ARC, but metadata only. MIGHT make directory browsing and similar slightly snappier
2. A fast pool for Jails / Containers (if you are planning lots of those). Be aware however that it will have no resiliancy

On that board you have 3 useful PCIe slots and the motherboard almost certainly supports bifurcation - so you could add a PCIe to NVMe card and add some more NVMe M.2 drives, thus allowing an array of NVMe drives. PCIe to NVMe cards are not expensive.

That CPU should last you a long time, its got decent cores, HT, high clock rate and QS (if you need it for transcoding)

 

[NugentS]

Drive Bay

External 3.5"：1 External 5.25"：3 Internal 3.5"：4 Slim ODD：1

You are going to need to add a 5.25" to 3.5" drive bay adapter - just make sure you aren't compromising the cooling of the drives (I don't know the case)

 

[tartan-prophesy]

Supermicro describe it as a 'server board' and it has IPMI on it, is it the use of UDIMMs over RDIMMs that make it look like a workstation board to you?

Hah, yes part of the motivation is to make use of the M.2 slot. I think I'll go down the, get it running, check some stats and then try it out as an L2ARC for metadata only.

I love the idea of getting a PCIe to NVMe card. I could totally run the jails / vm off of that. Which is making me think maybe I should go for an ATX size board so that I can get some more 16x slots (my brief research on the 4 NVMe card to PCIe is that they require a 16x bus. If I ever add another set of spinning disks then I'd need to get a HBA adapter which would also probably need a 16x bus. The only problem is Supermicro don't seem to do an X12 ATX board that doesn't use the Xeon Scalable processor (which I understand it totally unneccessary). Any recommendations?

The version of the case I got (RM42300H01) has 6x 5.25", 2x external 3.5", 4x internal 3.5" and 1 slim ODD. The 4x internal 3.5" can be cooled by an 80mm fan, and I'll get 2x drive 3x 5.25 to 5x 3.5"cages which take 120mm fans. I think this should provide adequate cooling. Along with plenty of room for expansion. It'll be a total rats nest of wires, but I only have a short depth rack so I think it's the best I can do.

 

[tartan-prophesy]

It looks like the Supermicro H12 boards offer a lot more in terms of PCI slots, they also offer RDIMMS. However, it looks like it uses Broadcom for the networking (either BCM5720 or BCM57416) and Broadcom 3008 for SAS which I understand are not well supported under FreeBSD. It also seems to include 2x M.2 slots as well so it's an absolute beast in terms of connectivity.

 

[NugentS]

The problem re server / workstation is that the CPU only supports 20 PCIe Lanes (and the chipset adds a few, generally slow ones)
This is why I define it as a workstation board (its more a workstation CPU).
[I realise that my description may be disagreed with]

Server CPU's support more lanes and thus you can do more with them.
The E5 2660V3's in my VMWare boxes have 40 lanes as do both the E5-2667 v3 in my TrueNAS Core Box (meaning 80 lanes - lots of PCIe goodness - and I make good use of them)

The 3008 SAS is good (I use one) - you will need to flash to IT Mode
As far as I am aware the broadcom networking is also good - but I don't use / have them so don't actually know

Have you considered some older hardware, second hand from a breaker. Thats what I did (although I bought the motherboard new). I bought CPU, Memory etc from Bargain Hardware in the UK (no association) who seem to be a proper dismantler of server systems. You can get an awful lot of CPU for your £ in comparison to buying new.

 

[tartan-prophesy]

That's good to know they are well supported. I specced out an EPYC system, it was heading for a similar price until the RAM! That said it gives an absolute ton of flexibility for the future which is nice. I'll check out Bargain Hardware and see what I can find.