Hardware advice please (yet another thread)

[Lord Baldrick]

I'm thinking of running Proxmox for virtualisation flexibility for things like home assistant / plex / jellyfin (currently running on an assortment of Raspberry Pi's). Maybe also syncthing (which I've never run but it sounds useful for me). Oh and probably a time machine server to back up my MacBook (soon to be replaced by a Framework 16 ). TrueNAS Core should be fine from what I've read.

I currently have a QNAP NAS with 2*6Tb mirror and 2*3Tb mirror (one of the drives in this mirror has been reporting its degraded for years but seems to still be working) but it just feels super slow these days (it was never rapid) and I'd like to extend my storage capability (I'm storing more media than I used to - ripping DVDs, etc. so am aiming for >14Tb usable space). I'd also like whatever I build to last ten years or more, allowing for HDD upgrades / fix on fail replacements.

So I'm thinking (having played a little with uk.pcpartpicker.com):

• Supermicro X10SRL
• Xeon E5-2680 (14 cores, 2.4GHz)
• Noctua NH-U9DXi4 cooler
• 4 * 32Gb ECC RAM (probably registered but change my mind?) - Crucial CT4K32G4LFQ4213 128 GB (4 x 32 GB) Registered DDR4-2133 CL15 Memory
• 4 * Toshiba N300 18Tb in RAIDZ2

I plan to reuse an existing midi tower with hot-swap capability (although I've never actually tested it), a new 650W PSU I have hanging around as well as an old graphics card (GTX285?) & monitor (if I can fix it) for setup, etc. purposes. I'm thinking I'll stick it in an open cupboard in my garage as it's both out of the way and relatively cool in summer - albeit with a larger temperature range than my house (probably 5 deg C to 35 deg C for the few hideous ultra-hot weeks we've had). I don't have a sensible place to have this machine inside the house though...

All thoughts / advice is appreciated!

 

[danb35]

probably registered but change my mind?

You'd definitely need registered with that CPU/motherboard combination.

 

[Davvo]

Having the drives below 20C is worse than having them run over 38C; frequent, sudden, and/or large thermal variation is extremely harmful to the drives: do take this into consideration (alongside the potential humidity difference).

If I understand correctly you will use this system as backup/media storage (and possibily streaming) only: do consider the option of spinning down the drives if it will enable you to put inside the house (the spin up and down of the drives is way better than 5-15C temps).

The ideal temperature for spinners is between 24C and 36C.

I'd go with a X11 motherboard, just to get a little bit more life in there since used X10 are starting to be a tad old... and you get the bonus of a HTML IPMI, dodging the Java stuff.

With IPMI you don't need a monitor or a graphic card (dedicated or integrated).

 

[Lord Baldrick]

Thanks for the feedback - especially about spinning disk temperatures! I need to think about that... So, X11, then how about (not on part picker this time):

• Supermicro X11SPI-TF (expensive though - can't find one for less than £550!)
  • (I considered X11SPW-TF before I realised it's not ATX size ... how special a case is required for those?)
  • (I've also wondered about X12 motherboards - they don't look much more expensive than X11's but the processors do...)
• Xeon Gold 6150 (18 core, 2.7GHz) (£96 used)
• Noctua NH-U12S DX 3647 (also expensive at £110 but I've used these before and am a fan)
• 32Gb ECC DDR4 2400MHz LRDIMM (or RDIMM?) (£164 for 128Gb on techbuyer.com)
• Same disks as before (I've had Seagate before and v happy but can't ignore the price of the Toshiba drives)

I'd be very grateful for your thoughts!

 

[chuck32]

and you get the bonus of a HTML IPMI, dodging the Java stuff.

X10 also supports that ;)

Supermicro X10SRi-F:

 

[Lord Baldrick]

Looking for places that've got these things in stock, these motherboards are not easy to find. May go for X11SPL-TF as it's almost £120 cheaper and should be delivered two weeks earlier... 8 SATA ports instead of 10 and 1Gb ethernet instead of 10Gb but I'm ok with that.

 

[Lord Baldrick]

In fact where do you all buy your hardware? I'm in the UK and am looking at lambda-tek for the motherboard but they say they don't take returns and it's over 2w for delivery. May as well go ebay... bargainhardware.co.uk and techbuyer.com for the CPU/RAM seem ok.

Also, any thoughts on dual CPU motherboards? e.g. X11DPL for two Xeon gold CPU's. Would two Noctua coolers with fans fit side by side?

 

[Davvo]

Me? eBay and other european sites.

 

[Etorix]

In fact where do you all buy your hardware?

Price engines are a great way to find resellers. For me, it's tweakers.net, hardware.nl and geizhals.de.
For second hand parts, local small ads (also tweakers.net) and the forum at servethehome.com.
eBay is the obvious place, but you have to be wary of occasionally inflated prices.
The hardest places to find are local recyclers (serverzaak.nl in my case).

Obviously, unless a UK resident chimes in, you'll have to find the local equivalents!

Also, any thoughts on dual CPU motherboards? e.g. X11DPL for two Xeon gold CPU's. Would two Noctua coolers with fans fit side by side?

Use, peruse and abuse the Noctua Compatibility Centre… (Basically, D9 should fit, U12 maybe, U14 likely not).

 

[Lord Baldrick]

Thanks both!

 

[danb35]

Also, any thoughts on dual CPU motherboards?

They seem to be more readily available used than single-socket boards of the higher-end sort (i.e., the sort that you'd be able to put a Xeon Gold in), which is why I'm using one, itself an upgrade from another one. I didn't (and don't) really need the extra CPU for the sake of computing power, and I could do without the additional power draw, but that's what was available at reasonable prices that had the other features (IPMI, RAP capability) I wanted.

But I hope you aren't coming up with a cunning plan...

 

[Lord Baldrick]

I’m definitely aiming for a cunning plan.

 

[danb35]

Who gave you permission to turn into an Alsatian?

 

[Lord Baldrick]

Oh God it’s a dream isn’t it, it’s just a bloody dream…

 

[danb35]

Indeed. But it will be over interfrastically.

 

[Lord Baldrick]

I hope I haven’t caused you any pericombobulations.

 

[Lord Baldrick]

Ok, I've got a motherboard coming (X11DPL-i) ... although I've just noticed it only supports CPUs up to 140W TDP and the Xeon Gold 6150 I've been looking at has 165W TDP. FWIW I plan to put Noctua NH-D9 DX-3647 coolers on them both (each rated to 205W TDP).

For the majority of the time I doubt I'll get close to fully loading the CPUs but just in case...

Does anyone else run CPUs with higher TDPs than their mobo's? Would you do it / not do it?

 

[Lord Baldrick]

I've switched to a pair of Xeon Gold 6132's - 14 cores each and 2.6GHz base clock speed. And bang on 140W TDP. Undoubtedly way more than I need ... just the way I like it. Also only £50 each instead of £100 so happy days.

 

[Lord Baldrick]

Cunning plan has changed, I spent most of yesterday fixing up the case then building the server … then fighting with proxmox to pass drives through before I realised an HBA card is basically a *requirement* for virtualising TN. So rather than buy one and wait, I’m trying TN Scale to see if its virtualisation is enough for me…

 

[Davvo]

Yup. Apparently none of us tought of linking you the following resource:

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com