Considerations when choosing server parts?

[MGone]

Hi there, I've been looking for parts for my first server build and have a few questions.

Firstly, the purpose! I'm looking to run a file sharing server for up to 4-10 concurrent users, 2-4 local and 2-6 remote working. I did consider cloud services but have found manipulating CAD and Media files (large Photoshop, Premiere and InDesign) painful unless copied locally.

I was considering a pre-built unit such as the 6-8 bays offered by Synology or Qnap, but would like to keep the unpopulated cost to under £800 if possible. Also, I enjoy PC building and the challenges of putting together a suitable server! :) I have been leaning towards ZFS for its snapshot functionality. (Backup is separate to this discussion...being handled onsite and via cloud services, OneDrive and SharePoint)

So, these are my thoughts so far and I'd appreciate any feedback from folk who have been doing this for a while.

1. £117 - Case: Fractal Design Define XL R2 Titanium - (I'm not too worried about looks or footprint, but do not have a rack. So a headless box is my aim, ATX or smaller but if e or ee ATX is required it's not a problem)
2. £528 - Storage Disks: 6x 4TB Ironwolf Pro/WD Red Pro/Toshiba N300 (SATA III)
3. £37 - Boot Disk: Crucial BX500 480 GB (SATA III) (would probably switch to M.2 SATA or NVMe if not needed for Slog/L2ARC)
4. £124 - SLOG M.2 NVMe: 2x 500GB Samsung 970 EVO PLUS (not 100% sure this is needed nor how I achieve power loss prevention)
5. £134 - Memory: 2x 8GB 3200MHz DDR4 ECC at fastest speed board and budget will allow (DDR3 or 4)
6. £80 - PSU: Seasonic Focus GX 550W, Full Modular, 80 Plus Gold

Which leaves about £300 for the MBoard & CPU (plus any add-ons).
Which is where things get more uncertain for me and I could do with some help assessing priorities and what I should look for being on the board vs what I can add in with PCIe.

Motherboard Requirements (budget ~£200)

• CPU support
  This seems to boil down to socket and chipset with the later pretty much determining the board functionality with regard to the other requirements below. At the moment I'm currently focusing my searches around FCLGA-2011-3 sockets and e-5 2600 chips, but the majority of 'affordable' boards seem to run the X99 chipset with lots of bells and whistles and waaaaay too much RGB! :P Would I be better aiming for the C612, even if that means higher cost boards like the Super Micro MBD-X10SRL-F-O? The other option would be something like this HP Z420 Workstation motherboard, but I'm concerned about the availability of drivers for FreeBDS. Is this something to worry about or are they easily found online?
• ECC Memory Support
  My main question here is if this is CPU dependant? Some of the X99 boards state they do support ECC, but I'm assuming that this is only with the E3-xeons and not i7's that also fit this 2011-3 sockets
• IPMI support
  A function of the Chipset? I noticed there are a few X99 boards with IPMI support, but is this the same as that provided on the C612 boards?
• Storage support (possibly extra cost ~£90-400)
  I've noticed that there is quite a variety of SATA III support from 4 to 12 drives (although I'm guessing that some of those you would loose if you utilise on board M.2 sockets and multiple PCIe slots. Is there any advantage to adding a PCIe expansion card such as this LSI SAS 9207-8i and if so, should I really be looking to SAS 12Gb/s and drives to match? I'm guessing that these more expensive 4TB Seagate Exos 7E8 SAS Hard Drives would work, but would you benefit from the full 12Gbs speed via the PCIe lanes?
• M.2 NVMe support or M.2 SATA (possibly extra cost ~£15)
  How important is it to have M.2 NVMe? I'm guessing M.2 SATA would be absolutely fine for SLOG & L2ARC if needed?
  Do you loose much if anything by utilising a PCIe board? Something like this Sabrent NVMe M.2 SSD to PCIe X16/X8/X4 Card
• Networking (possibly extra cost ~£90)
  I currently run a 1GbE over CAT5e but hope to upgrade the cabling to cat7 in the next 2-3yrs. So initially 2x or 4x 1GbE ports would be fine as I have managed switches. This needs to utilise Intel controllers. If the board doesn't support 2.5GbE I could add this via PCIe with something like this QNAP Card....I'm not sure I trust the nonbrand ones, but that could just be me worrying so again all suggestions welcomed.
• Video support (possibly extra cost ~£6)
  I'm unclear if it is better to go for an on chip solution or simply add a cheap second hand card like this Nvidia Quadro NVS 290

CPU Requirements (budget ~£100)
There's loads of Xeons out there and I lack the experience to fully understand the benefits of increasing cores/threads and cache except that I'm assuming higher is better? So I chose to base my decision around CPUMark scores and felt the E5-2660v3 seemed a good starting point at 13,080. This fits within budget too...but again, I'd love to hear the thoughts of you folks!

It also feels like there are some great bargains to be had with non ATX format boards, some of them more 'proprietary' than others, and I may consider one of these as a workbench project, but unless there are some easy to comprehend guides to the higher end enterprise sockets and boards I will have to skip those for now.

Finally, after such a long post I'd like just say thank you to anyone who has taken the time to reach the end. There's so much information out there it's a real privilege to be able to come to a forum like this and pick the brains of those who've already invested a great deal of time working through problems to help others. So, many thanks and I look forward to hearing any suggestions you have.

 

[MGone]

...also, am I better to have 2vdevs of 4 disks (raid z2) giving total 4 disk capacity, or a single vdev of 8 disks (raid z2) giving 6 disks of capacity? better is obviously subjective, but I'm thinking about future expansion and it would be easier to add 4 disks than 8....it just feels like I'm loosing a lot of capacity with z2 and 4 disks...I would probably be better in that case just to mirror?

 

[Ericloewe]

Would I be better aiming for the C612, even if that means higher cost boards like the Super Micro MBD-X10SRL-F-O?

Probably.

My main question here is if this is CPU dependant? Some of the X99 boards state they do support ECC, but I'm assuming that this is only with the E3-xeons and not i7's that also fit this 2011-3 sockets

Yup, you'd need a Xeon E5 for ECC support.

A function of the Chipset?

Nope, of the Baseboard Management Controller. Think of it as an independent computer with tentacles deep in the host system. Not the sort of thing you'll typically find on consumer stuff like X99 boards, but not unheard of either.

I've noticed that there is quite a variety of SATA III support from 4 to 12 drives

10 SATA ports is the standard complement supported by the PCH. Anything else is going to be either onboard LSI SAS2/SAS3 (typically good) or crap SATA controllers from the likes of Marvell (eh) or JMicron (eww).

Is there any advantage to adding a PCIe expansion card such as this LSI SAS 9207-8i

Maybe. Need to use a SAS expander? You'll need a SAS controller. Need more ports? The only viable solution is an LSI SAS controller.

should I really be looking to SAS 12Gb/s and drives to match

Not unless you enjoy paying more than you have to for your hard drives. Also, the SAS2308 as used onboard some motherboards of this generation and the LSI SAS 9207 is "only" SAS2 for 6 Gb/s per port.

but would you benefit from the full 12Gbs speed via the PCIe lanes?

Simple answer: no. Long answer: Mechanical disks are slow as molasses. If you want speed, you go for NVMe SSDs, which leave anything SAS in the dust and are considerably cheaper for many/most scenarios. "Performance" spinning rust is obsolete.

I'm guessing M.2 SATA would be absolutely fine for SLOG & L2ARC if needed?

Well, no. Especially not SLOG. Not to say that you need either of them, but using SATA for L2ARC or SLOG is a weird decision in 2022. Not to mention that M.2 SATA is very much obsolete.

Do you loose much if anything by utilising a PCIe board? Something like this Sabrent NVMe M.2 SSD to PCIe X16/X8/X4 Card

Well, with that one you lose anywhere from 0x to 12x PCIe lanes, depending on the slot you use it in. Considering that you're looking at higher-end gear that should be capable of bifurcating a PCIe x16 slot to x4/x4/x4/x4, one of the Quad M.2 cards with no PCIe switch should work well for you (e.g.: Asus' model is nice). Do check that the chosen motherboard natively has this option in the setup menu. It can be typically activated manually with a custom BIOS image and/or manual tweaking of the EFI settings, but it's best if it's natively supported by the setup utility.
More expensive versions exist which use a PCIe switch for ~100% compatibility with any x16 slot, at the cost of extra power and cash (e.g.: one Supermicro model).

I currently run a 1GbE over CAT5e but hope to upgrade the cabling to cat7

Consider fiber instead. It's no boogeyman if you buy it factory-terminated, it's cheaper and far more capable.

If the board doesn't support 2.5GbE I could add this via PCIe with something like this QNAP Card....

Not wishing to imply that 10 GbE over twisted pair is worth investing in (it's mostly not), but at those prices you might as well pick up an Intel X550-T2 and get 5 GbE and 10 GbE for not much more cost.

I'm unclear if it is better to go for an on chip solution or simply add a cheap second hand card like this Nvidia Quadro NVS 290

If you have a BMC and IPMI, you won't need anything. Otherwise, you will need something.

So I chose to base my decision around CPUMark scores and felt the E5-2660v3 seemed a good starting point at 13,080. This fits within budget too...but again, I'd love to hear the thoughts of you folks!

It's a good part at current prices, but it's not going to be fantastic at single-thread workloads.

Thing is... For six disks and 16 GB of RAM, I'm not sure you want an LGA2011-3 system. Maybe if things end up being cheaper, but otherwise think more along the lines of an X11SSM-F with a Xeon E3 v5/v6.
Your workload is slightly too vague for more detailed suggestions. Would you be using SMB shares and are you thinking of editing stuff in-place on the server?

 

[Etorix]

Firstly, the purpose! I'm looking to run a file sharing server for up to 4-10 concurrent users, 2-4 local and 2-6 remote working. I did consider cloud services but have found manipulating CAD and Media files (large Photoshop, Premiere and InDesign) painful unless copied locally.

So, just file sharing, presumably over SMB? The use case is important.

So, these are my thoughts so far and I'd appreciate any feedback from folk who have been doing this for a while.

1. £117 - Case: Fractal Design Define XL R2 Titanium - (I'm not too worried about looks or footprint, but do not have a rack. So a headless box is my aim, ATX or smaller but if e or ee ATX is required it's not a problem)

The Node 804 is possibly more suited for a NAS, if noise is not a concern.

1. £37 - Boot Disk: Crucial BX500 480 GB (SATA III) (would probably switch to M.2 SATA or NVMe if not needed for Slog/L2ARC)
2. £124 - SLOG M.2 NVMe: 2x 500GB Samsung 970 EVO PLUS (not 100% sure this is needed nor how I achieve power loss prevention)

PLP is a function of the drive. For SLOG you need Optane or a data centre SSD, not consumer drives. But you do not need a SLOG for file sharing.
Using NVMe for boot frees up one SATA port for storage.

1. £134 - Memory: 2x 8GB 3200MHz DDR4 ECC at fastest speed board and budget will allow (DDR3 or 4)

16 GB is the minimum. Get more if you want performance.
Do not even think of a L2ARC until you are over 64 GB RAM.

1. £80 - PSU: Seasonic Focus GX 550W, Full Modular, 80 Plus Goldhttps://www.amazon.co.uk/Seasonic-E...sonic+focus+px-550+80plus,computers,42&sr=1-1

Too small, see here:

Proper Power Supply Sizing Guidance

I've seen about 1,000 threads like this one where people decide that they can power a dozen hard drives off a 360 watt supply. DO NOT DO THIS. I've seen another 1,000 threads where people decide to buy the cheapest power supply that they can...

www.truenas.com

Which leaves about £300 for the MBoard & CPU (plus any add-ons).
Which is where things get more uncertain for me and I could do with some help assessing priorities and what I should look for being on the board vs what I can add in with PCIe.

Xeon E5, W-2000 or Scalable support RDIMM, which is good for capacity.
But for 16 GB RAM, and no more than 64 GB, you can as well go for a X11SCH-L and a Core i3-9100F (supports ECC UDIMM).
Add a Chelsio/Solarflare/Intel SFP+ card for 10 GbE.

It also feels like there are some great bargains to be had with non ATX format boards, some of them more 'proprietary' than others, and I may consider one of these as a workbench project, but unless there are some easy to comprehend guides to the higher end enterprise sockets and boards I will have to skip those for now.

If you go for a consumer case, get a standard motherboard size.
If you find a second-hand server which fits your needs, go for any proprietary sized motherboard which may be in there.

 

[MGone]

Thanks Ericloewe & Etorix! Your replies have clarified a few things and given me some options to think about when putting together my final spec.

Primary use will be SMB shares (everything from small office files to larger CAD and even larger Multimedia files) but I'm also interest in the use of VMs as system mirrors for testing upgrades, and possibly for hosting a VM for using single seat software that multiple users might need access to and they would otherwise have to switch machine to use. I'm not looking to get around licenses here, as there would only be one user at a time it seems that a windows VM on the server might work well for this purpose as it's not particularly resource intensive, it's more about access. I don't imagine more than one or two VMs running concurrently

Anyway, I'm off do do some more digging and reading....might put some price watches on some WD Red plus and Ironwolf drives though as I've been reading about the advantages of mirrors so I think 3 vdevs of 2x4TB mirrored with another 4TB allocated as a hot spare might be the way to go. That way I have sufficient storage and performance initially with the possibility of easily expanding the pool with another vdev in the future.

My aim will be to have something that meets the simple needs initially but maybe have the processor overheads and empty slots to add more memory, slog and L2ARC as or if required.

 

[Ericloewe]

Ok, so some thoughts:

• You'll definitely want more RAM if you'll be doing some relevant VMs. Aim for a minimum of 32 GB.
• A couple of extra CPU cores would also be useful.
• An X11SCH-F plus a Coffee Lake Xeon-E CPU could get eight cores and up to 128 GB of RAM, but it'll probably be more expensive than some of the used stuff (but lower-power and, in some cases, faster).
• Definitely no L2ARC before you're at 64 GB of RAM, and that's if it turns out to be useful
• Most likely no SLOG, unless you have Mac OS clients.

 

[MGone]

Ok, so some thoughts:

• You'll definitely want more RAM if you'll be doing some relevant VMs. Aim for a minimum of 32 GB.

Will do...with boards that support 8 memory slots I would be happy plumping for 32 up front with space to increase to 64 if and as when needed.

• A couple of extra CPU cores would also be useful.
• An X11SCH-F plus a Coffee Lake Xeon-E CPU could get eight cores and up to 128 GB of RAM, but it'll probably be more expensive than some of the used stuff (but lower-power and, in some cases, faster).

I think I'm leaning towards the Haswell or Broadwell at 14-18 cores but am a little unclear if I should prioritise cores, cache, speed or low TDP...I have a kind shortlist here.

• Definitely no L2ARC before you're at 64 GB of RAM, and that's if it turns out to be useful
• Most likely no SLOG, unless you have Mac OS clients.

Cheers, that makes sense...it's easy to over complicate things...as it is I'm feeling I'm over-spec'ing quite a bit and am wondering if I might be able to use this as a media server too? I had originally planned to build a smaller SSD based unit to act as a standalone plex server, that fetched files from this server, but did any processing and transcoding locally. I'm now wondering if I could just do that from this box? Definitely not project creep honest!

 

[Etorix]

I think I'm leaning towards the Haswell or Broadwell at 14-18 cores but am a little unclear if I should prioritise cores, cache, speed or low TDP...I have a kind shortlist here.

SMB is single-threaded and benefits from high clock speed, so extra cores would be for VMs. I suspect that your short-list is skewed too high in the stack and that you may actually benefit from CPUs with less cores and higher clock.

 

[ChrisRJ]

[..] low TDP [..]

With somewhat current (edit: meaning roughly ten years old or younger) CPUs TDP is about thermals but not power consumption, unless of course it is running full throttle. But that is hardly the case. In general newer CPUs are more energy efficient than older ones. So a newer but higher TDP CPU will usually consume less power than an older CPU with lower TDP.

 

[MGone]

SMB is single-threaded and benefits from high clock speed, so extra cores would be for VMs. I suspect that your short-list is skewed too high in the stack and that you may actually benefit from CPUs with less cores and higher clock.

Something like these?

Model	Cores/Threads	Speed (Turbo)	TDP	Cache	CPU Mark	Price
E5-2643 v3	6/12	3.40 (3.70) GHz	135	20 MB	10,395	£194
E5-2643 v2	6/12	3.50 (3.80) GHz	130	25 MB	9,083	£67
E5-2667 v3	8/16	3.20 (3.60) GHz	135	20 MB	12,354	£178
E5-2687W v2	8/16	3.40 (4.00) GHz	150	25 MB	12,241	£249
E5-2667 v2	8/16	3.30 (4.00) GHz	130	25 MB	12,218	£139
E5-2687W v3	10/20	3.10 (3.50) GHz	160	25 MB	14,787	£211

Having gone through lots more listings I'm seeing the high speed v3's and above going above my budget unless I drop to 4 cores...I'm also wondering if I'll take a big hit by going down to the v2's because they only support DDR3 memory?