Need suggestions for upcoming build!

[Nassi]

Hi,

I've been trying to choose a motherboard and processor for my upcoming Truenas Core build.

I am completely lost with these different options. I have looked at different xeon-D, Atom and Embedded epyc models.
The more I've looked at these, the more I'm lost with the different options.

Budget: 500€ - 1000€ (motherboard + processor)

Location: EU

Purpose of use:
Home server. Mainly a media server for movies, tv-series and music.
Also any family personal files such as photos etc.

Only NAS functions.

I want to run docker and VM's on a separate proxmox device. In this case, the proxmox device can be used as a "playground"
and the NAS can perform its tasks as stable and reliable as possible.

Because in home use the NAS is just idling approximately 95% of the time and the price of electricity is high today in Europe,
so idle consumption should be as low as possible. No need to squeeze a last drops of power consumption
however. Aim for idle consumption is 50w-100w. (with 6 HDD and possible pcie card(s))
Less is more in this case!

I already have a 24bay 4U rack case with 6gbit backplanes from my previous unRAID build.

What I want:
- Will start with 6 wide RaidZ2
- Server grade hardware (prefer supermicro)
- 10gbit NIC, doesn't matter is it embedder or pcie card if total power consumption is roughly the same. I think SFP+ would be a better option than rj45?
- ECC ram. Propably used (cheap) registered ram from ebay. Minimum 128GB, taking suggestions also for this!
- HBA card (pcie) Propably start with 8i card but want to have an option to swap it to 24i card if I will populate my whole server someday.

What this server should do:
- Powerful enough to serve files to all users and running services without any hiccups. At same time not to be an overkill.
- I would like to have an encrypted pool(s). That might effect to cpu choise?

Is any Atom powerful enough for my needs, should I pick some xeon-D instead?
It doesn't have to be embedded system.

I believe I have found two candidates which meet the given criteria and are very available. Of course, I will accept other suggestions with an open mind!

1. Supermicro X12SDV-4C-SP6F
I am concerned about compatibility since the motherboard is so new. Will SFP28 (SoC) work with Core?
I also couldn't find actual power consumption readings. Does anyone have experience with this?

2. Supermicro M11SDV-4C-LN4F
Idea would be use x16 to x8x8 bifurcation card to add external 10bit nic and HBA. Does this sound a good option?
Taking suggestions for good known bifurcation card.

Bonus question:
Do you prefer embedded or external cards with NIC and HBA? Will it make difference in total power consumption?

 

[QonoS]

Maybe let me bring you down to earth for your use case:
"
Home server. Mainly a media server for movies, tv-series and music.
Also any family personal files such as photos etc.
"

Are you sure you need 10Gbit/s ?
2.5Gbi/s could be enough and it requires less power and can be realized with Ethernet. No need to use Fiber.
Lots of MoBos come with 2.5G onboard, it is an option. I had 10G and now I am down to 1 (one) Gbit/s because I really do not need more and my use case is very similar. Also network equipment with 10G eats power and should you should count that in 24/7 too.

Are you sure you need an HBA ?
16-20TB SATA Enterprise grade disk are cheap per TB. You sure need more than 6 of these? HBA consume power. They definitely kill power efficiency. Less is more and if you can eliminate that HBA you will save like 8-12W 24/7.

 

[Whattteva]

Are you sure you need 10Gbit/s ?
2.5Gbi/s could be enough and it requires less power and can be realized with Ethernet. No need to use Fiber.
Lots of MoBos come with 2.5G onboard, it is an option. I had 10G and now I am down to 1 (one) Gbit/s because I really do not need more and my use case is very similar. Also network equipment with 10G eats power and should you should count that in 24/7 too.

I'd argue you don't even need 2.5G. I'm only using 1G and this is on a server that has 13 mostly idle and 2 very active VM's.

Are you sure you need an HBA ?
16-20TB SATA Enterprise grade disk are cheap per TB. You sure need more than 6 of these? HBA consume power. They definitely kill power efficiency. Less is more and if you can eliminate that HBA you will save like 8-12W 24/7.

Not just the HBA, the disks themselves add like 7W each. If OP's goal is power efficiency, they should aim for higher density disks.

I'd also add that 90% of users don't store anything sensitive enough to want/need encryption (which takes up more power also).
Also, one of the semi-common posts I've seen is a user that has self-imposed ransomware due to losing/forgetting their encryption keys/passphrase.

 

[Nassi]

I know that I don't NEED a 10Gbit connection. I've been using unRAID for years with a 1Gbit connection. However, I have noticed for example, editing large bluray files is significantly slower from my computer than through docker which has been running in unRAID server. (data has to go back and forth with a 1gbit connection), so the additional speed would not hurt now that the purpose is to build a pure NAS and run everything else on a separate proxmox server or on my desktop computer.
In short, 10Gbit is not necessary, rather "why not" because it is mainstream in server hardware. I haven't seen server hardware with a 2.5Gbit connection either? The intention is to build everything in a dedicated rack cabinet.

I'm starting with 6 x Exos X20 20TB hard drives, they'll definitely last me a really long time. However, I want to keep the option from a hardware perspective to add new pool or vdev files if needed. I have also not ruled out the idea of farming the Chia cryptocurrency in the same 24-bay enclosure, but in a different pool.

As I said about the power consumption, the last drops are not meant to be squeezed out. 16-core epyc idling at 150 watts is completely overkill, and correspondingly, 2-core Atom with 10w TDP does not perform the desired tasks without problems. So a golden mean should be found.

 

[QonoS]

A Intel Socket 1200 Motherboard with 6-8 SATA Ports could be a solution.

There are Xeons with 4-8 Cores available that do not suck too much power and can idle well.
Exos HDD can be connected directly and if you do Chia Mining with another HDD pool you can still add a HBA later.

ASUS P12R-M/10G-2T even has 10G onboard.

ASUS P12R-M/10G-2T ab € 367,50 (2024) | Preisvergleich Geizhals EU

✔ Preisvergleich für ASUS P12R-M/10G-2T ✔ Produktinfo ⇒ Formfaktor: µATX • Sockel: Intel 1200 • Chipsatz: Intel C252 • RAM: 4x DDR4 DIMM, dual PC4-25600E/DDR4-32… ✔ Intel Xeon ✔ Testberichte ✔ Günstig kaufen

geizhals.eu

More boards listed here.

 

[Etorix]

I already have a 24bay 4U rack case with 6gbit backplanes from my previous unRAID build.

If this is a SAS backplane with expanders, a -8i HBA (or even -4i) is all you need to fill it with drives.
Of course, once filled with spinners it will use over 100 W.

What I want:
- Will start with 6 wide RaidZ2
- Server grade hardware (prefer supermicro)
- 10gbit NIC, doesn't matter is it embedder or pcie card if total power consumption is roughly the same. I think SFP+ would be a better option than rj45?
- ECC ram. Propably used (cheap) registered ram from ebay. Minimum 128GB, taking suggestions also for this!

All correct, including SFP+ being lower power. Solarflare SF7122F NICs go for $50 on eBay, Chelsio T520-CR for double that.

Is any Atom powerful enough for my needs, should I pick some xeon-D instead?

Atom C3000 comes with one x4 PCIe slot. With a HBA and on-board 10 GbE (likely RJ45) it could just fit.
Xeon D-1500 (X10SDV series) would do better, as it provides 16 lanes of PCIe. Some boards have 10 GbE onboard ('T' in the final segment), sometimes even as SFP+ ('TP'). In this case, even a mini-ITX board would do, with the HBA in the x16 PCIe slot. If possible, do not bother with bifurcating risers: They exist, and they work, but add costs and trouble to fit everything in the case.
So, for instance, and looking for a European seller, this could do if you forego SFP+:

Supermicro X10SDV-F | Intel 8C D-1541 | PCIe x16 M.2 IPMI ITX DDR4 Server Board | eBay

Supermicro X10SDV-F PCIe x16 3.0 / max. Up to 64GB Unbuffered ECC/non-ECC UDIMM, DDR4-2133MHz, in 4 DIMM slots. 2400/2133/1866/1600MHz ECC DDR4 ECC RDIMM and ECC/Non-ECC UDIMM. 1 x Supermicro X10SDV-F.

www.ebay.de

Some X10SDV boards come in "flex-ATX" size (width of micro-ATX, short depth of mini-ITX) with two x8 slots. Perfect to fit a NIC and a HBA!
Some flex-ATX X10SDV board with a LSI HBA on board ('7' in the final segment of the name).
'-7TP4F' or -7TP8F' boards come with everything you need—though you may end up paying more than a board and add-in cards.

Supermicro Mainboard X10SDV-4C-7TP4F Single MBD-X10SDV-4C-7TP4F-O

Supermicro Mainboard X10SDV-4C-7TP4F Single, MBD-X10SDV-4C-7TP4F-O, EAN 0672042227198 günstig - ab 0 € portofrei kaufen

www.jacob.de

(Great lesson here: New comes out cheaper than second-hand eBay offers!)

I believe I have found two candidates which meet the given criteria and are very available. Of course, I will accept other suggestions with an open mind!

1. Supermicro X12SDV-4C-SP6F
I am concerned about compatibility since the motherboard is so new. Will SFP28 (SoC) work with Core?
I also couldn't find actual power consumption readings. Does anyone have experience with this?

CORE should have the ice driver for the embedded 823L NIC. This is overkill for a simple home NAS and comes with (unspecified) higher consumption that X10SDV. But I admit that a brand new X12SDV-4C-SP6F may be a competitive option over X10SDV from a financial point of view.

2. Supermicro M11SDV-4C-LN4F
Idea would be use x16 to x8x8 bifurcation card to add external 10bit nic and HBA. Does this sound a good option?
Taking suggestions for good known bifurcation card.

Look here, but think about how you'd fix everything in place:

PCIe Bifurcation Adapters - x8x8

C-Payne PCB Design

c-payne.com

 

[Nassi]

A Intel Socket 1200 Motherboard with 6-8 SATA Ports could be a solution.

There are Xeons with 4-8 Cores available that do not suck too much power and can idle well.
Exos HDD can be connected directly and if you do Chia Mining with another HDD pool you can still add a HBA later.

ASUS P12R-M/10G-2T even has 10G onboard.

ASUS P12R-M/10G-2T ab € 367,50 (2024) | Preisvergleich Geizhals EU

✔ Preisvergleich für ASUS P12R-M/10G-2T ✔ Produktinfo ⇒ Formfaktor: µATX • Sockel: Intel 1200 • Chipsatz: Intel C252 • RAM: 4x DDR4 DIMM, dual PC4-25600E/DDR4-32… ✔ Intel Xeon ✔ Testberichte ✔ Günstig kaufen

geizhals.eu

More boards listed here.

Looks like these don't support ECC or registered memory? Other E3 platforms seem to be limited to 64GB of RAM?

If this is a SAS backplane with expanders, a -8i HBA (or even -4i) is all you need to fill it with drives.
Of course, once filled with spinners it will use over 100 W.

CORE should have the ice driver for the embedded 823L NIC. This is overkill for a simple home NAS and comes with (unspecified) higher consumption that X10SDV. But I admit that a brand new X12SDV-4C-SP6F may be a competitive option over X10SDV from a financial point of view.

The case has a backplane for each disk row (SFF-8087)

Yeah, the SFP28 is absolutely overkill, definitely no real use in my case! however, that seems to be the standard for new stuff.
In terms of price, X12SDV and X10SDV are in the same class, so a significantly newer architecture would be more attractive.

 

[Etorix]

Looks like these don't support ECC or registered memory? Other E3 platforms seem to be limited to 64GB of RAM?

Xeon E3 / E-2000 use ECC UDIMM, not RDIMM. These are basically consumer Core CPU with ECC support.
The older E3 are indeed limited in total RAM—but 64 GB was plenty in their days.

The case has a backplane for each disk row (SFF-8087)

6 rows of 4, and 6 connectors?
If so, the cheap server rack (InterTech?) will cost you either a SAS expander card (preferred) or multiple SAS HBA (a -16i being more expensive than two -8i). Reverse breakout cables may allow to use SATA port from the motherboard.

Yeah, the SFP28 is absolutely overkill, definitely no real use in my case! however, that seems to be the standard for new stuff.
In terms of price, X12SDV and X10SDV are in the same class, so a significantly newer architecture would be more attractive.

X10SDV (D-1500) is known to sip power at idle.
X11SDV (D-2100) idles over 50W, which is around the level of the most powerful D-1587 at full load…
In the X12SDV line, the best indications I've found are that D-2700 is high power (70-75 W idle) but D-1700 may be better than D-2100 (25-25W idle for a D-1749T).
Tough call…

 

[Etorix]

Best deals come from forums… Second-hand X10SDV-TP8F (onboard SFP+) for 300 E:

EU - [WTS] Several X10SDV Mobo + Rasp PI 4B 8GB

Hello, I'm renewing some of my systems, so I have available: - X10SDV-4C-TLN2F : 250€ SOLD - X10SDV-4C-TLN2F : 250€ SOLD - X10SDV-TP8F : 350€ SOLD - X10SDV-TP8F : 300€ - X10SDV-6C+-TLN4F : 400€ SOLD - X10SDV-F : 350€ - Ubiquiti EdgeRouter 12 : UISP EdgeRouter 12 – Ubiquiti Inc. 160€ - 3...

forums.servethehome.com

 

[MrGuvernment]

Maybe let me bring you down to earth for your use case:
.............
Are you sure you need 10Gbit/s ?
2.5Gbi/s could be enough and it requires less power and can be realized with Ethernet. No need to use Fiber.
Lots of MoBos come with 2.5G onboard, it is an option. I had 10G and now I am down to 1 (one) Gbit/s because I really do not need more and my use case is very similar. Also network equipment with 10G eats power and should you should count that in 24/7 too.
....................

Considering a single NVMe can saturate 10Gb in certain situations, these days why not. 2.5/5Gb is a consumer stop gap. Buy a used Brocade switch off Ebay with 4 x 10Gb SFP and your set. You can buy used enterprise gear for cheap that is still 100% more reliable than most consumer grade switches and routers out there and do now consume excessive power either.. (EU maybe a little different)