Scale Setup with ECC - AMD Ryzen PRO or Xeon?

[pixelwave]

I am looking for a new build and from price / performance point of view the Ryzen (AM4) platform still looks quite intriguing.

But my biggest question would be what the current status of ECC support specifically for Truenas Scale is? Do Ryzen (Pro / AM4) CPUs run stable with full ECC support currently?

What I definitely want to improve over my current system is:

- ECC Ram (32GB)
- CPU Power (4C/8T)
- SFP+ or QSFP+ (10GbE integrated or Mellanox Conncet X3 or similar)

... as well as a proper M2 NVMe Storage pool (RAIDZ-1). For that ideally a mainboard supporting proper bifurcation (4x4x4x4x) on a x16 pcie slot would be necessary. Then I could plug in up to four cooled M2.NVMe cards in one M.2 Adapter Card (ASUS Hyper M.2 x16 Gen 4-Karte or ASRock Hyper Quad M.2 Card).

See planned drive layout:

Ideally idle power consumption for this 24/7 setup should be around 50 W or better below. Load should ideally not exceed 70-80 W. GPU not required.

Mainboard form-factor mATX or even mini-ITX (if onboard 10GbE).

 

[Arwen]

There are less choices for AMD AM4 server motherboards, than their are Intel. (But I too would prefer AMD...) I don't have any real advice, (though I ended up with a AMD Epyc mATX motherboard).

Make sure your WD Reds are not SMR... most new 2TB ones would likely be. Either verify or select WD Red Pluses.

Some lower end CPUs and motherboards don't support bifurcating a x16 PCIe slot into 4x4x4x4. So you may need to use a 4 x M.2 NVMe board, that has a built in x16 PCIe switch to 4x4x4x4. These are of course more expensive.

 

[pixelwave]

Make sure your WD Reds are not SMR... most new 2TB ones would likely be. Either verify or select WD Red Pluses.

Isnt "SMR" for HDD's? Since the 2TB drives are NVMe drives ...

Currently I have those two configs for CPU/MB in mind - the Xeon combination being 300€ more expansive then the AMD:

Intel Xeon E-2336, 6C/12T, 2.90-4.80GHz
ASUS P12R-M/10G-2T

- or -

AMD Ryzen 5 PRO 5650G, 6C/12T, 3.90-4.40GHz
ASRock X570M Pro4

 

[snave]

I have an Asrock X570D4U-2L2T with AMD 5600X with 64GB ECC dram and a bunch of shucked WD 12Gb disks. Think it does everything in your spec. 4x4x4x4 division of x16 PCIe 4 slot, ECC, built in 2*10GBe and 2*1GBe Intel nics etc. Was very reasonable cost when I built it 16 months ago, now of course everything is MUCH more costly.

Edit: You might struggle to get the idle power consumption of 50W without compromising some of the other specs. My system as above (which has two M2 boot NVME and an H310 storage controller) uses 80w at idle measured at the wall and can hit 180w during boot.

 

[pixelwave]

Thanks for the info!

That's why the "AMD Ryzen 5 PRO 5650G" seems intriguing. Cezanne with 45W-Mode. Only downside is PCIe 3.0 instead of 4.0. But the main reason I am looking at the "PRO" versions is because most manuals mention ECC support only for PRO models. Just to be on the safe side ...

80W idle ... with how many 3.5 HDDs in total? My total drive power consumption should be around 25-35W.

 

[snave]

Thanks for the info!

That's why the "AMD Ryzen 5 PRO 5650G" seems intriguing. Cezanne with 45W-Mode. Only downside is PCIe 3.0 instead of 4.0. But the main reason I am looking at the "PRO" versions is because most manuals mention ECC support only for PRO models. Just to be on the safe side ...

80W idle ... with how many 3.5 HDDs in total? My total drive power consumption should be around 25-35W.

That is with 7*12Gb HDD

 

[somewhatdamaged]

My 3950x and its replacement 3400G Ryzen Pro, both support and show all 64GB of my RAM as ECC. No difference from what i can see

 

[Arwen]

Isnt "SMR" for HDD's? Since the 2TB drives are NVMe drives ...
...

Thanks for the clarification. Yes, SMR is for HDDs only. I forgot they had "Red" NVMe drives. So you are good on that issue.

 

[pixelwave]

My 3950x and its replacement 3400G Ryzen Pro, both support and show all 64GB of my RAM as ECC. No difference from what i can see

Yes ... that I have read as well. Just when it comes to error reporting I heard there are some issues with the AM4 implementation. Or has that been fixed?

 

[pixelwave]

Ah ok ... according to ASUS the Ryzen 5000 series also "officially" supports ECC - not depending on the "PRO":

*5000 series supported without explicit mentioning of PRO requirement.


So in comparison:

AMD Ryzen 5 5600
+ 24x PCIe 4.0 (16+4+4)
- No Remote Control via DMTF DASH
+ 190€


AMD Ryzen 5 PRO 5650G
- 24x PCIe 3.0 (16+4+4)
+ Remote Control via DMTF DASH
- 240€

... also the PRO version has TDP-down .. but the "normal" version can potentially be underclocked?

The PCIe4 version of the 5600 would allow to utilize the full speed of the NVMe SSD as well as the 40GbE more easily.

 

[pixelwave]

So I startet the build .. you can read about my „adventure“ here:

AMD Ryzen with ECC and 6x M.2 NVMe build

UPDATE - For the final BOM and build please skip to: https://www.truenas.com/community/threads/amd-ryzen-with-ecc-and-6x-m-2-nvme-build.104574/post-724734 Read below in order to follow my entire journey starting with the initial build: ----------- I am wrapping up my final hardware choice to...

www.truenas.com

 

[ChrisRJ]

Cezanne with 45W-Mode.

Are we talking about TDP here?