AMD Ryzen with ECC and 6x M.2 NVMe build

[AnK]

But in case of the ASRock X570M Pro4 it should actually work - at least the manual states 16x + 4x config is valid (PCIE1/PCIE3: single at x16 (PCIE1); dual at x16 (PCIE1) / x4 (PCIE3))). So you basically would get the some setup like with my ASRock X470D4U .. if you want to ditch IPMI and go half price.

But even if you ditch IPMI and want to run Bifurcation with 4x4x4x4 you have to run your server headless (without iGPU/APU), aren't you?

 

[pixelwave]

But even if you ditch IPMI and want to run Bifurcation with 4x4x4x4 you have to run your server headless (without iGPU/APU), aren't you?

Yes ... in that case if you go with a consumer board you have no GPU / iGPU with the necessary CPU for 4x4.

But running headless should work with the AM4 CPU/MB combination according to LTT (10:30):
https://youtu.be/boKmZKTKXHc?t=628

It might just be a little less convenient but on the flipside you save further on potential power consumption ...

 

[jgreco]

Yes ... in that case if you go with a consumer board you have no GPU / iGPU with the necessary CPU for 4x4.

But running headless should work with the AM4 CPU/MB combination according to LTT (10:30):
https://youtu.be/boKmZKTKXHc?t=628

It might just be a little less convenient but on the flipside you save further on potential power consumption ...

I'm sorry, but I'm going to have to see what I can do to break your URL. The automatic unfurl is unpacking Freshbooks and Zoho advertising on the LTT YouTube spamvertising page; advertising is forbidden on the TrueNAS Forums. Even the link to LTT might be considered a bit dodgy, though I can see the relevance to the discussion.

@pixelwave , this is not to be construed as an admonishment or warning, just an explanation of the action taken and why.

 

[Davvo]

If I were you my approach to ltt, in regards to TrueNAS and ZFS in general, would be very cautious.

 

[pixelwave]

If I were you my approach to ltt, in regards to TrueNAS and ZFS in general, would be very cautious.

Agreed! I also followed their content regarding the petabyte server .. and the screw-ups. But from time to time there are useful hints and that channel led me down the TrueNAS path in the first place.

 

[Davvo]

That channel led me down the TrueNAS path in the first place.

Same, and I'm so glad I delved in this forum before doing anything. Pretty sure it saved me a lot of regrets.

 

[pixelwave]

As promised my 40GbE Mellanox journey starts here:

TrueNAS Scale and 25/40GbE (Mellanox ConnectX-3/4) Setup, Benchmark and Tuning ...

This is a follow-up from my NVMe TrueNAS Scale storage build. If you want to read about my journey so far start here: https://www.truenas.com/community/threads/amd-ryzen-with-ecc-and-6x-m-2-nvme-build.104574/ ——— I am trying to post my journey and learnings about 40GbE QSFP in this separate...

www.truenas.com

For all those who are interested.

 

[DigitalMinimalist]

Any tweaks to reduce power consumption?

I have X470D4U with 2700X 4x16GB ECC RAM, ASUS Hyper M.2 with 4x Micron 7300Pro 1TB , 4x SATA HDD and Intel X710-DA2…

I need to test a few different PSUs (currently 1000W Platinum) - any other options than upgrading CPU to 5700X?

 

[pixelwave]

I am currently looking into AMD PPT tweaking … will update shortly.

 

[ztx9259]

do you have plan to add any cheap GPU card to deal with videos?

 

[pixelwave]

do you have plan to add any cheap GPU card to deal with videos?

For that build NVMe storage was the focus ... actually there are not many PCIe lanes (if any) left for a GPU.

For the future I am looking into switching to Epyc platform (Zen 2 currently seems like a good price/performance/consumption choice). With that the PCIe lane issues are pretty much non existent with 128x lanes ... compared to the 24/28 lanes of the current build.

Really depends on what you want to do .. for just fast NAS storage (with ZFS/ECC/NVMe) and some non-GPU intense docker stuff I think the build is pretty awesome and has a good performance.

If you plan to do more, VM stuff, passthrough, GPU intense work (..) and with headroom for future explorations - Epyc/Xeon is the way to go.

 

[ztx9259]

For that build NVMe storage was the focus ... actually there are not many PCIe lanes (if any) left for a GPU.

For the future I am looking into switching to Epyc platform (Zen 2 currently seems like a good price/performance/consumption choice). With that the PCIe lane issues are pretty much non existent with 128x lanes ... compared to the 24/28 lanes of the current build.

Really depends on what you want to do .. for just fast NAS storage (with ZFS/ECC/NVMe) and some non-GPU intense docker stuff I think the build is pretty awesome and has a good performance.

If you plan to do more, VM stuff, passthrough, GPU intense work (..) and with headroom for future explorations - Epyc/Xeon is the way to go.

I am looking forward to Ryzen 7000.
Current products have 2cu GPU and more PCI-E 5.0, and if wanting to have 5950X performance on 7950X, it will only take 90W and lower
Really my expected platform and wait for new low-power-consumption products

 

[pixelwave]

I am looking forward to Ryzen 7000.
Current products have 2cu GPU and more PCI-E 5.0, and if wanting to have 5950X performance on 7950X, it will only take 90W and lower
Really my expected platform and wait for new low-power-consumption products

I also waited a long time for the new Atom C5000 low power platform and it still is not really that well available for NAS and Home Server applications.

But since tinkering and experimentation increases naturally as you dive deeper into the whole server topic .. I need lanes! :) Epyc Siena looks promising from a low-power-consumption / high-performance / plenty-lane point of view. But who knows when hardware is available there ...

Currently Zen2 Epyc looks the way to go for my next endeavour also because you can get it dirt cheap on eBay with a Supermicro H12SSL. Paired with 16/24 orr 32 core version and cheap RAM you are good to go for the next couple of projects ... ;) And the extra waste in higher energy consumption is more then made up for by the cheaper price.

 

[alexnetdoc]

I would like to ask what is the power consumption of this build? I am more interested in idle. Do you get like 20-30W idle?
The reason I am asking is that currently I am testing out quite some Hardware:
Fujitsu D3401-B11, Intel i3-6100. No ECC support 32GB memory, 13Watts idle with 2 x 4TB drives and 2x1TB SSD's
Huananzhi QD4 Aliexpress MB, Xeon E5-2650L 12C 24T 65W TDP, 64GB REG ECC support, 60Watts idle with 2 x 4TB drives and 2x1TB SSD's

Edit: I have an offsite backup Synology DS418J 2TB x4 drives to have a backup with --checksum from Truenas
Edit2: Now I have under the radar: B450M Pro4-F R2.0 with ECC support, AMD Ryzen 3 PRO 3200G, Micron 16GB ECC UDIMM MTA9ASF2G72AZ-3G2

I am torn between the power consumption and the safety of the data. Usually it's photos, videos and music that we store there with my wife...
What would you choose?

 

[Davvo]

What would you choose?

ECC.

 

[alexnetdoc]

ECC.

Which one? AMD or Huananzhi?

 

[Davvo]

Which one? AMD or Huananzhi?

Neither since both have Realtek LAN (the Huananzhi wasn't clear about it).
Please open your own thread.

 

[pixelwave]

I would like to ask what is the power consumption of this build? I am more interested in idle. Do you get like 20-30W idle?
The reason I am asking is that currently I am testing out quite some Hardware:
Fujitsu D3401-B11, Intel i3-6100. No ECC support 32GB memory, 13Watts idle with 2 x 4TB drives and 2x1TB SSD's
Huananzhi QD4 Aliexpress MB, Xeon E5-2650L 12C 24T 65W TDP, 64GB REG ECC support, 60Watts idle with 2 x 4TB drives and 2x1TB SSD's

Edit: I have an offsite backup Synology DS418J 2TB x4 drives to have a backup with --checksum from Truenas
Edit2: Now I have under the radar: B450M Pro4-F R2.0 with ECC support, AMD Ryzen 3 PRO 3200G, Micron 16GB ECC UDIMM MTA9ASF2G72AZ-3G2

I am torn between the power consumption and the safety of the data. Usually it's photos, videos and music that we store there with my wife...
What would you choose?

If you go with TrueNAS you might as well go all the way picking ECC RAM. It is not required but still my reason to switch from Synology a while ago was due to higher data safety and better snapshot / backup options given by the ZFS filesystem. Otherwise safe yourself the hassle and stay within the Synology ecosystem since you already seem to have one. I have now a setup were my old TrueNAS test system (small ITX cube see in signature) serves as my backup location - so I am all in on TrueNAS.

Regarding power consumption - while my current system fully stacked idles around 50-60W without optimizations I still want to experiment with AMDs PPT feature in BIOS. Basically I can set my own (TDP) power limit - by sacrificing performance (https://youtu.be/HXDo9KcSovE -> video in German but basically lowers TDP from 65W to 45W ... but you could go even lower).

 

[alexnetdoc]

Thank you! I am definitely thinking on going with ECC. Ofc, I had no problems with ECC and none-ECC for 4 years, my ALL devices do not use ECC at home (as my wife's Macbook Air). My workflow is mostly idle:
* Get pictures and videos from the trip
* make a movie of the trip, store it
* kids watch movies through emby via Apps on truenas
* I have a redundant pihole on truenas with keepalived
* my all DATA at the moment is 1,8TB in size starting from 2013 :)

so for the most part me or my wife are working with pictures on a monthly basis.

That's where I started to look for builds like a german woolfgang to make sure the server has lowest wattage. I very like your build, it's just for me the biggest question to go with the asrock rack is the power consumption. if it's the same as the aliexpress chinese motherboards with x99 chipsets and 4 channel DDR4 memories, then I don't see a point switching. Huananzhi qd4 + intel dual gig network card + 6 sata ports and 2 m.2 nvmie and DDR4 ECC reg 64GB memory will fit my needs

But when we talk about 20-30 watts idle with ECC, that's where I want to land.

By the way, if you are using Truenas Scale - try going to shell and run "sudo powertop --auto-tune". it should reduce your wattage use!

 

[luuk]

The memory panel isn't actually showing any "ECC" title .. I saw this with some other installs.

Is this an issue? Can you have ECC ram installed but the "ECC feature" not being active ...?

View attachment 59899

BIOS says:

View attachment 59900

View attachment 59902

dmidecode -t memory output:

I thought that I read somewhere that the non-X Ryzen cpu's do not offer any ECC capability, but apparently your TrueNAS does show up with ECC.
Does anyone here know how to find error correction events in the logs of Truenas Scale?