AMD Ryzen with ECC and 6x M.2 NVMe build

[pixelwave]

I wouldn't personally run docker/kubernetes off spinning disks - SSD all the way imo

My docker/kubernetes is running from 3x M.2 NVMe array (R-Z1) ..

 

[pixelwave]

The memory panel isn't actually showing any "ECC" title .. I saw this with some other installs.

Is this an issue? Can you have ECC ram installed but the "ECC feature" not being active ...?

BIOS says:

dmidecode -t memory output:

root@truenas-pwave[~]# dmidecode -t memory
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.

Handle 0x0018, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: None
Maximum Capacity: 128 GB
Error Information Handle: 0x0017
Number Of Devices: 4

Handle 0x0020, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x001F
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: Unknown
Set: None
Locator: DIMM 0
Bank Locator: P0 CHANNEL A
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Unknown
Serial Number: Unknown
Asset Tag: Not Specified
Part Number: Unknown
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: Unknown
Memory Operating Mode Capability: Unknown
Firmware Version: Unknown
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None

Handle 0x0022, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x0021
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMM 1
Bank Locator: P0 CHANNEL A
Type: DDR4
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 3200 MT/s
Manufacturer: Kingston
Serial Number: 85763954
Asset Tag: Not Specified
Part Number: 9965745-020.A00G
Rank: 2
Configured Memory Speed: 3200 MT/s
Minimum Voltage: 1.2 V
Maximum Voltage: 1.2 V
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Unknown
Module Manufacturer ID: Bank 2, Hex 0x98
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

Handle 0x0025, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x0024
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: Unknown
Set: None
Locator: DIMM 0
Bank Locator: P0 CHANNEL B
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Unknown
Serial Number: Unknown
Asset Tag: Not Specified
Part Number: Unknown
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: Unknown
Memory Operating Mode Capability: Unknown
Firmware Version: Unknown
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None

Handle 0x0027, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x0026
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMM 1
Bank Locator: P0 CHANNEL B
Type: DDR4
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 3200 MT/s
Manufacturer: Kingston
Serial Number: E9763CC4
Asset Tag: Not Specified
Part Number: 9965745-020.A00G
Rank: 2
Configured Memory Speed: 3200 MT/s
Minimum Voltage: 1.2 V
Maximum Voltage: 1.2 V
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Unknown
Module Manufacturer ID: Bank 2, Hex 0x98
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

 

[pixelwave]

Interesting .. I changed the value from "AUTO" to "ENABLED":

And now it is showing in TrueNAS!

and the output also looks better now:

root@truenas-pwave[~]# dmidecode -t memory
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.

Handle 0x0018, DMI type 16, 23 bytes
Physical Memory Array
Location: System Board Or Motherboard
Use: System Memory
Error Correction Type: Multi-bit ECC
Maximum Capacity: 128 GB
Error Information Handle: 0x0017
Number Of Devices: 4

Handle 0x0020, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x001F
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: Unknown
Set: None
Locator: DIMM 0
Bank Locator: P0 CHANNEL A
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Unknown
Serial Number: Unknown
Asset Tag: Not Specified
Part Number: Unknown
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: Unknown
Memory Operating Mode Capability: Unknown
Firmware Version: Unknown
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None

Handle 0x0022, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x0021
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMM 1
Bank Locator: P0 CHANNEL A
Type: DDR4
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 3200 MT/s
Manufacturer: Kingston
Serial Number: 85763954
Asset Tag: Not Specified
Part Number: 9965745-020.A00G
Rank: 2
Configured Memory Speed: 3200 MT/s
Minimum Voltage: 1.2 V
Maximum Voltage: 1.2 V
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Unknown
Module Manufacturer ID: Bank 2, Hex 0x98
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

Handle 0x0025, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x0024
Total Width: Unknown
Data Width: Unknown
Size: No Module Installed
Form Factor: Unknown
Set: None
Locator: DIMM 0
Bank Locator: P0 CHANNEL B
Type: Unknown
Type Detail: Unknown
Speed: Unknown
Manufacturer: Unknown
Serial Number: Unknown
Asset Tag: Not Specified
Part Number: Unknown
Rank: Unknown
Configured Memory Speed: Unknown
Minimum Voltage: Unknown
Maximum Voltage: Unknown
Configured Voltage: Unknown
Memory Technology: Unknown
Memory Operating Mode Capability: Unknown
Firmware Version: Unknown
Module Manufacturer ID: Unknown
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: None
Cache Size: None
Logical Size: None

Handle 0x0027, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0018
Error Information Handle: 0x0026
Total Width: 72 bits
Data Width: 64 bits
Size: 32 GB
Form Factor: DIMM
Set: None
Locator: DIMM 1
Bank Locator: P0 CHANNEL B
Type: DDR4
Type Detail: Synchronous Unbuffered (Unregistered)
Speed: 3200 MT/s
Manufacturer: Kingston
Serial Number: E9763CC4
Asset Tag: Not Specified
Part Number: 9965745-020.A00G
Rank: 2
Configured Memory Speed: 3200 MT/s
Minimum Voltage: 1.2 V
Maximum Voltage: 1.2 V
Configured Voltage: 1.2 V
Memory Technology: DRAM
Memory Operating Mode Capability: Volatile memory
Firmware Version: Unknown
Module Manufacturer ID: Bank 2, Hex 0x98
Module Product ID: Unknown
Memory Subsystem Controller Manufacturer ID: Unknown
Memory Subsystem Controller Product ID: Unknown
Non-Volatile Size: None
Volatile Size: 32 GB
Cache Size: None
Logical Size: None

 

[Glowtape]

Weird. I have the same UDIMMs apparently, according to the part number in dmidecode. It says ECC just fine here in the TrueNAS memory widget. X570D4U with Ryzen 5750G.

--edit: Too slow.

 

[Etorix]

I suppose these are the quirks of using ECC on a platform where the feature "may" work but is not officially validated.

 

[pixelwave]

Memory usage will come up with time.

Yepp ... you are rightly. Filling up quickly. :)

 

[mistermanko]

I run idle with all components above installed and active (+IPMI, +1GBe, +40GBe) below 60W measured from the plug without any optimizations yet (BIOS / P-states / Undervolting / ..):

Hm, kinda underwhelming, my setup (LSI HBA, 6x spinning rust, 4x sata ssd, xeon e3, workload: multiple user nextcloud and plex) idles at 61W and averaged over last 12 month at about 80W.

 

[Davvo]

Yepp ... you are rightly. Filling up quickly. :)

View attachment 59948

Also iirc in SCALE by default you can't use more than 50% of total available for ZFS cache. There is a workaround I believe, but don't happen to remember it at the moment.
My knowledge about SCALE it's all second hand though since I don't use it so I might be wrong.

 

[pixelwave]

Hm, kinda underwhelming, my setup (LSI HBA, 6x spinning rust, 4x sata ssd, xeon e3, workload: multiple user nextcloud and plex) idles at 61W and averaged over last 12 month at about 80W.

I think there should be some room for improvements ...

But I also fully entertain 24x PCIe lanes, 2C/4T more and your Red drives consume also less power then the faster spinning WD helium series.

 

[Etorix]

Also iirc in SCALE by default you can't use more than 50% of total available for ZFS cache. There is a workaround I believe, but don't happen to remember it at the moment.

Correct, and one should NOT try to override this because the limitation comes from the way memory is allocated in Linux.
One could, of course, contribute to rewriting the kernel allocator. Good luck with that…

 

[NickF]

Correct, and one should NOT try to override this because the limitation comes from the way memory is allocated in Linux.
One could, of course, contribute to rewriting the kernel allocator. Good luck with that…

I manually increased mine with no ill effect, though, I have 256GB of memory.

 

[pixelwave]

Funny Level1Tech also posted a quite similar Ryzen config recently:

Low Power/High Performance Server GRUDGE MATCH! DeskMeet, Ryzen, and Supermicro Micro-Tower Server

Thank you to Fractal for sponsoring this video! Check out the Meshify 2 RGB here: https://www.fractal-design.com/products/cases/meshify/meshify-2-rgb/black-t...

youtu.be

 

[mistermanko]

Can you give some numbers concerning total cost of your setup. I'd like to know if I ever could ROI on such a system.

 

[pixelwave]

Can you give some numbers concerning total cost of your setup. I'd like to know if I ever could ROI on such a system.

This is probably more on the expansive side because many components are bought brand-new.

Some caviats:

• AMD Ryzen 5 5600*boxed
  • new - cheaper options also with previous gen on ebay
• ASRock X470D4U
  • new - very depending on availbility, purchased additionally with BIOS update service because of 5000 series CPU
• Mellanox Connect-X 3 MCX354A-FCBT
  • used - on ebay
• SilverStone Temjin Evolution TJ08-E
  • used - on ebay, but any other case fitting m-atx works as well
• all NVMe and HDDs are purchased brand-new

So system cost around 1K and drives more or less double that.

 

[diedrichg]

It works!

CPU: AMD Ryzen 5 5600 6C/12T (boxed cooler)
Mainboard: ASRock X470D4U (with onboard GPU and IPMI)

Thank you very much for posting this, I'm looking at the X470D4U for my backup server and the Ryzen 5 5600 isn't specifically mentioned in the QVL list but the G and X parts are. Glad to see that all is okay! That's a pretty interesting setup you've got with the ASUS Hyper M.2. *filing that away for future expansion idea*

 

[pixelwave]

Thank you very much for posting this, I'm looking at the X470D4U for my backup server and the Ryzen 5 5600 isn't specifically mentioned in the QVL list but the G and X parts are. Glad to see that all is okay! That's a pretty interesting setup you've got with the ASUS Hyper M.2. *filing that away for future expansion idea*

Just keep in mind for the AMD 5000 series CPUs you need the BIOS version 4.20 and upwards to work.

I hope it gets easier in the future to find suitable platforms (below Epyc / Xeon class) supporting proper 4x4 bifurcation. Looking specifically at AM5 potential. But maybe "Epyc Siena" will be the solution in the future ...

The NVMe opportunities are quite cool in a storage server that also can handle the throughput via a beefy network connection (minimum 10GbE). Do not try this with 1 GbE connection - of course complete waste of money / resources.

I am still doing some optimisations and speed tests and will follow up regarding 40GbE speeds and potential improvements concerning power consumption.

 

[AnK]

I want to go for a similar system. Why you choose a 470 motherboard and not a 500-series motherboard so you can use PCIe 4.0 prior to 3.0?

 

[pixelwave]

I want to go for a similar system. Why you choose a 470 motherboard and not a 500-series motherboard so you can use PCIe 4.0 prior to 3.0?

I actually started with the X570 platform.

First issue I ran into - 4x4 bifurcation was not showing up in BIOS. As we later found out this is actually a CPU limitation. Basically if you use a AMD CPU with APU (iGPU) you will not be able to use full 4x4 bifurcation on either x470/570 platform.

At the end .. and after tons of research .. I choose the x470 platform from ASRockRack because it has IPMI and consumes less power then the x570. Also I wanted to add an additional PCIe card (the 40 GbE Mellanox) which would have not worked in the (much more expansive and barely available!) ASRock Rack X570 while fully 4x4 bifurcating. In that case it only leaves an additional x1 slot. Also if you do not upgrade the gigabit onboard ethernet to anything at least 10GbE ... the whole NVMe speed endeavour becomes pretty pointless for most cases.

Same goes for PCIe 4.0 vs 3.0 ... the limiting factor is ethernet for me. Precisely 40 GbE from the Mellanox card. With those .. even in the wildest best case scenario I will never be able to move beyond PCIe 3.0 speeds because the slot it sits in is electrically limited to PCIe 3.0 x4 (=3.9GB/s MAX). So the theoretical PCIe 4.0 advantages of super fast NVMe is pretty much gone.

Now that being said - if you want to save huge and IPMI is not important I recommend going with my original pick the ASRock X570M Pro4. But I would double check if the two PCIe x16 slots there are truly independent.

I will be looking into piKVM (https://youtu.be/fnd6wojrw3c) anyways in the future which would wipe the necessity of IMPI in most cases and allow to have ONE central (cheap!) management device for all computers which I plan to move into a centralised rack anyways.

 

[Etorix]

Now that being said - if you want to save huge and IPMI is not important I recommend going with my original pick the ASRock X570M Pro4. But I would double check if the two PCIe x16 slots there are truly independent.

They cannot be. Anything less than Threadripper/EPYC has not enough PCIe lanes to feed two x16 slots.
So its either:
* a choice of x16/x0 or x8/x8 from CPU lanes intended for GPU; or
* x16 from the CPU and x4 electrical from the PCH, in a x16 mechanical slot.

 

[pixelwave]

They cannot be. Anything less than Threadripper/EPYC has not enough PCIe lanes to feed two x16 slots.
So its either:
* a choice of x16/x0 or x8/x8 from CPU lanes intended for GPU; or
* x16 from the CPU and x4 electrical from the PCH, in a x16 mechanical slot.

Yes, that is clear ... 24x CPU PCIe lanes max for this AM4 gen. I guess my wording was misleading ... I was referring to the case of the ASRock Rack X570 where both x16 slots were automatically diminished if one was occupied.

But in case of the ASRock X570M Pro4 it should actually work - at least the manual states 16x + 4x config is valid (PCIE1/PCIE3: single at x16 (PCIE1); dual at x16 (PCIE1) / x4 (PCIE3))). So you basically would get the some setup like with my ASRock X470D4U .. if you want to ditch IPMI and go half price.

In general pretty tricky if you plan to use every last PCIe channel the system has to offer ... inlcuding the chipset ones. Looking forward to upgrading to an Epyc setup in the future with lanes to spare ... ;)