Supermicro H11DSi only sees 768GB RAM

[Psynapsx]

Hi guys,

I have a Supermicro H11DSi board (rev 1.0) with 2x Epyc 7551. I installed 16x Hynix ECC 128GB DDR4 LRDIMM 2666 modules. According to manual the rev 1.0 should handle 2TB memory and the rev 2.0 4TB.
But in IPMI "hardware information" menu it only sees 6 DIMMs = 768GB. When I go to sensor readings it correctly displays temperatures for all 16 modules.
Seriously, I'm losing my mind, the modules are exactly according to specs and yet only 6 out of 16 detected. It cost a lot of money and now maybe it was all thrown out of the window....

 

[jgreco]

What did Supermicro technical support say?

 

[Psynapsx]

haven't contacted them yet. Tbh a little bit tired of their copy-paste bs replies.
They will tell me to use a tested module.
When you go to see the "tested" memory page for the H11DSi, you get this:

Memory List | Supermicro

www.supermicro.com

yes, exactly nothing

 

[Psynapsx]

 

[jgreco]

Well the fact that the DIMM's are seen over the system management bus isn't really that big a thing, it just means they're plugged in and powered on, and able to talk to the BMC subsystem. That's entirely different from being detected, configured, and usable by the main system.

What happens when eight modules are installed?

Not to put too fine a point on it, but this kind of stuff is just part of server builds. The reason companies like mine and like iXsystems charge a significant margin on these things is because we often run into strange problems that make it necessary to try different memory, try a different board, spend hours of employee time on debugging and burn-in, etc., and we burn through cash trying to identify problems sometimes.

Part of that is definitely interfacing with Supermicro tech support, and being uninterested in communicating with them is really only hurting you. Memory modules are not always compatible even if their stats suggest that they should be. The reason Supermicro lists compatible parts is to help reduce this sort of issue.

 

[blanchet]

You may try to upgrade the lastest BIOS version, it may solve the issue.

 

[Psynapsx]

I fully understand.
In my case they do not list any compatible parts though. There is not a single tested 128GB dimm listed on their site so all I could do is buy the exact specs. They have to stop shipping junk.

 

[Psynapsx]

You may try to upgrade the lastest BIOS version, it may solve the issue.

latest bios and ipmi firmware is installed...

 

[Etorix]

So IPMI reports temperatures for 16 DIMM modules but you're worried because the tree display shows only 6 modules. How much RAM is seen by TrueNAS? This, after all, is a TrueNAS forum, not a Supermicro forum.

 

[Psynapsx]

So IPMI reports temperatures for 16 DIMM modules but you're worried because the tree display shows only 6 modules. How much RAM is seen by TrueNAS? This, after all, is a TrueNAS forum, not a Supermicro forum.

768GB reported by TrueNAS

 

[Newfoundland.Republic]

Based on Supermicro's website only the Epyc 7002 series is verified with 128 GB DDR4 LRDIMM-2666 DIMMs. That is on the 2.x revision of the board.

--- Edit ---
Take a look at Memory Population Guidelines for AMD EPYC™ Processors - it might provide some additional insight.

 

[Etorix]

Then it is an issue but I'm afraid the solution involves insisting with Supermicro technical support until they have a useful answer.
Newfoundland.Republic has a point, but the specification on the main do list rev.1 as supporting up to 2 TB. If the board is officially listed as supporting 2 TB RAM, it should have been tested with 128 GB modules. Ask them which modules and escalate until someone at Supermicro understands that, if the revision 1 of the board is advertised as supporting 2 TB but has not been actually tested in this scenario, they have been caught with their pants off…

 

[Psynapsx]

Based on Supermicro's website only the Epyc 7002 series is verified with 128 GB DDR4 LRDIMM-2666 DIMMs. That is on the 2.x revision of the board.

--- Edit ---
Take a look at Memory Population Guidelines for AMD EPYC™ Processors - it might provide some additional insight.

thanks, that’s right, but according to user manual rev1 supports 128gb too, just not 256gb dimms.

 

[Psynapsx]

Then it is an issue but I'm afraid the solution involves insisting with Supermicro technical support until they have a useful answer.
Newfoundland.Republic has a point, but the specification on the main do list rev.1 as supporting up to 2 TB. If the board is officially listed as supporting 2 TB RAM, it should have been tested with 128 GB modules. Ask them which modules and escalate until someone at Supermicro understands that, if the revision 1 of the board is advertised as supporting 2 TB but has not been actually tested in this scenario, they have been caught with their pants off…

thanks, yes I contacted them earlier today, we will see what their respons is...
I thoroughly tested all modules in an Asrock Rack TrueNAS box that even has the same CPU and there were no issues with the modules

 

[Psynapsx]

I think I found the issue.
This garbage does not support 2S4R modules probably, only 8R. I have the Hynix 2S4Rx4 128GB 2666 LRDIMM modules.
Porbably because 1st gen AMD EPYC does not support 2S4R 128GB.
On memory.net none of the 2S4R modules are listed as supported for this board.

Now I have 18k USD worth of memory I cannot use.

 

[Psynapsx]

What did Supermicro technical support say?

as expected their reply just arrived:) clown world

"Hello David,

We did not validate any 128GB modules

When you only popuplate P1-DIMMBA1 or B1 will the memory be detected ?
Did you swap the memory on non detected slots with detected slots to rule out memory failures ?"

 

[Newfoundland.Republic]

@Psynapsx - I'll put on my day-job manager's hat here: My first qustion to the person reporting to me who was in this situation would be: Did you validate that (in this case) the RAM you ordered was compatible with the main board? This would not mean "I think" - it would mean, if necessary, contacting the vendor and confirming - especially if there was any lack of clarity in the vendor's specs. The bigger the purchase, the more important it is to validate before purchasing.

"Think" means "don't know"; "don't know" means "ask".

 

[Psynapsx]

@Psynapsx - I'll put on my day-job manager's hat here: My first qustion to the person reporting to me who was in this situation would be: Did you validate that (in this case) the RAM you ordered was compatible with the main board? This would not mean "I think" - it would mean, if necessary, contacting the vendor and confirming - especially if there was any lack of clarity in the vendor's specs. The bigger the purchase, the more important it is to validate before purchasing.

"Think" means "don't know"; "don't know" means "ask".

I'm not reporting to anyone, except myself:)
My first question would be: why does Supermicro list FAKE specifications on their site and user manual? Why list 128GB DIMMs as supported when they did not validate any?

 

[Newfoundland.Republic]

Well, one thing on your side is you don't have to report to the "spousal unit" (a/k/a my CFO )

 

[Psynapsx]

I can confirm Epyc 7001 series can handle 2S4R 128GB modules.I tested the DIMMs in an Asrock Rack ROMED8-2T with the same CPU and it handles 8x Hynix 2S4Rx4 128GB 2666 LRDIMM modules without issue (1TB).So it's the Supermicro motherboard, maybe due to it being rev1.0.
I will now test this in a Supermicro H11SSL rev2.0.