Memory in GUI does not reflect amount physically installed?

[Koop]

Maybe this is a really obvious question but I couldn't find an obvious answer.

I have 512GB of memory installed in my system but TrueNAS-SCALE-23.10.2 only reports 502.4GiB total available (ECC) memory.

I feel like this is a simple thing I'm missing. I assume that there must be some memory reserved for hardware and other things I may not be thinking of but 10GB seems like a lot to not show up? Is it a % of physical memory being reserved or unusable in some way? I know typically you'll see some of your memory not available in a system like my windows desktop- 32GB installed but only 31.9 as usable- is this perhaps the same thing but since I'm using such high capacity and number of DIMMs I'm seeing it on a higher scale?

Just looking to learn and understand why. Thanks!

Code:

Motherboard: Supermicro X11SPi-TF
Memory: 512GB DDR4 ECC memory (SK Hynix 8x64GB LRDIMM, HMAA8GL7MMR4N-UH)
Intel Xeon Platinum 8153
AOC-S3008L-L8e+ HBA connect to BPN-SAS2-846EL Backplane
Supermicro 846 chassis w/ 2x PWS-920P-SQs
2x Supermicro SSD-DM128-SMCMVN1 used for boot mirror
2x Intel OPTANE P1600X 58GB for SLOG mirror (one in m.2 motherboard slot the other in the  PCIe 3.0 x4 (in x8) slot)

 

[Bainnor]

I have 512GB of memory installed in my system but TrueNAS-SCALE-23.10.2 only reports 502.4GiB total available (ECC) memory.

Just looking to learn and understand why. Thanks!

View attachment 76365

It's due to some confusing unit notation. Giga denotes 10^9 in metric notation, thus a gigabyte (GB) is 1 billion bytes. Computers, however, use powers of 2, so for computer usage a gigabyte should be 1024^3, or 1, 073, 741, 824 bytes. In 2009, a new unit was standardized, the gibibyte (GiB), which is 2^30 (aka 1024^3). Effectively, your ram is all there, it' s just a more accurate unit is being used to display it, rather than the unit that marketing has settled on.

 

[winnielinnie]

Onboard video can take some RAM, depending on your hardware.

 

[Koop]

It's due to some confusing unit notation. Giga denotes 10^9 in metric notation, thus a gigabyte (GB) is 1 billion bytes. Computers, however, use powers of 2, so for computer usage a gigabyte should be 1024^3, or 1, 073, 741, 824 bytes. In 2009, a new unit was standardized, the gibibyte (GiB), which is 2^30 (aka 1024^3). Effectively, your ram is all there, it' s just a more accurate unit is being used to display it, rather than the unit that marketing has settled on.

Ok I did figure it was something to do with how TrueNAS specifically states it as GiB but I didn't fully understand the math. I just knew that the GB you see on the the tin isn't what you end up actually getting. Or at least that's what I assumed even for memory. Didn't expect it to be a 10GB loss though, that's like- a whole bunch! Booo.

I would assume it's this as both a combination of this and potentially hardware reserving some before TrueNAS even has a chance to see it.

Onboard video can take some RAM, depending on your hardware.

VGA, BMC, etc?

 

[winnielinnie]

What is your hardware?

 

[Koop]

What is your hardware?

Code:

Motherboard: Supermicro X11SPi-TF
Memory: 512GB DDR4 ECC memory (SK Hynix 8x64GB LRDIMM, HMAA8GL7MMR4N-UH)
Intel Xeon Platinum 8153
AOC-S3008L-L8e+ HBA connect to BPN-SAS2-846EL Backplane
Supermicro 846 chassis w/ 2x PWS-920P-SQs
2x Supermicro SSD-DM128-SMCMVN1 used for boot mirror
2x Intel OPTANE P1600X 58GB for SLOG mirror (one in m.2 motherboard slot the other in the  PCIe 3.0 x4 (in x8) slot)

 

[winnielinnie]

I have no idea where that extra 10 GiB of RAM is being lost...

As far as I know, the GUI correctly reports in GiB units of measurement for the RAM.

In my case, I have an onboard Intel GPU that gets 128 MiB, leaving me with just shy under 32 GiB (~31.8) of available RAM, as reported in the dashboard. (The same value is reported by "htop".)

But to go from 512 GiB to 502? That's a steep loss.

 

[Koop]

I have no idea where that extra 10 GiB of RAM is being lost...

As far as I know, the GUI correctly reports in GiB units of measurement for the RAM.

In my case, I have an onboard Intel GPU that gets 128 MiB, leaving me with just shy under 32 GiB (~31.8) of available RAM, as reported in the dashboard. (The same value is reported by "htop".)

But to go from 512 GiB to 502? That's a steep loss.

Is it the difference of GB vs GiB as @Bainnor explained? I guess in reality I don't know what the true GiB per 64GB stick is?

 

[danb35]

As far as I know, the GUI correctly reports in GiB units of measurement for the RAM.

I don't think so; here's what my SCALE system reports for my 128 GB installed:

That's a difference of 2.5 GiB. Multiply by 4 to get 512 GB, and it's a 10 GB delta. Looks about right.

 

[winnielinnie]

What does htop reveal on your SCALE system?


EDIT:

* Because I can't get 128 from either conversion, from either direction.

125.5 GiB = 134.75 GB

125.5 GB = 116.88 GiB

Neither comes close to 128. (And 128 would be the "marketed" value of the RAM sticks.)

 

[danb35]

 

[chuck32]

But to go from 512 GiB to 502? That's a steep loss.

That seems pretty consistent, my 128 Gigabyte system reports as 125,7 GiB. My 16 GB report as 15,6 GiB.

It's kind of curious, the conversion for memory modules seem to differ.

For my 4 TB and 8 TB HDDs it's pretty consistent.

4*10^12 Bytes are advertised as 4 TB which are 4*10^12 * 2^{-40} = 3,6379 TiB which is roughly the reported 3,64 TiB in Truenas.

With the same conversion 512*10^9 Bytes should yield 512*10^9 * 2^{-30} = 476,83 GiB as @winnielinnie correctly assumed.

I think the magic lies in the difference between HDDs and memory modules. Edit: the following is probably wrong, see for a better idea of an solution.

For a HDD they will say 1 TB is 10^12 Bytes. A memory module consists of multiple chips on a stick. This is where my knowledge about hardware leaves me, someone feel free to step in ;) If the misadvertisement doesn't happen on the TB scale but on a much smaller scale, let's say KB and KiB the difference in the end gets much smaller (due to the exponential nature of the systems). Hence I assume for memory modules the conversion happens around 1 KB <-> 1 KiB.

Converting 64 GB = 10^9 Bytes to GiB is different from converting 64 000 000 times 1 KB = 1000 Bytes to KiB, you only "lose" 24 bytes originally and then scale it. This would result in 64 000 000 * 24 Bytes = 1 536 000 000 Bytes = 1.536 GiB

OP is using 8 x 64 GB modules, so the difference is 8 * 1.536 GiB = 12,288 GiB which is much closer to what we are seeing.

It doesn't seem to be as straight forward with RAM modules as it is with HDDs. I assume @danb35 is also running 4 sticks and his reporting deviates a little from mine.

Decimal	Binary	Difference	Decimal / Binary
Bytes	Bytes	Bytes	%
1000	1024	24	97,65625
1000000	1048576	48576	95,36743164
1000000000	1073741824	73741824	93,13225746
1E+12	1,0995E+12	9,9512E+10	90,94947018
1E+15	1,1259E+15	1,259E+14	88,81784197
1E+18	1,1529E+18	1,5292E+17	86,7361738
1E+21	1,1806E+21	1,8059E+20	84,70329473

 

[bonox]

RAM has been made as GiB forever I believe. It's only the HDD makers that brought in the screwed up power 10 instead of power 2 measures.

 

[bonox]

as sizes get bigger, it's becoming more of a pita that even the memory manufacturers refer to their parts as GB rather than GiB. I've got a dozen DDR3 boxen here that report exactly what you'd expect - 384 GB installed for example and Truenas shows 383.8GiB on the dashboard. Perhaps someone changed the rules since then.

edit: I should say my boxes are all running CORE not SCALE. See bottom of post #16.

The 'missing' memory is usually hardware reserved space by BIOS/UEFI so the OS never sees it and reports only what's available to it not what's physically installed - same as a VM would. I know nothing about supermicro boards, but is it possible there's:

1. memory protection/interleaving/some other bios option that relates to multiple memory channels that is set to use some space for itself for reliability/uptime at the expense of total space?
2. Do you have a large chunk of memory set aside for an onboard video card? Not like IPMI to use that but some other kind you may have installed? Perhaps some other peripheral?
3. need new glasses? (jk)
4. the RAM manufacturer short changed the modules? Or perhaps just one of 16 modules is short and they're all pulled back to the same value?
5. what does CORE say instead of SCALE? Just in case there's operating system relevant shenanigans going on.

 

[chuck32]

It's only the HDD makers that brought in the screwed up power 10 instead of power 2 measures.

I disagree, the decimal system stems from the SI prefixes kilo, mega, ... that were well established before kibi, mini, ... They didn't bring it up they just stuck to it.

as sizes get bigger, it's becoming more of a pita

Yes, in the early days when "640K ought to be enough for anybody" it simply didn't matter that there was a difference in the binary system of computers.

 

[bonox]

I disagree, the decimal system stems from the SI prefixes kilo, mega, ... that were well established before kibi, mini, ... They didn't bring it up they just stuck to it.

I disagree. The SI units are fine - my contention is that they jumped ship. A case of playing both sides of the measurement coin and claiming whatever advantage you can.

My memory of old tech like IBM 29 "MB" disk packs is that they were made to store 29MiB and aligned very nicely with memory sizes, all being binary rather than decimal base. At some point in the chain, some marketing wowser decided that they could advertise more capacity without actually having to manufacture said usable capacity by switching to decimal orders of magnitude and inserting a too small to read disclaimer on the box.

But in the context of this thread, all that's irrelevant. The memory makers never did this (at least to my knowledge) and they remain in base 2 land; the way RAM is addressed and directly linked to the CPU means base 10 doesn't work without (slow) translation. Also the reason intel's Optane 'ram' storage models are base 2 size not base 10. However memory makers also fail in nomenclature by generically referring to a true 64GiB as 64GB on packaging. That's the issue here as I see it. You can argue over SI semantics all you like, but I can't find a reference anywhere that makers of RAM have moved to 10^x recently instead of the traditional 2^x to account for the numbers presented by the OP.

We've known for a while that SCALE has funny things going on with ARC sizes - would be interesting for the OP to spin up a temporary install of CORE to see what it says.

 

[winnielinnie]

The numbers don't "math" correctly either. If this was simply a case of binary vs decimal, then we should expect a consistent, clean conversation to result in 32, 64, 128, 512, etc. But we don't have that. We instead have "sort of / slightly" away from the desired capacity.

Something else is likely to blame.

512 GB translates to 477 GiB

512 GiB translates to 550 GB

In neither case does 502 exist (which is the capacity in the OP's post.)

If this was purely a matter of "SCALE is reporting the wrong units", then their dashboard should show "477 GiB of RAM".

And like @bonox mentioned, the HDD manufacturers might have played this marketing trick on the consumers, but RAM manufacturers never really did so.

 

[chuck32]

A case of playing both sides of the measurement coin and claiming whatever advantage you can.

Agreed :)

We've known for a while that SCALE has funny things going on with ARC sizes - would be interesting for the OP to spin up a temporary install of CORE to see what it says.

This may be the case, I found it interesting that the difference aligned quite nicely though.

Windows reports 63.9 GB for my 64 GB, so there's that. But windows also shows GB and TB but actually means GiB and TiB, sooo....

But in the context of this thread, all that's irrelevant. The memory makers never did this (at least to my knowledge) and they remain in base 2 land;

It seems the solution is: Truenas seems to report whatever is displayed in /proc/meminfo under MemTotal

Code:

root@truenas[~]# grep MemTotal /proc/meminfo
MemTotal:       131790332 kB

125,68 GiB (as displayed in the dashboard)

Code:

root@truenas[~]# dmesg | grep Memory:
[    0.118088] Memory: 1882208K/134105584K available (14342K kernel code, 2326K rwdata, 8912K rodata, 2796K init, 17420K bss, 2489520K reserved, 262144K cma-reserved)

127,89 GiB (close enough ;) )

 

[Koop]

I'm glad I don't feel completely crazy for asking now. I had tried converting GB to GiB at the full amount and said ah but wait I should do it per DIMM- but the numbers didn't make much sense to me still.

Perhaps it has to do with the configuration in terms of numbers of DIMMs used @ X Capacity and even number of chips/memory modules per DIMM that make a difference?

See the documentation here: https://download.datasheets.com/pdf..._ds_8gb_ddr4m-verbased_lrdimmsrev.1.3.pdf.pdf

I would assume the Organization and the Component Composition outlined here also impact the overall math?

Thanks for breaking down your math so thoroughly @chuck32

 

[joeschmuck]

RAM has been made as GiB forever I believe.

Sort of true. By today's definition yes but back in the day this definition didn't exist. It wasn't around in the 1970's, 1980's, and most of the 1990's. GiB became defined in 1998 as a way to define the two measurements (metric and binary). Yes, I know this was likely the very first "Shrinkflation" in the corporate world. If the Cookie Monster knew back then what he knows now. The greedy corporate hard drive makers introducing the metric version to sell more product. So when us old timers say GB, most of the time we are really referencing GiB. I very rarely use GiB unless I need to be extremely technical about something.

I am curious where the 10GB of RAM went as well so I will watch and see where this goes.