What metrics can I use to determine if I need more ram?

[imsocold]

I am running Freenas on a C2550D4i and while this cpu is pretty weak it seems to me that performance of the web interface should be much better than it is.
I am running an 8*8TB Raid Z2 setup with 8GB of ram and a plex docker instance.

I am getting tons of conflicting information on whether or not this is enough ram, and obviously this is a highly subjective.
So what I am asking is whether other not there is a simple way of determining that you need additional ram.
Back in the day an easy way to tell if your windows box was low on ram was to check if there was excessive paging.
I am not sure if this is the same on Truenas, and even it is true how I would go about checking.

Thanks.

 

[GlueFactoryBJJ]

FWIW, if I remember correctly, the recommendation used to be that you "needed" 1GB of RAM for every 1TB of drive space. Though maybe that was only if you had deduplication turned on.

With that much storage, if definitely recommend at least 16GB, if not 32 GB of RAM.

 

[imsocold]

as I understand that the 1GB per TB was for systems running Deduplication.

 

[GlueFactoryBJJ]

I said that in my previous post...

Since you are running Plex, I'd suggest at least 16GB or 32GB, as I mentioned above. It really depends on how much your NAS is actually doing. If you only watch videos occasionally and the NAS only sees weekly backups, then you can probably get away with less.

However, more storage requires ZFS to hold more info in memory. So I'd say 16GB is a minimum for your system. I have 32GB in my 48TB system.

 

[imsocold]

I mean no offense but I am looking for a metric to determine sizing, rather than being arbitrary.
I could easily throw a large amount of ram at the system, but my goal is not to be wasteful.
I have separate box dedicated to VMs and at one point I was running plex in a Linux vm with 512MB of ram and it ran superbly.
Plex's memory footprint simply does not justify a large amount of ram.

What I am actually concerned about is whether or not the ARC cache is large enough, and I would assume there is a metric I can use to determine if I need more ram.

I have spent a significant amount of money on this build already. $2000 and counting, and I don't want to add more ram than is truly actually necessary.

 

[awil95]

Plex really doesn't use up hardly any RAM. I have 32GB and watching a 4K movie plex says its using 1.5% of system memory. Now it may use more if you are CPU transcoding or when adding lots of media the server while it is pulling metadata. I have a Nvidia GPU for HW transcoding. As far as ZFS is concerned, no matter how much RAM you have it will always fill 50% to use as ARC. Since your pool is ~48TB I would recommend 32GB but it's not entirely necessary, 16GB would probably be the minimum.

 

[jgreco]

Plex's memory footprint simply does not justify a large amount of ram.

Plex's memory footprint is tied to numerous factors, primarily involving database sizes and the number of things the server is doing.

at one point I was running plex in a Linux vm with 512MB of ram and it ran superbly.

And at one point I was running Plex in a VM with only 4GB of RAM, and needed to boost it to 6GB because it was running like $#!+. ESXi still reports periods where it is using 95% of that 6GB, and some modest swapping at the OS level.

I mean no offense but I am looking for a metric to determine sizing, rather than being arbitrary.
I could easily throw a large amount of ram at the system, but my goal is not to be wasteful.

So the correct answer here is to NOT do slot-stuffing, which is where you use a larger number of lower density memory parts. For example, if your system can handle 32GB of RAM in 4 DIMM's, do not use 4GB DIMM's, instead use 8GB DIMM's, and you can start off with just two and see how it goes. Putting 4GB DIMM's in there means you run into a situation where you eventually need to pull the 4GB's, wasting the money you spent to buy those.

What I am actually concerned about is whether or not the ARC cache is large enough, and I would assume there is a metric I can use to determine if I need more ram.

You can look at the various ARC statistics to see what the hit rates are. However, you as an administrator need to have some reasonable insight into what sort of ARC rates might be appropriate for your workload. If you are seeing low ARC hit rates but are never accessing the same thing twice, that's perfectly acceptable. However, if you should be benefitting from caching and you don't seem to be getting good hit rates, then adding ARC is called for.

In general, not taking into account any jails, it's a good idea to shoot for 8GB of RAM plus 1GB additional for every TB of used space as a very minimum amount of RAM, at least until you get out past 24 or 32GB of RAM. This typically gets you into a reasonable ballpark for non-crappy ARC performance. Once you get out there to the larger numbers, it becomes more subjective. However, this is really almost always dependent on your specific workload and performance requirements/desires. For example, you can run 32TB of HDD on an 8TB RAM system but performance will be disappointing.

 

[GlueFactoryBJJ]

I mean no offense but I am looking for a metric to determine sizing, rather than being arbitrary.
I could easily throw a large amount of ram at the system, but my goal is not to be wasteful.
I have separate box dedicated to VMs and at one point I was running plex in a Linux vm with 512MB of ram and it ran superbly.
Plex's memory footprint simply does not justify a large amount of ram.

What I am actually concerned about is whether or not the ARC cache is large enough, and I would assume there is a metric I can use to determine if I need more ram.

I have spent a significant amount of money on this build already. $2000 and counting, and I don't want to add more ram than is truly actually necessary.

No offense taken. The problem I've found is that, other than for deduplication, there doesn't seem to be any OFFICIAL sizing information. This is why you'll probably get "soft" estimates for how much RAM you need.

Regardless, as you've seen, the general consensus is 8GB isn't enough for the size of your system.

Like you, I wish that TrueNAS would come out with an "official" table so that we can get a more solid sense of how much RAM to get for various system usage cases.

*Sigh*

Scott

 

[jgreco]

TrueNAS would come out with an "official" table so that we can get a more solid sense of how much RAM to get for various system usage cases.

The problem is that there's a relatively solid component to this, but most of the rest is workload sensitive.

For example, you NEED 8GB of RAM, which is based on OS/middleware/network buffering/minimal ZFS ARC needs.

But you'll get people who said "but it works on 4GB". Well, yes, I've run it on as little as 1GB. But just for giggles. And it works horribly-to-not-at-all.

Beyond that, the classic recommendation of 1GB per TB of disk is still very much a thing. As I've written numerous times over the years, this could be considered to be "raw space" or "pool space" or even "consumed space", but this is only likely to get you into a ballpark-reasonable realm. If you try to run 100TB of HDD on an 8GB system, bad things will happen, eventually. There's just too much of a mismatch in size. But you could definitely run 100TB of HDD on 64GB, and even 100TB on 32GB could probably work if your workload wasn't heavy. As your numbers get out past perhaps 24-32GB, there's more elbow room.

Certain workloads like block storage or database storage naturally require more ARC/L2ARC, and need to be sized appropriately. Dedup requires a ton of memory too.

But it's "officially" been in the manual in the past and people just keep asking anyways. The real answer is to see how you're performing, and if your performance is acceptable, then you're probably fine, unless you've got a totally crazy ratio or a tiny amount of memory. Those are the areas for fatalities.

 

[GlueFactoryBJJ]

The problem is that there's a relatively solid component to this, but most of the rest is workload sensitive.

For example, you NEED 8GB of RAM, which is based on OS/middleware/network buffering/minimal ZFS ARC needs.

But you'll get people who said "but it works on 4GB". Well, yes, I've run it on as little as 1GB. But just for giggles. And it works horribly-to-not-at-all.

Beyond that, the classic recommendation of 1GB per TB of disk is still very much a thing. As I've written numerous times over the years, this could be considered to be "raw space" or "pool space" or even "consumed space", but this is only likely to get you into a ballpark-reasonable realm. If you try to run 100TB of HDD on an 8GB system, bad things will happen, eventually. There's just too much of a mismatch in size. But you could definitely run 100TB of HDD on 64GB, and even 100TB on 32GB could probably work if your workload wasn't heavy. As your numbers get out past perhaps 24-32GB, there's more elbow room.

Certain workloads like block storage or database storage naturally require more ARC/L2ARC, and need to be sized appropriately. Dedup requires a ton of memory too.

But it's "officially" been in the manual in the past and people just keep asking anyways. The real answer is to see how you're performing, and if your performance is acceptable, then you're probably fine, unless you've got a totally crazy ratio or a tiny amount of memory. Those are the areas for fatalities.

So... basically what I've said above.

Regardless, IMO, by now ix systems should have enough experience that they could put out a "rule of thumb" guide for memory re: storage, jails, etc.

 

[jgreco]

iXsystems is creating systems for enterprise users, and their sales engineers absolutely provide guidance for memory requirements. Several years ago, I established that they don't sell any TrueNAS systems with less than 64GB RAM, although apparently the FreeNAS Mini has a 32GB option. No iX system comes with less than that.

The support for free product users happens here on these forums. You are currently chatting with the person who wrote some of the earliest iterations of the guidance in the FreeNAS Manual for FreeNAS memory sizing; I was the person who established the 8GB minimum RAM for ZFS, and I've talked extensively in the forums over the years about these issues. I've just finished writing a relatively detailed message in this thread on the topic.

I do not work for iXsystems, and you've really gotten what there is to be had. Use the ZFS TB/GB ARC rule-of-thumb sizing, plus 8GB for base, as a starting point. For jails, increase the RAM according to the running size of the daemons involved. I already explained that Plex, for example, CAN take a lot of memory, but a counterexample by another user suggests less. For VM's, the VM size should be ADDED to the rule-of-thumb I've provided, so if the rule of thumb suggests 16GB and you have a 4GB VM, shoot for 20GB of RAM (probably 24 or 32 in the real world). This is merely a starting point to get you into what is hopefully a reasonable neighborhood. It won't be right for high traffic fileservers, block storage,
or database, all of which can benefit from two to sixteen times the memory my suggestion in this paragraph would otherwise suggest.

The developers at iXsystems are not home hobbyists trying to do things on the cheap, and have relatively little incentive to post information on topics they are relatively unfamiliar with. You really do need to listen to the things you are being told in this thread. There is not going to be a greater authority that comes along with some magic document, I'm sorry. If you are interested in buying a TrueNAS system, I do believe that they'll have a sales engineer consult with you on your needs, but I am guessing that this is an equally subjective analysis of your needs.

 

[imsocold]

Plex's memory footprint is tied to numerous factors, primarily involving database sizes and the number of things the server is doing.



And at one point I was running Plex in a VM with only 4GB of RAM, and needed to boost it to 6GB because it was running like $#!+. ESXi still reports periods where it is using 95% of that 6GB, and some modest swapping at the OS level.



So the correct answer here is to NOT do slot-stuffing, which is where you use a larger number of lower density memory parts. For example, if your system can handle 32GB of RAM in 4 DIMM's, do not use 4GB DIMM's, instead use 8GB DIMM's, and you can start off with just two and see how it goes. Putting 4GB DIMM's in there means you run into a situation where you eventually need to pull the 4GB's, wasting the money you spent to buy those.



You can look at the various ARC statistics to see what the hit rates are. However, you as an administrator need to have some reasonable insight into what sort of ARC rates might be appropriate for your workload. If you are seeing low ARC hit rates but are never accessing the same thing twice, that's perfectly acceptable. However, if you should be benefitting from caching and you don't seem to be getting good hit rates, then adding ARC is called for.

In general, not taking into account any jails, it's a good idea to shoot for 8GB of RAM plus 1GB additional for every TB of used space as a very minimum amount of RAM, at least until you get out past 24 or 32GB of RAM. This typically gets you into a reasonable ballpark for non-crappy ARC performance. Once you get out there to the larger numbers, it becomes more subjective. However, this is really almost always dependent on your specific workload and performance requirements/desires. For example, you can run 32TB of HDD on an 8TB RAM system but performance will be disappointing.

Not to be argumentative but I can only speak from personal experience.
the first time I built a plex vm it had 4GB of ram and over time I noticed that memory utilization was almost nil.
since at the time my esxi box only had 16GB I bumped it down to 2GB with no serious issues, then I bumped it down to 1GB and added a P400 card for hardware transcod. Still low memory usage.
So I bumped it down again because frankly I liked the idea of a lean mean plexing machine.

for reference plex was running on ubuntu server 20 with a A mixture of 4K and 1080p content.

obviously other people have had different experience, but in my experience plex is remarkably lightweight.
hats off to plexs Dev team.

 

[imsocold]

iXsystems is creating systems for enterprise users, and their sales engineers absolutely provide guidance for memory requirements. Several years ago, I established that they don't sell any TrueNAS systems with less than 64GB RAM, although apparently the FreeNAS Mini has a 32GB option. No iX system comes with less than that.

The support for free product users happens here on these forums. You are currently chatting with the person who wrote some of the earliest iterations of the guidance in the FreeNAS Manual for FreeNAS memory sizing; I was the person who established the 8GB minimum RAM for ZFS, and I've talked extensively in the forums over the years about these issues. I've just finished writing a relatively detailed message in this thread on the topic.

I do not work for iXsystems, and you've really gotten what there is to be had. Use the ZFS TB/GB ARC rule-of-thumb sizing, plus 8GB for base, as a starting point. For jails, increase the RAM according to the running size of the daemons involved. I already explained that Plex, for example, CAN take a lot of memory, but a counterexample by another user suggests less. For VM's, the VM size should be ADDED to the rule-of-thumb I've provided, so if the rule of thumb suggests 16GB and you have a 4GB VM, shoot for 20GB of RAM (probably 24 or 32 in the real world). This is merely a starting point to get you into what is hopefully a reasonable neighborhood. It won't be right for high traffic fileservers, block storage,
or database, all of which can benefit from two to sixteen times the memory my suggestion in this paragraph would otherwise suggest.

The developers at iXsystems are not home hobbyists trying to do things on the cheap, and have relatively little incentive to post information on topics they are relatively unfamiliar with. You really do need to listen to the things you are being told in this thread. There is not going to be a greater authority that comes along with some magic document, I'm sorry. If you are interested in buying a TrueNAS system, I do believe that they'll have a sales engineer consult with you on your needs, but I am guessing that this is an equally subjective analysis of your needs.

This is well understood at least by myself, enterprise users are pretty cavalier in the way they spend money.
I should know I have signed off on invoices for the storage team in my office.
that said typically enterprise users have heavy use cases and require 999 uptime, I just need a place to store training videos, bad movies and porn.

all that said, the reason why I asked for a metric is that I could run as lean as possible. I understand that it’s fairly impossible to determine ram requirements for every use case which is why I asked for metrics information.

my intentions was to get to an acceptable level of performance specifically for the storage itself. In all hones the plex service is A secondary concern.

 

[jgreco]

You run as lean as possible by avoiding slot-stuffing, starting on the lean side, and then just slowly increase it as evidence of need is demonstrated.