Tweaking RAM

[William Bravin]

Hello all

my 2 truenas systems are running fine. I have been running rsync tasks for the last 3 weeks. I feel that the rsync processes are somewhat slow.

However I notice that most of my system RAM on my 2 servers is used up by zfs cache and i have little to no available ram left

As I read most posts relating to RAM configuration , I see that this is a normal occurrence and apparently the only solution is to add ram.

I am not really in favour of installing additional RAM just to have it eaten up by the cache..

Since Caching is not a process and only processes should use ram, is there a way i can move the ZFS cache to a 120gb ssd or USB?

by doing this, would this improve my rsync processes?

Thank you for reading this an and for taking the time to respond

 

[Chris Moore]

I feel that the rsync processes are somewhat slow.

They are. That is just a feature of rsync.

However I notice that most of my system RAM on my 2 servers is used up by zfs cache and i have little to no available ram left

Also normal. What else do you need it for. ZFS uses RAM as a cache for IO.

and apparently the only solution is to add ram.

If you need more performance, more RAM might help, depending on what you are doing, but that won't make rsync faster.

I am not really in favour of installing additional RAM just to have it eaten up by the cache..

Again, what else are you going to use it for. It is perfectly normal and expected to have most of RAM used by ZFS cache, I have systems at work with 256GB of RAM and 80% of it is consumed with cache.

Since Caching is not a process and only processes should use ram, is there a way i can move the ZFS cache to a 120gb ssd or USB?

What are you talking about? USB is super dangerous to begin with because it can just randomly stop working. Just don't use it.

Why are you so wrapped up with freeing RAM? What do you plan to use it for? FreeNAS and TreuNAS are NAS operating systems. Their purpose is to feed data to the LAN and that is what the cache is for, IO caching, why would you want to change that? Makes no sense what you are asking.

What you are thinking might be true on some other OS, but the system is working the way it is intended. If you just want to make the backup go faster, there are other ways, but that wasn't your question.

 

[jgreco]

Since Caching is not a process and only processes should use ram,

What a quaint theory. Where does your kernel live in such a world? Where do your network buffers live in such a world? Just two obvious counterexamples.

I'm curious where you got such an idea.

 

[sretalla]

Since Caching is not a process and only processes should use ram, is there a way i can move the ZFS cache to a 120gb ssd or USB?

by doing this, would this improve my rsync processes?

I think you're a little mixed up on how things work, but your exact case is discussed here:

L2ARC impact on rsync performance for largely dormant data

I've been doing some more testing and the evidence suggests that using a L2ARC with a metadata only flag has a potential substantial positive impact on rsync performance over AFP. For largely dormant data, the combination of L2ARC dedicated to metadata seems to benefit rsync performance a lot...

www.truenas.com

You could use that together with Persistent L2ARC for even better results under TrueNAS 12 +.

How do I enable persistent L2ARC?

Keep seeing, "L2ARC rebuild disabled" on startup. Is there a flag I need to set to enable this new OpenZFS feature?

www.truenas.com

 

[William Bravin]

Gee Thank you all for responding so quickly.

Maybe I am just pedant, I do not use my solution for anything more than to distribute my music, tv shows, movies and document to my other machines. Simple I just do not like seeing that i am maxing out on any component.

I use rsync to backup my data to a second server once a week (so eventually i can learn how to make the second server redundant to the first )

Even as Chris Moore said even if i increase the ram to 64 gb on both machines this will not improve

jgreco I see cache as just a temporary location for IO therefore I see it as a location that needs quick access. I feel that , as you said the kernel, the processes and the instruction sets, are that really need superfast IO (hence in RAM). In this case the zfs cache would have ocean of space to "buffer" the IO in the background and the save the information to it's destination (again in the background.

sretalla thank you for the links I will look into them

I am by no means a tech savvy guy so i may be talking gibberish

Once again thank you all ... an please be all safe out there.

 

[ThreeDee]

TrueNAS ...this is not your father's Windows XP ..

Unused RAM is wasted RAM

 

[Arwen]

@William Bravin, the others did not mention that FreeBSD, (the base OS used in FreeNAS/TrueNAS), will AUTOMATICALLY free up ZFS cache IF needed for a program.

So, their is no reason to leave RAM fallow. Unlike a farmer's field, RAM still has to be powered on and working for the entire up time of a server.

Their are slight exceptions to this. I have seen, (and extremely recently), programs that improperly check for free RAM before allocation. And finding that their is not enough RAM free, fail to launch. However, if this program did the correct thing and simply asked for the RAM, it would have succeeded. My team & back-line support both agree that this is a bug in the software. Note that was seen on Solaris 10 or 11, not FreeBSD or FreeNAS/TrueNAS.

 

[danb35]

Simple I just do not like seeing that i am maxing out on any component.

Why not? You paid for the RAM, why would you want to leave a significant portion of it unused?

 

[William Bravin]

Thank you all for your thoughts.

@Arwen I would agree with you.

Now that rsync has finally finishes I truenass is still using 10.7gb for ZFS, 5.3 for services and leaving 0;.5 available. I do not believe that this is correct.

In restarting the server, the usage of the ram becomes zfs cache 1 gb services 3.9 and finally 10.9 available.

This i understand and feel it is correct and the way it should be when a process is finished .

I should not need to restart the server to release the used ram. Once a process /task is finished the ram usage should revert to the above state as in after restarting. It does not.

Yes rsync is a hog and maybe more ram (to 64GB ) maybe improve performance.

I know nothing about L2ARC that @sretalla mentioned here above and i will look into it. from the looks of it this seams like a possible solution

I still feel that the IO cache for any program should be relegate to a storage media not to ram.

With ssd speed of 500 to 600 MB/s, IO would not be taxed too much compared to ram. it is easy and cheaper to add 250gb or 500gb ssd for cache and the process would have available space to swim. Ram on the other hand is more expensive to purchase and limited in possible size and not as easily expandable.

In this case, truenass processes would have more ram to run

Once3 again thank you for your contribution

 

[ThreeDee]

hmm .. you are still not understanding how RAM is used/being used/will be used.

 

[William Bravin]

I could be wrong it's a question of process design.

I am always ready to learn. One thing I know is that when a process terminates ram used by that process should be released.

As i mentioned I needed to restart truenas to to clear the ram occupied by the zfs process. I should not need to do this

 

[ChrisRJ]

The (simplified) point is that RAM that is not needed otherwise, will be used by ZFS as cache. If another program is started, the amount of RAM allocated for ZFS will dynamically be reduced, and the "released" RAM be made available to the "new" program. This is a very common approach, that you can even see (kind-of) in languages like Java, where the Java Virtual Machine will dynamically handle memory (this is also highly simplified).

 

[danb35]

This i understand and feel it is correct and the way it should be when a process is finished .

Your understanding and feeling is incorrect. Period.

I still feel that the IO cache for any program should be relegate to a storage media not to ram.

That completely defeats the purpose of cache. Cache is stored in RAM so that it is accessible more quickly than if it were on the disk--even a SSD is orders of magnitude slower than RAM.

I am always ready to learn.

Your posts in this thread demonstrate otherwise.

This isn't a new question; it's come up several times before, and no doubt a forum search would have found several instances of the same answer you've gotten several times. But you seem to be more stubborn in your ignorance than is common.

 

[Dan Tudora]

hello
somebody say sometime (long time a go) and sound like "640 KB of RAM must to be enough for any program to run "
I do not remember who say that !!

 

[William Bravin]

Thank you all for responding
@ChrisRJ thank you for taking the time and your patience in explaining it to me. I understand what you are saying, however why do i need to restart the server to clear the ram. Should the ram be cleared when the process is finished

@danb35 I take your point. Yes i am stubborn and yes i am not an it guy. therefor i learn as you did when you started in this trade.

Thank you all

 

[William Bravin]

hello
somebody say sometime (long time a go) and sound like "640 KB of RAM must to be enough for any program to run "
I do not remember who say that !!

You are absolutely right. Bill Gates did

 

[danb35]

i am not an it guy

Neither am I--I'm a lawyer.

i learn

No, you refuse to learn. You've been given the answer several times in this thread, but you won't listen. You're stuck, for reasons known only to you, on the idea that the RAM needs to be "cleared"--when to do so would be not only useless, but counterproductive (it would be exactly the opposite of the purpose of cache).

For the nth time: the RAM is being used to cache ZFS data. If something else needs it, it will be released to whatever that "something else" is. The system is operating perfectly normally; the way you expect it to operate would be very broken.

 

[ChrisRJ]

Should the ram be cleared when the process is finished

What benefit would that bring? I must admit, that I am just as "suprised" as @danb35 on why you insist that the system should do this. And the argument about memory being released when an OS process terminates, is not relevant anyway because the "ZFS file system process" only terminates upon shutdown of the OS. So in that light the system behaves exactly as you argue that it should.

 

[ThreeDee]

I could be wrong it's a question of process design.

I am always ready to learn. One thing I know is that when a process terminates ram used by that process should be released.

As i mentioned I needed to restart truenas to to clear the ram occupied by the zfs process. I should not need to do this

only correct part of that post is "I should not need to do this" ..

 

[Arwen]

Thank you all for your thoughts.

@Arwen I would agree with you.

Now that rsync has finally finishes I truenass is still using 10.7gb for ZFS, 5.3 for services and leaving 0;.5 available. I do not believe that this is correct.
...

You are wrong. That IS correct.

This is EXACTLY what RAM cache is for, to be USED.

Think of it this way. You have a bus that goes between point A and point B, every hour. Their are high priority passengers that must make the journey, so they board first. But, after that, any available seats can be given to standby passengers. After all, the bus is going regardless of how many people are on it. So the high priority passengers are programs, and the standby is the ZFS cache.

Please understand, ZFS was designed by big boys, aka Sun Microsystems, for their server oriented OS, Solaris. To apply home PC or laptop constraints on a server OS leads to confusion.