Recommendations for 256GB RAM upgrade

[wtfR6a]

I suspect my 'small' home server is reaching the point where I need to start tuning some settings, it looks like it hasn't correctly configured itself for the recent RAM upgrade it received. Before I go fiddling - any advice of war to tune and what to leave alone. Id rather not get involve din auto-tune, i'd prefer to understand whats being changed and why.

 

[Spearfoot]

That image looks like you just now booted the server? Usually you have to run FreeNAS under load for several days to see how memory ends up being used.

Is it possible you have more RAM than you really need!? I didn't know such a thing was possible with ZFS! :)

 

[wtfR6a]

Its been up for a paltry 12 hours under somewhat moderate use although I would have expected ARC use to continue to rise until free RAM was consumed. Strikes me as odd that it tailed off as its approached the previous max of circa 100GB of 128GB fitted.

 

[jgreco]

Make sure you don't have any old autotune settings sitting around (System->Tunables). Delete and reboot if you do.

ARC doesn't just cache anything and everything it can. It's fairly intelligent, so unless you've got very heavy data access patterns, it's possible that it has merely identified the approximate size of your working set as being around 100G. In such a case, ARC will likely show some additional growth over time, but it is also very likely that additional memory isn't doing all that much for you.

 

[jgreco]

Is it possible you have more RAM than you really need!? I didn't know such a thing was possible with ZFS! :)

Sure, put 1TB of RAM on a system with a 500GB pool.

 

[wtfR6a]

Thanks for the feedback jgreco. Im sure the working set is larger than 100GB as its currently a ~30TB dataset with fairly random access patterns so I would have expected some additional caching overnight however its ruler flat. I'll let it soak over a few days and see how it progresses.

I've never touched auto-tune for fear of the FreeNAS gods raining down on me so theres definitely no old settings hanging around in there.

Heres the seemingly important sysctl's

Code:

vfs.zfs.arc_max: 266604064768
vm.kmem_size_max: 1319413950874
vm.kmem_size: 267677806592

and dashboard stats of this morning.

 

[wtfR6a]

the reboot gods fixed everything.....same sysctl's but different results.

Code:

vfs.zfs.arc_max: 266604064768
vm.kmem_size: 267677806592
vm.kmem_size_max: 1319413950874