Tunables - kmem_size_max / kmem_size / zfs.arc_max

[Ericloewe]

Gotta love the personal runway, though you have to wonder how it came to be...

"We're going to build massive, unbelievably expensive underground missile silos and hide them. Then we'll build a totally inconspicuous runway right next to them, so the soviets can spot them more easily."

 

[jgreco]

I expect the runway was an add by the people who bought it from the military. Besides, some of these installations date from a time before high res satellite imagery.

 

[Ericloewe]

I expect the runway was an add by the people who bought it from the military.

That's the only conclusion I arrived at that made sense.

Besides, some of these installations date from a time before high res satellite imagery.

Well, runways tend to be conspicuous things. :D

 

[jgreco]

Think that was the one up in Alaska. Lots of homes and cabins that have no easy access to primary roads have air strips up there...

 

[Pointeo13]

Wanted to give another update on the thread, not only did FreeNas 9.10 fix my delete issue, it also looks like I can now surpass my previous max ARC Size of 673GB. For testing I did:

vm.kmem_size = 800G
vm.kmem_size_max = 800G
vfs.zfs.arc_max = 780G

As of right now I hit a max Arc Size of 790.5G with these settings, this week I'll continue to test to see how big I can get my ARC Size by increasing those numbers since I have over 1TB of Memory. Thank you so much freenas developers for this release, my freenas has never ran so smooth before.

 

[jgreco]

That's pretty impressive.

 

[bestboy]

I just checked my tunables and they are kinda weird

Code:

~# sysctl -a | egrep "vm.kmem_size|hw.phys|kern.maxbcache|vfs.maxbufspace|vfs.zfs.arc_max"
kern.maxbcache: 0
vm.kmem_size_scale: 1
vm.kmem_size_max: 1319413950874
vm.kmem_size_min: 0
vm.kmem_size: 33266360320
vfs.zfs.arc_max: 32212254720
vfs.maxbufspace: 1735573504
hw.physmem: 34291179520

vm.kmem_size_max is set to 1.2 TiB for my puny 32 GiB RAM. Is this overshooting normal or is this maybe an autotune artefact? I guess I had autotune enabled once in the beginning when I set up the box in 9.1 or so.
I'm inclined to fix that and set vm.kmem_size = vm.kmem_size_max = 33266360320.

ZFS does not, however, use the regular file system buffer cache. The ARC
takes its place, and the ARC abuses the kernel's heap like nothing
else. So, if you are running a machine that only makes trivial use of a
non-ZFS file system, like you boot from UFS, but store all of your data
in ZFS, then you can dramatically reduce the size of the buffer map via
boot loader tuneables and proportionately increase vm.kmem_size.
[...]
Basically, for every byte that you subtract from vfs.maxbufspace, through setting
kern.maxbcache, you can add a byte to vm.kmem_size{,_max}.

I'm also thinking about reducing vfs.maxbufspace from 1.6 GiB to 256 MiB by setting kern.maxbcache = 268435456 as there are no file systems other than ZFS. Would this be useful now that even /boot is on ZFS?

 

[Pointeo13]

bestboy, yeah for whatever reason auto tune sets vm.kmem_size_max super high like that, I went ahead and changed it to match my vfs.zfs.arc_max, it really shouldn't matter from what I'v seen and read, you can change it or leave it. As for your second question, not sure, mine is also set

vfs.maxbufspace: 1735573504

In the end it probably doesn't really matter.

 

[bestboy]

Alright then it's settled. I'm going to set vm.kmem_size_max manually.

I'm not so sure about vfs.maxbufspace, tho. 1.6 GiB might be neglectable when you got 1 TiB RAM to begin with. But for home user NASes in the common range of 16 to 32 GiB this (unused?) kernel buffer takes up 5% - 10% of the total RAM.