Tunables - kmem_size_max / kmem_size / zfs.arc_max

[Pointeo13]

Currently I have my home freenas server installed with over 1TB of memory, I was wondering if someone could point me in the right direction or give me an idea on how I should tune my vm.kmem_size_max, vm.kmem_size and vfs.zfs.arc_max. If I understand correctly vm.kmem_size_max should be = or > the vm.kmem_size and vfs.zfs.arc_max should be less then the vm.kmem_size. As of right now freenas boots fine at these settings:

vm.kmem_size_max=920000000000
vm.kmem_size=910000000000
vfs.zfs.arc_max=900000000000

Late last night I was trying to adjust the sizes even higher, but it wouldnt boot so I set it back, am I heading in the right direction or am I doing this all wrong with the vm.kem_size_max and vm.kmem_size? Thanks for the help guys!

 

[mav@]

Usually I would say it may be reasonable to set kmem_size to 1.25x-1.5x of physical RAM size (or 1.5x-1.75x of ARC size) to not suffer from its fragmentation after some run time under high pressure. But when things are going about 1TB of RAM, that is quite terra incognita to recommend something for sure.

 

[toadman]

I had to reread the original post. It said "home freenas server" and "over 1 TB of memory" in the same sentence. :)

That is awesome. I can't recall a discussion of a system with a larger memory footprint. Wow.

I can't help your question directly, but I do recall some similar discussions of 384-512GB systems with similar issues. It may be an underlying FreeBSD issue if I remember correctly.

Try a good search on the forums here and maybe even a general search (not freenas forums) on large memory FreeBSD systems.

 

[Ericloewe]

home freenas server installed with over 1TB of memory

That moment when you go "surely he meant disk", only to realize it is actually RAM.

The largest system seen on the forum until now was probably the odd 512GB system.

 

[Pointeo13]

Thanks mav, I'll give that a formal a go tonight, this time I'll slowly crank up the numbers to hopefully achieve the correct kmem and arc sizes.

Hey toadman, I remember that thread, I was actually trying to look for it, no such luck yet.

lol Ericlowew, if this all works out, I will have to order IBM Max5 which will give me another 32 DIMMs so I could add another 512GB Memory :)

 

[toadman]

Maybe this applies... http://freebsd.1045724.n5.nabble.co...e-capped-at-329853485875-307GB-td5734895.html

 

[Pointeo13]

Thanks toadman, as for an update, I spent most of the night tweaking the vm.kmem_size trying to find out the max number (until it panics) before I do any additional changes and the number I came up with is 896G. So as of right now my settings are

vm.kmem_size_max=1015G
vm.kmem_size=896G
vfs.zfs.arc_max=890G

I'll spend tonight reading what toadman posted and see if those tricks can get my vm.kmem_size higher without a panic during bootup.

 

[Pointeo13]

Wanted to give an update, I decided to let it sit for 30 day's before I made any more changes, it looks like my max arc size hit's around 673GB, tonight I'll begin to see if I can increase this...

 

[toadman]

Interesting. Thanks for posting. Out of curiosity what does the ARC hit ratio look like over the 30 days?

 

[Pointeo13]

Interesting. Thanks for posting. Out of curiosity what does the ARC hit ratio look like over the 30 days?

This is what I posted in the thread, "What are you Arc statistics" a few weeks ago

[*]59.3TiB / 97.5TiB (Raidz_2_SATA) ***Archive Media / Backups***
[*]1.11TiB / 3.23TiB (Raidz_Mirror_SAS) ***VMware VM's***
[*]2.75GiB / 14.8GiB (freenas-boot)
[*]669.98GiB (MRU: 601.51GiB, MFU: 68.50GiB) / 1.00TiB
[*]Hit ratio -> 80.68% (higher is better)
[*]Prefetch -> 64.43% (higher is better)
[*]Hit MFU:MRU -> 68.67%:27.14% (higher ratio is better)
[*]Hit MRU Ghost -> 0.06% (lower is better)
[*]Hit MFU Ghost -> 0.59% (lower is better)

 

[tvsjr]

97.5TiB for your SATA array? Assuming those are 4TB drives in RAIDz2, that's, what, 36 drives?

Pics or it didn't happen.

 

[Pointeo13]

97.5TiB for your SATA array? Assuming those are 4TB drives in RAIDz2, that's, what, 36 drives?

Pics or it didn't happen.

I wasn't going to post pics yet until I'm done, but here is a sneak peek, first array 24 300GB 15K SAS drives, second array 36 3TB 7200RPM drives (12 drives on backside), third array that will be taking place of the first array 24 600GB 15K SAS drives, then I'll re-purpose the first array for 24 4TB drives to expand the second array. A lot more changes will come in about a month, then I'll give an updated pic.

 

[Pointeo13]

I spent a couple hours tonight trying to increase the arc size above 673GB but still no luck. I am struggling to increase the vm.kmem_size above 896G, when I do I keep receiving kernel panics. Well off to bed, I'll have to try again tomorrow night.

 

[Darren Myers]

I wasn't going to post pics yet until I'm done, but here is a sneak peek, first array 24 300GB 15K SAS drives, second array 36 3TB 7200RPM drives (12 drives on backside), third array that will be taking place of the first array 24 600GB 15K SAS drives, then I'll re-purpose the first array for 24 4TB drives to expand the second array. A lot more changes will come in about a month, then I'll give an updated pic.

<picture of awesomeness here>

See....if i brought home a 42U rack, and couldnt hide it the misses would murder me, if i then managed to hide it and load it with x4 4U and a 2U server and STILL hide it i would be sooooooooo happy.... /sigh, guess ill make due with my dinky lab :/

 

[jgreco]

See....if i brought home a 42U rack, and couldnt hide it the misses would murder me, if i then managed to hide it and load it with x4 4U and a 2U server and STILL hide it i would be sooooooooo happy.... /sigh, guess ill make due with my dinky lab :/

Go in the backyard, dig a nice big 20'x20' hole about 20' deep, build a foundation and some walls, lay some spancrete on top, waterproof, and make a tunnel to your house. Cover with dirt. Instant evil genius secret lab.

I'd love to have an old missile silo for an evil lair.

 

[Darren Myers]

I'd love to have an old missile silo for an evil lair.

Could we share said evil layer? We can invite @cyberjock for some datahoarding goodness.

I'm also ok with the airstrip...but id prefer a underwater entrance for cool factor

 

[Pointeo13]

See....if i brought home a 42U rack, and couldnt hide it the misses would murder me, if i then managed to hide it and load it with x4 4U and a 2U server and STILL hide it i would be sooooooooo happy.... /sigh, guess ill make due with my dinky lab :/

LOL, yeah I love having my setup, I couldn't live without it, I'm also lucky because I'll be moving soon to an area that will give me 1Gbps or 10Gbps internet for a very reasonable price. I'm starting with the 1Gbps and then in a year I'll sign up for the 10Gbps.

 

[vikingboy]

cool - you can run an ISP and back up the NSA then :) Thats an impressive system, thanks for sharing and good luck sorting the memory issue out.

 

[jgreco]

Could we share said evil layer? We can invite @cyberjock for some datahoarding goodness.

I think there are probably bigger datahoarders around than CJ.

I'm also ok with the airstrip...but id prefer a underwater entrance for cool factor

I figure that sooner or later Roughs Tower might be for sale. (obDisclosure: I know one of the HavenCo founders).

 

[cyberjock]

That evil genius secret lab is amazing. Count me in!