8GB of ram with 4gb always free

[didolgi]

I have a recycled machine I'm setting up FreeNAS-8.3.1-RELEASE-x64 (r13452) on. I have 8gb of ram reported at 8172mb on the system info tab.

My disk config is, 6x750gb in raidz1 and 8x2tb in raidz2. Both are housing a single iSCSI file extent, and I only have a single Windows 7 iscsi client mounting both disks.

Upon boot I can see the used ram climb to about 4gb and it stops. I have never seen it go above ~4gb used. I always have around 3.7-9 free. I had thought with this much storage ZFS would be eating the 8gb of RAM and asking for more.

Is this normal? or have I setup something wrong?



Thanks

EDIT: I just noticed FreeNAS 8.3.1-RELEASE-p2 is out - going to update to that right now to see if it helps.

 

[ProtoSD]

http://doc.freenas.org/index.php/Settings#Autotune

 

[didolgi]

Yes I do have autotune enabled, I did that as part of the initial install from reading the docs.

Here are the boot loader settings it created

Code:

freenas# cat /boot/loader.conf.local 
vfs.zfs.arc_max="4431943526" # Generated by autotune
vm.kmem_size_max="6155477120" # Generated by autotune
vm.kmem_size="4924381696" # Generated by autotune

Sorry I didn't think to post that in the first part of the thread. I had thought about adjusting the settings, but I assumed they were written by people much smarter than me.

Thanks.

 

[JaimieV]

I haven't used iSCSI with FreeNAS, so may be complete wrong, but... I've seen mentions here in the forums that accessing via iSCSI uses a lot less RAM than "normal" shares of ZFS filesystems, presumably because the client does the read ahead and not the host ZFS?

 

[jgreco]

No, istgt, Samba, and afpd are userland programs. The biggest difference I can think of, from ZFS's point of view, would be that istgt might be asked to use a device extent. There is a possibility that this could imply different semantics to the kernel (for example, a raw disk device doesn't enjoy the benefits of caching), but for ZFS, with respect to memory consumption, it appears that it mostly does very similar things to the file extent case.

 

[JaimieV]

Thanks - I'll keep that in mind. Apologies for minor thread derailment - nothing to see here, move along!

 

[didolgi]

I had wondered about the iSCSI vs. CIFS or NFS also. Given the fact that there is only "two" files on the entire FreeNAS box there isn't much in that respect for ZFS to keep track of. The block info of course is still busy as always. It just seems odd to me the line is right at 4gb, knowing the 32 vs 64 limits. The OS is reporting the 8gb, and the FreeNAS version is clearly x64, so I know thats not it, but its still in the back of my mind...

As a test if I were to tweak the autotune values by say 25% and reboot the worse thing would be poor performance, or a possible kernel panic - right?

 

[cyberjock]

Can you post your hardware? Some motherboards from days gone by supported 64 bit CPUs, but you couldn't actually use more than 4GB of RAM because of chipset limitations. I'm wondering if that's the problem here. It is very peculiar to say the least.

 

[didolgi]

Of course!

Also, I did update to FreeNAS-8.3.1-RELEASE-p2-x64 (r12686+b770da6_dirty), with no change.

ASUS Maximus Formula http://www.asus.com/Motherboards/MAXIMUS_FORMULA/
Intel X38 chipset http://ark.intel.com/products/31919/Intel-82X38-Memory-Controller
RAM: DDR-800 4x2MB DDR2
CPU: Intel C2Q 2.4(stock) been stable overclock at 3.005 for more than 5 years.

Code:

freenas# dmesg | grep -i memory
real memory  = 8589934592 (8192 MB)
avail memory = 8242028544 (7860 MB)
freenas# dmesg | grep -i icpu 
freenas# dmesg | grep -i cpu
CPU: Intel(R) Core(TM)2 Quad CPU           @ 2.40GHz (3005.57-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs

 

[paleoN]

Code:

freenas# cat /boot/loader.conf.local 
vfs.zfs.arc_max="4431943526" # Generated by autotune
vm.kmem_size_max="6155477120" # Generated by autotune
vm.kmem_size="4924381696" # Generated by autotune

Where's the mystery? You have vfs.zfs.arc_max set to slightly over 4GB. You told the system to use no more. It didn't.

I haven't used iSCSI with FreeNAS, so may be complete wrong, but... I've seen mentions here in the forums that accessing via iSCSI uses a lot less RAM than "normal" shares of ZFS filesystems, presumably because the client does the read ahead and not the host ZFS?

If I'm not mistaken those people were directly using the disks without any filesystems on them.

 

[didolgi]

Where's the mystery? You have vfs.zfs.arc_max set to slightly over 4GB. You told the system to use no more. It didn't.

Ah! That goes back to my earlier statement about assuming autotune's values were optimal. I'll try adjusting the value up in 25% increments.

I realize autotune has to deal with everything under the sun so it can't be optimal in all cases.

 

[paleoN]

Ah! That goes back to my earlier statement about assuming autotune's values were optimal. I'll try adjusting the value up in 25% increments.

Then why didn't you look up what the values did? Autotune is quite conservative. Furthermore, it's a bit outdated. With the newer ZFS code, or associated code, you shouldn't need to specify vm.kmem settings. Disable autotune and delete the three autotuned settings under Tunables. Alternatively, leave vfs.zfs.arc_max and set it to 7GB or 6GB depending on what else your NAS is doing.

 

[didolgi]

Then why didn't you look up what the values did? Autotune is quite conservative. Furthermore, it's a bit outdated. With the newer ZFS code, or associated code, you shouldn't need to specify vm.kmem settings. Disable autotune and delete the three autotuned settings under Tunables. Alternatively, leave vfs.zfs.arc_max and set it to 7GB or 6GB depending on what else your NAS is doing.

I did look up what the values did, but it appeared to me that autotune was reserving 4gb for Kernel VM and 4GB for ARC. I assumed that would add up to the 8gb of total RAM in the box. I asked the question because it seems there is quite a bit of conflicting advice on the use of autotune. The values autotune used for my network buffers did improve performance for me so I thought it had made some reasonable choices. Now I see that not all the choices were. I'll remove all 3 values from the GUI and disable autotune.

Thanks.

 

[didolgi]

Yep - that was it. I removed the 3 autotune created values, and disabled it. Upon reboot restarted all the traffic and got the following:



Running with 1.6gb of free RAM now. Lesson learned - don't trust autotune.

 

[paleoN]

Running with 1.6gb of free RAM now. Lesson learned - don't trust autotune.

Or simply understand it. ;)