Is my 32GB RAM being properly recognized and utilized?

[diedrichg]

• FreeNAS 11.2-U2.1 STABLE
• Supermicro X10SLM-F-O uATX
• Intel Xeon E3-1270 v3
• Kingston 32GB (8GB x 4) ECC w/Hynix DRAM, 1333MHz, 1.5V - KVR1333D3E9SK2/16G
• (4) WD Red 2TB in RaidZ2
• Boot drive: Supermicro 16GB SATADOM - SSD-DM016-PHI

I recently upgraded my RAM from 16GB to 32GB. I did the upgrade while still on 11.1-U7. I saw 32GB in the Reporting section and was happy but I didn't pay attention to how much was actually being used. With 16GB FreeNAS was using the full amount - cool.

I then upgraded to 11.2-U2.1 and that's when I noticed that the system was recognizing 32GB as being available but my reporting sections show the system only using what appears to be exactly 16GB.

Is my FreeNAS installation somehow stuck at using 16GB? Shouldn't it be using the full 32GB?

 

[Mlovelace]

• FreeNAS 11.2-U2.1 STABLE
• Supermicro X10SLM-F-O uATX
• Intel Xeon E3-1270 v3
• Kingston 32GB (8GB x 4) ECC w/Hynix DRAM, 1333MHz, 1.5V - KVR1333D3E9SK2/16G
• (4) WD Red 2TB in RaidZ2
• Boot drive: Supermicro 16GB SATADOM - SSD-DM016-PHI

I recently upgraded my RAM from 16GB to 32GB. I did the upgrade while still on 11.1-U7. I saw 32GB in the Reporting section and was happy but I didn't pay attention to how much was actually being used. With 16GB FreeNAS was using the full amount - cool.

I then upgraded to 11.2-U2.1 and that's when I noticed that the system was recognizing 32GB as being available but my reporting sections show the system only using what appears to be exactly 16GB.

Is my FreeNAS installation somehow stuck at using 16GB? Shouldn't it be using the full 32GB?

Do you have any tunable values that would set a limit on the size of the ARC? Such as running the 'autotune' function (vfs.zfs.arc_max comes to mind).

 

[diedrichg]

Do you have any tunable values that would set a limit on the size of the ARC? Such as running the 'autotune' function (vfs.zfs.arc_max comes to mind).

No. No tunables set.

 

[Mlovelace]

No. No tunables set.

What's the output of sysctl -a | grep vfs.zfs.arc ?

 

[diedrichg]

What's strange is that Top is reporting 15G wired 15G available as in my screenshot from above.

What's the output of sysctl -a | grep vfs.zfs.arc ?

Code:

[root@freenas ~]# sysctl -a | grep vfs.zfs.arc                                
vfs.zfs.arc_min_prescient_prefetch_ms: 6                                      
vfs.zfs.arc_min_prefetch_ms: 1                                                
vfs.zfs.arc_meta_limit: 8083366912                                            
vfs.zfs.arc_free_target: 173801                                                
vfs.zfs.arc_kmem_cache_reap_retry_ms: 0                                        
vfs.zfs.arc_grow_retry: 60                                                    
vfs.zfs.arc_shrink_shift: 7                                                    
vfs.zfs.arc_average_blocksize: 8192                                            
vfs.zfs.arc_no_grow_shift: 5                                                  
vfs.zfs.arc_min: 4041683456                                                    
vfs.zfs.arc_max: 32333467648

 

[Mlovelace]

What's strange is that Top is reporting 15G wired 15G available as in my screenshot from above.

Code:

[root@freenas ~]# sysctl -a | grep vfs.zfs.arc                               
vfs.zfs.arc_min_prescient_prefetch_ms: 6                                     
vfs.zfs.arc_min_prefetch_ms: 1                                               
vfs.zfs.arc_meta_limit: 8083366912                                           
vfs.zfs.arc_free_target: 173801                                               
vfs.zfs.arc_kmem_cache_reap_retry_ms: 0                                       
vfs.zfs.arc_grow_retry: 60                                                   
vfs.zfs.arc_shrink_shift: 7                                                   
vfs.zfs.arc_average_blocksize: 8192                                           
vfs.zfs.arc_no_grow_shift: 5                                                 
vfs.zfs.arc_min: 4041683456                                                   
vfs.zfs.arc_max: 32333467648

This looks normal, and as you stated the 32GB is recognized correctly. You could try running some read tests from the pool and monitor the ARC.

 

[diedrichg]

This looks normal, and as you stated the 32GB is recognized correctly. You could try running some read tests from the pool and monitor the ARC.

Okay, thanks. It was my understanding that FN would use all available RAM - I've never had over the 16GB so I've not experienced this proportional usage before. I also found it strange that it was exactly half the RAM I have now and the max that I had previously.

Pictured is up-time of about a week with a reboot today, Monday.

 

[diedrichg]

This looks normal, and as you stated the 32GB is recognized correctly. You could try running some read tests from the pool and monitor the ARC.

Check this out though. You can see where I had just the 16GB RAM and then installed the 32GB. At the far right you see the 32GB max out at the 16GB limit just prior to today's reboot.

 

[jgreco]

So you upgraded on 05-Mar, had massive free RAM on 07-Mar, and sllllllowly dwindled it until around 11-Mar.

This really looks to me like you're not DOING anything with your NAS, what little you're doing is fitting within ~10-15GB of ARC.

The gradual increases towards the ~15-16GB limit suggest that you just happened to notice this at the perfect time, right before it might have consumed a little more than 16GB. You needed to wait a little longer. Your resource consumption is on a slow path.

Repeat what you've done, put more workload on the box.

And I'll say a very unusual thing here, something you rarely hear in ZFS-land: you might not have needed the extra RAM.

 

[diedrichg]

So you upgraded on 05-Mar, had massive free RAM on 07-Mar, and sllllllowly dwindled it until around 11-Mar.

This really looks to me like you're not DOING anything with your NAS, what little you're doing is fitting within ~10-15GB of ARC.

The gradual increases towards the ~15-16GB limit suggest that you just happened to notice this at the perfect time, right before it might have consumed a little more than 16GB. You needed to wait a little longer. Your resource consumption is on a slow path.

Repeat what you've done, put more workload on the box.

And I'll say a very unusual thing here, something you rarely hear in ZFS-land: you might not have needed the extra RAM.

Thank you, sir. I appreciate the feedback. I'll just let it sit, and you are right - we really don't do much to the box on a daily basis since it's a basic file server.