After upgrade to 9.10.2 my ARC hit ratio dropped significantly

Status
Not open for further replies.

toadman

Guru
Joined
Jun 4, 2013
Messages
619
I recently upgraded from 9.10.1-U4 to 9.10.2 and saw a significant drop in ARC hit ratio. For the last year the hit ratio was reported at an average of 95%. Now it's reported at 68%. The ARC size has remained constant at about 6GB (on an 8GB system). The system is a lightly loaded home system serving media to a few clients and hosting an NFS datastore for an esxi host with about 7 lightly loaded VMs running.

Anyone else seeing this after an upgrade to 9.10.2? I'm curious if it's a difference in the reporting vs. an actual change in ARC performance. I have yet to do any tests on the performance myself, but there doesn't appear to be a noticeable change in responsiveness from any interactive client as of yet.

Freenas 9.10.2
Running in a VM with 8GB memory and 2 vCPU
ESXi 6.0 host with AMD 1100T, ASUS M5A97 r2.0, 32GB Crucial ECC memory
Storage is 6x2TB and 4x3TB WD RED in 5x2 mirrors running off an passed through LSI 9211-8i in IT mode
 

Attachments

  • freenas.jpg
    freenas.jpg
    81 KB · Views: 215

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338

toadman

Guru
Joined
Jun 4, 2013
Messages
619
Brilliant. Yep, snapshots enabled. Thanks for the pointer! 16901 doesn't look to be a publicly visible bug. But it's certainly a similar issue as the poster on the link you provided.
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
Next question: Are the scheduled snapshots executed with a delay after the upgrade on your system, too?

On my system, snapshots are scheduled from 6:00 though 23:00 every 6 hours. In previous FreeNAS versions this lead to execution at 6:00/12:00/18:00. In 9.10.2, snapshots scheduled this way are executed at 6:31/12:31/18:31.
 

toadman

Guru
Joined
Jun 4, 2013
Messages
619
There is no delay on mine. Or rather it's expected. The snapshots for 3 datasets fire at 1am as scheduled every time. However my 4th is scheduled to fire every hour all day long. If I have to take the system down for some reason and it's been beyond 1 hour since the last snapshot when rebooting, then this 4th one fires immediately. i.e on the minute when the system comes up.

So in your case if the system was down for some reason and booted at 6:31 AM that sequence would explain what you are seeing. i.e. it's been more than 6 hours and 6:31 was the time when the snapshot task ran. So boom, it snapped. It then makes sense it would snap at 12:31 and 18:31.
 

MrToddsFriends

Documentation Browser
Joined
Jan 12, 2015
Messages
1,338
So in your case if the system was down for some reason and booted at 6:31 AM that sequence would explain what you are seeing. i.e. it's been more than 6 hours and 6:31 was the time when the snapshot task ran. So boom, it snapped. It then makes sense it would snap at 12:31 and 18:31.

As far I can tell, this behavior is new in 9.10.2 and is caused by this commit:
https://github.com/freenas/freenas/commit/baab2ecce52291a36b5c63fa7382f8a916be7918

I must admit, though, that I don't speak Python, but I never saw snapshots being fired at other times than at 6:00/12:00/18:00 when configured as described above, regardless of the time of the last boot, in prior 9.3/9.10 versions of FreeNAS.
 
Last edited:

toadman

Guru
Joined
Jun 4, 2013
Messages
619
Yep, even my 1:00am 1x per day snapshots are at 1:13am now. Interesting.
 
Status
Not open for further replies.
Top