Nightly increase in CPU Usage

[jerryjharrison]

I have reduced my Freenas installation to be as vanilla as possible. There are NO Cron jobs enabled, No plugins, No Jails, 1 ZFS Pool with a Long SMART test enabled 2 times a month, and a short SMART test 2 times a day (not at midnight). I am getting no other errors.

For some reason, the cpu usage increases at Midnight (00:00) every night by about 20%. This happens every night until after 5 nights it locks out at 100%, and I start getting Page swap errors, whereupon I reboot and start the process all over again.

Report 1 is the domed in version from last night.
Report 2 is a bit longer term from last week, so you can see the step pattern. The reductions in CPU image are all reboots.

I am not sure what logs to pull, but can get any of them necessary.

I would appreciate any direction you can provide.

 

[cyberjock]

How about hardware specs, freenas version, etc. You know.. all that stuff asked for in the forum rules when making a new post...

 

[jerryjharrison]

Sorry about that... I forgot you aren't a mindreader...

Freenas 9.2.1.3

Processor is AMD 6100, 6 core, 16 G of memory, 6 WD 3TB drives (the NAS drives, but I cant't remember the color), and I do not have access today to the specific model of Motherboard today, but can get it tomorrow. All hardware has been the same, without this issue occuring for several months. This issue began about a month ago, after upgrading from 9.2.1 to the point releases, and my attempting to install a jail (via GUI) that I have since deleted.

 

[warri]

Can you see which process is consuming the CPU and memory? Check "top" in the shell or Running System Processes in the Web GUI when you notice increased CPU usage.

Can you also post a screenshot of your memory utilization?

I noticed that 9.2.1.4-BETA contains a fix for a memory leak which resulted in the GUI being killed (#4759) and 9.2.2-ALPHA one for 100% CPU usage when using zpool history / scrubbing a volume (#4736). Maybe it's one of those two problems?

 

[jerryjharrison]

Warri,

Attached is the memory report, but I think one of your bug reports is my issue. I perform a scrub of the ZFS Pool every 15 days, and it is scheduled to check the need for the scrub at 00:00 each night. That appears to be the precise time that the jump in CPU, (and memory) is occurring. I will add my notes to the bug report, and have un-enabled the scrub for now, in order to see if the jump in resources occurs tonight.

 

[cyberjock]

Rule 1: Never schedule stuff to start at an even hour.. 1am, 5:30am. Doing things like 5:17 makes it easier to determine what is going wrong. In your case, there's a whole lot of stuff that happens at midnight automatically, so now you have to rule out your scrub as a cause...

 

[jerryjharrison]

Ok, I seem to have the issue Warri pointed out in bug report: https://bugs.freenas.org/issues/4736.

When I disabled the check for the scrub of the Zpool, the nightly increase in the cpu and memory usage has stopped. I rebooted the server, and the usage has dropped to a steady state of less than 5%. I will monitor for another week, and report back here, but would appreciate someone letting me know which logs would be relevant to post to the existing bug report.

 

[jerryjharrison]

5 days without a single uptick in resource usage. I am convinced that this was my issue.

 

[warri]

Great! Time to upgrade to FreeNAS 9.2.1.4.1 (seriously long version number!) and to enable your scrubs again :)