SOLVED System Load increasing over time

[LeoSum]

Hi there,
on my FreeNAS system I see a continuously incresing system load over time.
After a reboot all is fine.
A the same time the CPU is mostly ~99% in idle but the load might reach high values around 8 or so. (See attachment)

I can also practically notice the system becoming less responsive with increasing uptime:

• when trying to connect to the webinterface or via ssh it will then take a few seconds before the interface is loaded or the password prompt appears (this works instantly after boot)
• An MPD instance which plays an internet radio stream without issues at first, will be interrupted for a few seconds every now and then when the system has been up for a longer time.

See below my System Details:

Build FreeNAS-9.10.2-U1 (86c7ef5)
Platform Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz

Memory 24454MB

How would I best go about finding the root of this evil?

 

[m0nkey_]

Not a lot to go on. Once you've SSH'd in, run top to see what processes are taking the most CPU.

 

[LeoSum]

That is the weird thing, at the times of high load, there will be nothing really consuming CPU time under top. The CPU is 99.x% idle.

I'm guessing it could be something I/O related but I just don't know how to locate what is going wrong.

 

[dlavigne]

It might be related to https://bugs.freenas.org/issues/20122.

 

[LeoSum]

Thanks for the hint!
So are you suggesting that I check the checkbox under System -> System Dataset -> Reporting Database ?
That is currently unchecked.

 

[dlavigne]

It would be interesting to see if doing so resolves the issue. Please let us know!

 

[DrKK]

The Load is spiking, but the CPU usage in the FreeNAS appliance shows 99% idle?

That can't be good.

 

[LeoSum]

It would be interesting to see if doing so resolves the issue. Please let us know!

I have enabled said checkbox and will pay closer attention. So far, after the load did not spike after the last reboot, but it wouln't be unusual to take a few days to reappear. Will report back on my findings.

That can't be good.

Thank you for confirming my worries :)

But does anybody have any hints on how to go ahead with a more detailed diagnosis?

 

[LeoSum]

Meanwhile I have removed two older drives from my system and that seems to have gotten rid of the higher CPU load.

However I still observe that after a few days of uptime, my system gets less responsive (e.g. webinterface or ssh password prompt only loading after several seconds of delay).

Could this be related to the system starting to use swap some time after boot?

I haven't yet established a clear correlation between the start of swap usage and the beginning of delayed responsiveness, but will keep an eye on the graph.

Does this make sense or is swapping normal and should not lead to the described behavior?

Actually I don't really understand why swapping happens at all, as there is still more free RAM available than used by swap (see attached image)

Does anyone have a clue?

 

[LeoSum]

I found this thread: https://forums.freenas.org/index.php?threads/swap-with-9-10.42749/page-2
and this script: https://forums.freenas.org/index.ph...ny-used-swap-to-prevent-kernel-crashes.46206/

It seems like some processes are swapped out during memory intense procedures like scrubbing and not swapped in afterwards.

I set up the cron job for the script now and will see how it goes.

 

[LeoSum]

So after a few days it now seems that the above-mentioned re-page-in script does the job reliably.
The system remains "snappy" even after longer uptime.