High system load being reported

[sunny52266]

Hello everyone.

I have a situation where a high system load is suddenly being reported. The system has had an uptime of 43 days.

However, the drives do not show much activity.

The system processes tab shows the following:

How can I proceed with diagnosis?

 

[sunny52266]

top -a shows the following:

 

[sunny52266]

procstat -c 387 shows the following:

btw, these are just some commands I came accross while searching for the possible cause

 

[sunny52266]

Anyone? Newbie here. Kindly give me a starting point so I can find the issue.

 

[joeschmuck]

Welcome to the forums.

A few things: Please read the forum rules and post the information asked, such as what version of TrueNAS you are running, your system specs. Assume we know nothing and you will be starting at a good point because we don't know your setup. List any jails/VM's/Plug-In's you have setup, other processes.

To troubleshoot:
1) Disconnect the LAN cable, does the problem go away?
2) Do you have any jails/VM's/Plug-In's running? If yes, stop them one at a time and evaluate.
4) Reboot. Does the problem come back? If yes, did it come back quickly or take a while?

You see where I'm going with this stuff, you need to try to isolate the issue. Most of the time it's not easy or quick but it can be done. I'm thinking you setup something and then you noticed the usage afterward and didn't put the two events together.

Also, Do a google search for "kqread high truenas" and you may find some other folks having the same or similar issues.

Good Luck!

 

[sunny52266]

Welcome to the forums.

Thank you, sir!

It is a simple storage server with no jails or VM. I am running the latest truenas core.

Also, Do a google search for "kqread high truenas" and you may find some other folks having the same or similar issues.

I tried that. Did not come close. people had issues with other process ids and used the above commands to try to narrow down the issue.

However, a good old reboot does usually solve problems.

I'm thinking you setup something and then you noticed the usage afterward and didn't put the two events together

What made me post the issue was this point. I do not remember logging into the server for the last 10 days. and no one else has access.

I shall try your suggestions and report!

Thanks again!

 

[joeschmuck]

It can be something as simple as a failing boot drive. Post your system specs, make/model as the forum rules indicate. Otherwise we are just guessing at the cause. Often it can be something like an SMR hard drive or failing USB Flash media / DOM. If yopu are just testing on substandard hardware, then it could be just that. It appears you have enough RAM since your SWAP space is not being used (a good thing) and looks like you have three drives in your pool (just a guess).

 

[sunny52266]

Thank you for your help Joe.

a system update and reboot seem to have done the trick.

I have updated my signature with the build information. all the drives are new.

I did some research before making the server. However, could still include some sub-standard hardware. your advice will be appreciated if any.

Oh, and I am also using a cache drive 240GB SSD in the pool. forgot to include in the signature.

 

[joeschmuck]

a system update and reboot seem to have done the trick.

That is good news.

Oh, and I am also using a cache drive 240GB SSD in the pool. forgot to include in the signature.

That is generally not good news, it depends on what you are using the NAS for. Odds are you are slowing down your NAS. There are a lot of forum topics with respect to adding a cache. I will sum it up for you in a few words and you can go read up on it if you feel like it. In a nut shell, when you access a file the NAS will first look in the Cache (L2ARC) and see if the data requested is there. If the data is not there then it looks for it on the VDEV/Pool. If the file is not a file which is used frequently (like a video/movie server or just routine backups) then the file you are looking for will not be in the L2ARC. This means you are spending time searching 240GB for a file that isn't there. Additionally, RAM (ARC) is much faster than L2ARC so if you do access specific files frequently then more physical RAM is the best solution, once you have maxed out you RAM then you can resort to the L2ARC, if it makes sense.

Your SATA Port Expander "might" lead to issues communication with the hard drives so keep an eye on that, but if all is working then you may be just fine.

The rest of your system looks fine for a simple file server. My advice: Remove the 240GB Cache unless you are in fact accessing the same file repeatedly. The cache is just one more thing to fail that you can eliminate. But then again, I do like to say "If it ain't broke, don't fix it".

Lastly, ensure you are running SMART Short and Long tests on all your hard drives and have email reporting established. I recommend running the SMART Short test daily and the Long test once a week, all during times when the NAS is not expected to be active. If it is active, the tests will just take longer, in particular the Long test.

Good luck.

 

[sunny52266]

Additionally, RAM (ARC) is much faster than L2ARC so if you do access specific files frequently then more physical RAM is the best solution, once you have maxed out you RAM then you can resort to the L2ARC, if it makes sense

Oh. Okay, I get it now. Thanks for your explanation. I'll try and disconnect it for now. I watched an LTT video and they were using an L2ARC so I thought why shouldn't I. lol.

Your SATA Port Expander "might" lead to issues communication with the hard drives so keep an eye on that, but if all is working then you may be just fine.

I'll keep an eye out. Thanks!

Lastly, ensure you are running SMART Short and Long tests on all your hard drives and have email reporting established.

Yes, I have done that. tests are scheduled for Sundays.

Thanks again for taking the time to reply. Cheers.

 

[joeschmuck]

Yes, watch out for those video's, some are very misleading and many are old meaning the FreeNAS/TrueNAS software and requirements change. Some video's have some very good information but it needs to be tempered by the use case, depending on what the server is used for will dictate the configuration you will want to use. A simple backup sever or even video server does not require much.

Cheers