TrueNAS Core Stops Responding

[jaredrsowers]

I have a NAS that I have had for several years, and I just upgraded from FreeNAS, but now the TrueNAS services will stop responding, but all the jails and VMs continue to work. I cannot connect to the web UI, NFS, Samba, VNC for the VMs, etc, but all the jails are working fine. It will work fine for a couple of hours, up to a day, and then stop responding.

I have no idea where to begin to troubleshoot it, any suggestions would be appreciated.

 

[sretalla]

Do you have a jail running homebridge or hoobs? do you have the SNMP service running?

Both of those are known to end up with problems like what you are reporting.

Try stopping/disabling the jail or SNMP service and reboot (you may need to first reboot just to get to the settings).

 

[Newfoundland.Republic]

Hi @sretalla - Do you mean the SNMP client service? I'm having some slowness as well since upgrading from FN to TN and have the SNMP client running.

Thanks

 

[sretalla]

Hi @sretalla - Do you mean the SNMP client service? I'm having some slowness as well since upgrading from FN to TN and have the SNMP client running.

Thanks

The SNMP service (under Services)

If it's the culprit for you, you will see python38 with snmpd behind it in htop.

 

[Newfoundland.Republic]

..and I do see phython38 with high usage. I'll turn it off and see if that works.

I need to check Jira and see if there is any reports/bug fix in the works for this update

 

[Newfoundland.Republic]

Yes, looks like a bug: SNMP consuming 100% CPU and becomes unavailable

To be fixed in 12.0-U1

 

[sretalla]

To be fixed in 12.0-U1

Which was just released.... and is downloading at 1997 modem speeds

 

[c77dk]

Which was just released.... and is downloading at 1997 modem speeds

Could be fun seeing some stats for the downloads - I'm happy It's not my servers getting hammered

 

[Newfoundland.Republic]

Hmmm... Reported in TrueNAS 12.0-U1 Plans that: "This is not a production release, and should only be used for testing."

 

[sretalla]

Reported in TrueNAS 12.0-U1 Plans that: "This is not a production release, and should only be used for testing."

I checked with the dev team in the slack channel and they are agreeing it looks odd and are checking.

 

[HoneyBadger]

1997 modem speeds

That's about twenty times faster than you'd realistically be getting back in 1997.

Source: Was alive and using modems back then, am tickled pink by the idea that I now find my gigabit sync line slow at times. John Carmack's universal truth remains - "the speed of light sucks."

 

[sretalla]

"the speed of light sucks."

Well, right, but there would be consequences if it were faster... (not only more dangerous nuclear weapons, but lots of things at quantum and atomic level that would be much more dangerous than they are as it is now...)

Anyway, I saw 57K and was reminded of the dialup modem sound, later seeing that it was K Bytes, not K bits, so I was out by a best case of factor 8 anyway. I know that connecting to a server with that much bandwidth available for each client was no guarantee in those days either. What fun we still had anyway waiting for images to load line by line...

The modern day download failed finally on checksum and when I restarted it, completed at over 50MB/s so nothing to complain about there.

 

[jaredrsowers]

Do you have a jail running homebridge or hoobs? do you have the SNMP service running?

Both of those are known to end up with problems like what you are reporting.

Try stopping/disabling the jail or SNMP service and reboot (you may need to first reboot just to get to the settings).

No, I do not have homebridge, hoobs or the SNMP service enabled... and it does come back to life after some time... I will see if i can get any process info to see if there is something eating up all the cpu... but why would the jails continue to work as normal and be responsive if the cpu was maxed out?

 

[jaredrsowers]

I'll update to 12.0-U1 now and see if it helps any.

 

[olly86]

I've tried downloading the update, but each time it gets part way through and returns the error FAILED base-os, any ideas?

Code:

Error: Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 361, in run
    await self.future
  File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 399, in __run_body
    rv = await self.middleware.run_in_thread(self.method, *([self] + args))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
    return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/update.py", line 315, in download
    return self.middleware.call_sync('update.download_update', job, train, location, 100)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1249, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1289, in run_coroutine
    return fut.result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/update.py", line 319, in download_update
    success = await self.middleware.call('update.download_impl', *args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1233, in call
    return await self._call(
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1202, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1106, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/site-packages/middlewared/utils/io_thread_pool_executor.py", line 25, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/update_/download_freebsd.py", line 16, in download_impl
    Update.DownloadUpdate(
  File "/usr/local/lib/freenasOS/Update.py", line 1106, in DownloadUpdate
    pkg_file = conf.FindPackageFile(
  File "/usr/local/lib/freenasOS/Configuration.py", line 1274, in FindPackageFile
    raise Exceptions.UpdatePackageNotFound(package.Name())
freenasOS.Exceptions.UpdatePackageNotFound: base-os

 

[jaredrsowers]

It looks like I still have the isse after the update to 12.0-U1. The system just came back from being unresponsive for about 30 minutes, and nothing looks abnormal in the CPU usage during that time...