Jack828
Dabbler
- Joined
- Nov 11, 2020
- Messages
- 16
Hi folks.
I have a particularly worrying issue with my NAS at the moment.
It frequently locks itself up - I can't access the webui, jail services, etc. I am able to ping. I have no idea why this occurs - but when it does, I have to press the power button to get it to shutdown safely. (I do _not_ hard reboot)
System specifications:
- AMD Ryzen 5 3400G 3.7 GHz Quad-Core Processor
- MSI B450I GAMING PLUS AC Mini ITX AM4 Motherboard
- Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-2133 CL13 Memory
- Fractal Design Node 304 Mini ITX Tower Case
- be quiet! SFX L Power 500 W 80+ Gold Certified Fully Modular SFX Power Supply
- 6x Seagate IronWolf 4 TB
- LSI 9200-8i
However, today my home lost power, and as I went down to the NAS to switch it off safely (thanks to my partner for buying me a UPS!) it was totally unresponsive to pressing the button.
I'll list the symptoms to save getting wordy:
- Power button was ignored
- System & jails responded to pings
- System & jails did not allow SSH
- WebUI not responding
I tried:
- Plugging a monitor and keyboard in to get to the shell on the device itself - no display output
- Unplugging the ethernet and plugging it back in (was clutching at straws at this point)
- Pressing the power button a few times in succession whilst unplugging display/ethernet/keyboard
In the end, I was forced to hard shut down myself. Once back online I trawled through logs to see if there was anything interesting:
Important timestamps:
- 2021/07/01 12:07 (approx) - power lost
- 2021/07/01 12:09 (approx) - probably pressed reboot button first time here
- 2021/07/01 12:56 (approx) - hard shutdown & reboot
console.log
daemon.log
messages
middlewared.log
After seeing
Please, does anyone know what I can do to prevent this issue?
If you need more info, please let me know. I will be able to provide more info today and probably tomorrow - and then I am away for the weekend.
Thanks in advance!
I have a particularly worrying issue with my NAS at the moment.
It frequently locks itself up - I can't access the webui, jail services, etc. I am able to ping. I have no idea why this occurs - but when it does, I have to press the power button to get it to shutdown safely. (I do _not_ hard reboot)
System specifications:
- AMD Ryzen 5 3400G 3.7 GHz Quad-Core Processor
- MSI B450I GAMING PLUS AC Mini ITX AM4 Motherboard
- Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-2133 CL13 Memory
- Fractal Design Node 304 Mini ITX Tower Case
- be quiet! SFX L Power 500 W 80+ Gold Certified Fully Modular SFX Power Supply
- 6x Seagate IronWolf 4 TB
- LSI 9200-8i
However, today my home lost power, and as I went down to the NAS to switch it off safely (thanks to my partner for buying me a UPS!) it was totally unresponsive to pressing the button.
I'll list the symptoms to save getting wordy:
- Power button was ignored
- System & jails responded to pings
- System & jails did not allow SSH
- WebUI not responding
I tried:
- Plugging a monitor and keyboard in to get to the shell on the device itself - no display output
- Unplugging the ethernet and plugging it back in (was clutching at straws at this point)
- Pressing the power button a few times in succession whilst unplugging display/ethernet/keyboard
In the end, I was forced to hard shut down myself. Once back online I trawled through logs to see if there was anything interesting:
Important timestamps:
- 2021/07/01 12:07 (approx) - power lost
- 2021/07/01 12:09 (approx) - probably pressed reboot button first time here
- 2021/07/01 12:56 (approx) - hard shutdown & reboot
console.log
Code:
Jun 30 20:21:04 moneta Wed Jun 30 20:21:04 BST 2021 Jul 1 12:09:09 moneta net.inet.carp.allow: 1 -> 0 Jul 1 12:09:09 moneta Stopping cron. Jul 1 12:09:09 moneta Waiting for PIDS: 1798. Jul 1 12:09:09 moneta Stopping collectd_daemon. Jul 1 12:09:09 moneta Waiting for PIDS: 1759. Jul 1 12:09:09 moneta Waiting for collectd to terminate. Jul 1 12:09:09 moneta Waiting for PIDS: 1760. Jul 1 12:09:10 moneta Stopping sshd. Jul 1 12:09:10 moneta Waiting for PIDS: 1732. Jul 1 12:09:10 moneta Stopping nginx. Jul 1 12:09:10 moneta Waiting for PIDS: 1736. Jul 1 12:09:10 moneta Stopping rrdcached. Jul 1 12:09:10 moneta Waiting for PIDS: 1723. Jul 1 12:09:10 moneta Stopping rsyncd. Jul 1 12:09:10 moneta Waiting for PIDS: 1717. Jul 1 12:09:10 moneta Stopping smartd_daemon. Jul 1 12:09:10 moneta Waiting for PIDS: 1709. Jul 1 12:09:10 moneta Cannot 'stop' nmbd. Set nmbd_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'. Jul 1 12:09:10 moneta Stopping smbd. Jul 1 12:09:10 moneta Waiting for PIDS: 1686. Jul 1 12:09:10 moneta Stopping winbindd. Jul 1 12:09:11 moneta Waiting for PIDS: 1694. Jul 1 12:09:11 moneta Stopping ntpd. Jul 1 12:09:11 moneta Waiting for PIDS: 1649. Jul 1 12:09:11 moneta Shutting down local daemons:. Jul 1 12:09:11 moneta Writing entropy file: . Jul 1 12:09:11 moneta Writing early boot entropy file: . Jul 1 12:09:11 moneta Stopping middlewared. Jul 1 12:09:20 moneta Waiting for PIDS: 393 Jul 1 12:09:21 moneta Jul 1 12:59:42 moneta Starting devd. Jul 1 12:59:42 moneta Autoloading module: intpm.ko Jul 1 12:59:42 moneta Starting zfsd. Jul 1 12:59:42 moneta <118>middlewared: starting
daemon.log
Code:
Jul 1 12:09:10 moneta 1 2021-07-01T12:09:10.750829+01:00 moneta.local rsyncd 1717 - - sent 0 bytes received 0 bytes total size 0 Jul 1 12:09:10 moneta 1 2021-07-01T12:09:10.755512+01:00 moneta.local smartd 1709 - - smartd received signal 15: Terminated Jul 1 12:09:10 moneta 1 2021-07-01T12:09:10.755572+01:00 moneta.local smartd 1709 - - smartd is exiting (exit status 0) Jul 1 12:09:21 moneta 1 2021-07-01T11:09:21.708794+00:00 moneta.local zfsd 375 - - Disconnecting from devd. Jul 1 12:09:21 moneta 1 2021-07-01T12:09:21.717232+01:00 moneta.local avahi-daemon 1766 - - Got SIGTERM, quitting. Jul 1 12:09:21 moneta 1 2021-07-01T12:09:21.717661+01:00 moneta.local avahi-daemon 1766 - - Leaving mDNS multicast group on interface re0.IPv4 with address 192.168.1.10. Jul 1 12:09:21 moneta 1 2021-07-01T12:09:21.717877+01:00 moneta.local avahi-daemon 1766 - - avahi-daemon 0.7 exiting. Jul 1 12:59:47 moneta 1 2021-07-01T12:59:47.435171+01:00 moneta.local smartd 1717 - - smartd 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
messages
Code:
Jul 1 11:46:26 moneta kernel[1438]: Last message 'Limiting closed port' repeated 1 times, suppressed by syslog-ng on moneta.local Jul 1 12:09:11 moneta 1 2021-07-01T12:09:11.021041+01:00 moneta.local ntpd 1649 - - ntpd exiting on signal 15 (Terminated) Jul 1 12:09:20 moneta kernel: vnet0.1: link state changed to DOWN Jul 1 12:09:20 moneta kernel: epair0b: link state changed to DOWN Jul 1 12:09:21 moneta kernel: wg0: link state changed to DOWN Jul 1 12:09:21 moneta syslog-ng[1438]: syslog-ng shutting down; version='3.25.1' Jul 1 12:59:42 moneta syslog-ng[1446]: syslog-ng starting up; version='3.25.1' Jul 1 12:59:42 moneta ---<<BOOT>>--- Jul 1 12:59:42 moneta Copyright (c) 1992-2020 The FreeBSD Project.
middlewared.log
Code:
[2021/07/01 11:53:34] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem']): Read 1028 bytes total [2021/07/01 12:09:09] (DEBUG) middlewared.__event_system_ready():2359 - VM(s) stopped successfully [2021/07/01 12:09:09] (WARNING) middlewared.plugins.service_.services.base_freebsd.freebsd_service():134 - libvirtd forcestop failed with code 1: 'libvirtd not running? (check /var/run/libvirtd.pid).\n' [2021/07/01 12:09:09] (DEBUG) JailService.stop_on_shutdown():1607 - Stopping jails on shutdown: PENDING [2021/07/01 12:09:10] (INFO) iocage.callback():71 - * Stopping Netdata [2021/07/01 12:09:10] (INFO) iocage.callback():71 - + Executing prestop OK [2021/07/01 12:09:11] (INFO) middlewared.terminate():1494 - Terminating [2021/07/01 12:09:20] (INFO) iocage.callback():71 - + Stopping services OK [2021/07/01 12:09:20] (INFO) iocage.callback():71 - + Tearing down VNET OK [2021/07/01 12:09:20] (INFO) iocage.callback():71 - + Removing devfs_ruleset: 1000 OK [2021/07/01 12:09:20] (INFO) iocage.callback():71 - + Removing jail process OK [2021/07/01 12:09:20] (INFO) iocage.callback():71 - + Executing poststop OK [2021/07/01 12:09:21] (INFO) iocage.callback():71 - * Stopping Tautulli [2021/07/01 12:09:21] (INFO) iocage.callback():71 - + Executing prestop OK [2021/07/01 12:09:21] (ERROR) middlewared.__terminate():1518 - Failed to terminate jail Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1516, in __terminate await asyncio.wait_for(service.terminate(), timeout) File "/usr/local/lib/python3.8/asyncio/tasks.py", line 490, in wait_for raise exceptions.TimeoutError() asyncio.exceptions.TimeoutError [2021/07/01 11:57:32] (INFO) middlewared.__init__():792 - Starting TrueNAS-12.0-RELEASE middleware [2021/07/01 12:57:42] (DEBUG) middlewared.setup():1641 - Timezone set to Europe/London [2021/07/01 12:57:44] (DEBUG) middlewared.setup():2834 - Certificate setup for System complete [2021/07/01 12:57:44] (INFO) middlewared.devd_listen():58 - devd connection established [2021/07/01 12:57:45] (DEBUG) middlewared.__plugins_setup():903 - All plugins loaded
After seeing
* Stopping Tautulli
and the error lines after it, I uninstalled that plugin. But I need to make sure it wont be happening again.Please, does anyone know what I can do to prevent this issue?
If you need more info, please let me know. I will be able to provide more info today and probably tomorrow - and then I am away for the weekend.
Thanks in advance!