TrueNAS 12.0 Frequent Freeze & Unable to Safe Shutdown

Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Hi folks.

I have a particularly worrying issue with my NAS at the moment.

It frequently locks itself up - I can't access the webui, jail services, etc. I am able to ping. I have no idea why this occurs - but when it does, I have to press the power button to get it to shutdown safely. (I do _not_ hard reboot)

System specifications:
- AMD Ryzen 5 3400G 3.7 GHz Quad-Core Processor
- MSI B450I GAMING PLUS AC Mini ITX AM4 Motherboard
- Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-2133 CL13 Memory
- Fractal Design Node 304 Mini ITX Tower Case
- be quiet! SFX L Power 500 W 80+ Gold Certified Fully Modular SFX Power Supply
- 6x Seagate IronWolf 4 TB
- LSI 9200-8i


However, today my home lost power, and as I went down to the NAS to switch it off safely (thanks to my partner for buying me a UPS!) it was totally unresponsive to pressing the button.

I'll list the symptoms to save getting wordy:
- Power button was ignored
- System & jails responded to pings
- System & jails did not allow SSH
- WebUI not responding

I tried:
- Plugging a monitor and keyboard in to get to the shell on the device itself - no display output
- Unplugging the ethernet and plugging it back in (was clutching at straws at this point)
- Pressing the power button a few times in succession whilst unplugging display/ethernet/keyboard

In the end, I was forced to hard shut down myself. Once back online I trawled through logs to see if there was anything interesting:

Important timestamps:
- 2021/07/01 12:07 (approx) - power lost
- 2021/07/01 12:09 (approx) - probably pressed reboot button first time here
- 2021/07/01 12:56 (approx) - hard shutdown & reboot

console.log

Code:
Jun 30 20:21:04 moneta Wed Jun 30 20:21:04 BST 2021
Jul  1 12:09:09 moneta net.inet.carp.allow: 1 -> 0
Jul  1 12:09:09 moneta Stopping cron.
Jul  1 12:09:09 moneta Waiting for PIDS: 1798.
Jul  1 12:09:09 moneta Stopping collectd_daemon.
Jul  1 12:09:09 moneta Waiting for PIDS: 1759.
Jul  1 12:09:09 moneta Waiting for collectd to terminate.
Jul  1 12:09:09 moneta Waiting for PIDS: 1760.
Jul  1 12:09:10 moneta Stopping sshd.
Jul  1 12:09:10 moneta Waiting for PIDS: 1732.
Jul  1 12:09:10 moneta Stopping nginx.
Jul  1 12:09:10 moneta Waiting for PIDS: 1736.
Jul  1 12:09:10 moneta Stopping rrdcached.
Jul  1 12:09:10 moneta Waiting for PIDS: 1723.
Jul  1 12:09:10 moneta Stopping rsyncd.
Jul  1 12:09:10 moneta Waiting for PIDS: 1717.
Jul  1 12:09:10 moneta Stopping smartd_daemon.
Jul  1 12:09:10 moneta Waiting for PIDS: 1709.
Jul  1 12:09:10 moneta Cannot 'stop' nmbd. Set nmbd_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
Jul  1 12:09:10 moneta Stopping smbd.
Jul  1 12:09:10 moneta Waiting for PIDS: 1686.
Jul  1 12:09:10 moneta Stopping winbindd.
Jul  1 12:09:11 moneta Waiting for PIDS: 1694.
Jul  1 12:09:11 moneta Stopping ntpd.
Jul  1 12:09:11 moneta Waiting for PIDS: 1649.
Jul  1 12:09:11 moneta Shutting down local daemons:.
Jul  1 12:09:11 moneta Writing entropy file: .
Jul  1 12:09:11 moneta Writing early boot entropy file: .
Jul  1 12:09:11 moneta Stopping middlewared.
Jul  1 12:09:20 moneta Waiting for PIDS: 393
Jul  1 12:09:21 moneta 
Jul  1 12:59:42 moneta Starting devd.
Jul  1 12:59:42 moneta Autoloading module: intpm.ko
Jul  1 12:59:42 moneta Starting zfsd.
Jul  1 12:59:42 moneta <118>middlewared: starting


daemon.log

Code:
Jul  1 12:09:10 moneta 1 2021-07-01T12:09:10.750829+01:00 moneta.local rsyncd 1717 - - sent 0 bytes  received 0 bytes  total size 0
Jul  1 12:09:10 moneta 1 2021-07-01T12:09:10.755512+01:00 moneta.local smartd 1709 - - smartd received signal 15: Terminated
Jul  1 12:09:10 moneta 1 2021-07-01T12:09:10.755572+01:00 moneta.local smartd 1709 - - smartd is exiting (exit status 0)
Jul  1 12:09:21 moneta 1 2021-07-01T11:09:21.708794+00:00 moneta.local zfsd 375 - - Disconnecting from devd.
Jul  1 12:09:21 moneta 1 2021-07-01T12:09:21.717232+01:00 moneta.local avahi-daemon 1766 - - Got SIGTERM, quitting.
Jul  1 12:09:21 moneta 1 2021-07-01T12:09:21.717661+01:00 moneta.local avahi-daemon 1766 - - Leaving mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  1 12:09:21 moneta 1 2021-07-01T12:09:21.717877+01:00 moneta.local avahi-daemon 1766 - - avahi-daemon 0.7 exiting.
Jul  1 12:59:47 moneta 1 2021-07-01T12:59:47.435171+01:00 moneta.local smartd 1717 - - smartd 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)


messages

Code:
Jul  1 11:46:26 moneta kernel[1438]: Last message 'Limiting closed port' repeated 1 times, suppressed by syslog-ng on moneta.local
Jul  1 12:09:11 moneta 1 2021-07-01T12:09:11.021041+01:00 moneta.local ntpd 1649 - - ntpd exiting on signal 15 (Terminated)
Jul  1 12:09:20 moneta kernel: vnet0.1: link state changed to DOWN
Jul  1 12:09:20 moneta kernel: epair0b: link state changed to DOWN
Jul  1 12:09:21 moneta kernel: wg0: link state changed to DOWN
Jul  1 12:09:21 moneta syslog-ng[1438]: syslog-ng shutting down; version='3.25.1'
Jul  1 12:59:42 moneta syslog-ng[1446]: syslog-ng starting up; version='3.25.1'
Jul  1 12:59:42 moneta ---<<BOOT>>---
Jul  1 12:59:42 moneta Copyright (c) 1992-2020 The FreeBSD Project.


middlewared.log

Code:
[2021/07/01 11:53:34] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem']):  Read 1028 bytes total
[2021/07/01 12:09:09] (DEBUG) middlewared.__event_system_ready():2359 - VM(s) stopped successfully
[2021/07/01 12:09:09] (WARNING) middlewared.plugins.service_.services.base_freebsd.freebsd_service():134 - libvirtd forcestop failed with code 1: 'libvirtd not running? (check /var/run/libvirtd.pid).\n'
[2021/07/01 12:09:09] (DEBUG) JailService.stop_on_shutdown():1607 - Stopping jails on shutdown: PENDING
[2021/07/01 12:09:10] (INFO) iocage.callback():71 - * Stopping Netdata
[2021/07/01 12:09:10] (INFO) iocage.callback():71 -   + Executing prestop OK
[2021/07/01 12:09:11] (INFO) middlewared.terminate():1494 - Terminating
[2021/07/01 12:09:20] (INFO) iocage.callback():71 -   + Stopping services OK
[2021/07/01 12:09:20] (INFO) iocage.callback():71 -   + Tearing down VNET OK
[2021/07/01 12:09:20] (INFO) iocage.callback():71 -   + Removing devfs_ruleset: 1000 OK
[2021/07/01 12:09:20] (INFO) iocage.callback():71 -   + Removing jail process OK
[2021/07/01 12:09:20] (INFO) iocage.callback():71 -   + Executing poststop OK
[2021/07/01 12:09:21] (INFO) iocage.callback():71 - * Stopping Tautulli
[2021/07/01 12:09:21] (INFO) iocage.callback():71 -   + Executing prestop OK
[2021/07/01 12:09:21] (ERROR) middlewared.__terminate():1518 - Failed to terminate jail
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1516, in __terminate
    await asyncio.wait_for(service.terminate(), timeout)
  File "/usr/local/lib/python3.8/asyncio/tasks.py", line 490, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
[2021/07/01 11:57:32] (INFO) middlewared.__init__():792 - Starting TrueNAS-12.0-RELEASE middleware
[2021/07/01 12:57:42] (DEBUG) middlewared.setup():1641 - Timezone set to Europe/London
[2021/07/01 12:57:44] (DEBUG) middlewared.setup():2834 - Certificate setup for System complete
[2021/07/01 12:57:44] (INFO) middlewared.devd_listen():58 - devd connection established
[2021/07/01 12:57:45] (DEBUG) middlewared.__plugins_setup():903 - All plugins loaded


After seeing * Stopping Tautulli and the error lines after it, I uninstalled that plugin. But I need to make sure it wont be happening again.

Please, does anyone know what I can do to prevent this issue?

If you need more info, please let me know. I will be able to provide more info today and probably tomorrow - and then I am away for the weekend.

Thanks in advance!
 
A

Alecmascot

Guru
Joined
Mar 18, 2014
Messages
1,177
Have you done the Bios settings for Ryzen cpus as detailed in other threads ?
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Alecmascot said:
Have you done the Bios settings for Ryzen cpus as detailed in other threads ?
Click to expand...

Do you happen to have a link to these threads? Forum searches + duckduckgo gives me the following changes:

- Update BIOS and Play with Advanced > AMD CBS > Zen Common Options > Power Supply Idle Control. -- Should be three options there. You need to select the option that disables (some)C6 power saving.
From https://www.truenas.com/community/threads/ryzen-stability-on-11-0-u4.59017/post-460023
  • Advanced\CPU Configuration
    • Cool'n'Quiet -> Disabled
  • Advanced\AMD CBS\Zen Common Options
    • Power Supply Idle Control -> Typical Current Idle
From: https://www.truenas.com/community/t...-ryzen-stability-on-11-2-u5.78263/post-544741

- I disabled C-States, AMD Cool & Quiet and ErP Ready
- C-States and Cool & Quiet was in the CPU Features
- ErP Ready was under Advanced and Power Management Setup
From: https://www.truenas.com/community/threads/stability-issues-new-system.88771/post-615431

I will double check what my system uses after work today - but thank you for pointing me in that direction.
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
So I finally managed to find the options in the BIOS I've done:
- ErP - already disabled
- Cool'n'Quiet - disabled
- Power Supply Idle Control - set to "Typical Current Idle"

Pics attached to help someone in future, hopefully.
Before:
before.jpg

After:
after.jpg
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Update:

That didn't help - system locked up at about 8AM today :(

EDIT: i think this is networking related now?!?

daemon.log
Code:
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700612+01:00 moneta.local avahi-daemon 1796 - - Withdrawing address record for 192.168.1.10 on re0.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700655+01:00 moneta.local avahi-daemon 1796 - - Leaving mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700679+01:00 moneta.local avahi-daemon 1796 - - Interface re0.IPv4 no longer relevant for mDNS.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700692+01:00 moneta.local avahi-daemon 1796 - - Joining mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700706+01:00 moneta.local avahi-daemon 1796 - - New relevant interface re0.IPv4 for mDNS.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700729+01:00 moneta.local avahi-daemon 1796 - - Registering new address record for 192.168.1.10 on re0.IPv4.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.770968+01:00 moneta.local avahi-daemon 1796 - - Withdrawing address record for 192.168.1.10 on re0.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.771005+01:00 moneta.local avahi-daemon 1796 - - Leaving mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.771032+01:00 moneta.local avahi-daemon 1796 - - Interface re0.IPv4 no longer relevant for mDNS.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.783940+01:00 moneta.local dhclient 1261 - - No working leases in persistent database - sleeping.
Jul  7 08:14:39 moneta 1 2021-07-07T08:14:39.332581+01:00 moneta.local dhclient 1261 - - re0 link state up -> down
Jul  7 08:14:43 moneta 1 2021-07-07T08:14:43.390965+01:00 moneta.local dhclient 1261 - - re0 link state down -> up
Jul  7 08:14:43 moneta 1 2021-07-07T08:14:43.391032+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 3
Jul  7 08:14:47 moneta 1 2021-07-07T08:14:47.003564+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 5
Jul  7 08:14:52 moneta 1 2021-07-07T08:14:52.237200+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 11
Jul  7 08:15:03 moneta 1 2021-07-07T08:15:03.240974+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 13
Jul  7 08:15:16 moneta 1 2021-07-07T08:15:16.180195+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 13
Jul  7 08:15:29 moneta 1 2021-07-07T08:15:29.732068+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 14
Jul  7 08:15:37 moneta 1 2021-07-07T08:15:37.128303+01:00 moneta.local dhclient 1261 - - re0 link state up -> down
Jul  7 08:15:41 moneta 1 2021-07-07T08:15:41.170285+01:00 moneta.local dhclient 1261 - - re0 link state down -> up
Jul  7 08:15:41 moneta 1 2021-07-07T08:15:41.170347+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 8
Jul  7 08:15:49 moneta 1 2021-07-07T08:15:49.761391+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 8
Jul  7 08:15:57 moneta 1 2021-07-07T08:15:57.837466+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 15
Jul  7 08:16:12 moneta 1 2021-07-07T08:16:12.767639+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 11
Jul  7 08:16:23 moneta 1 2021-07-07T08:16:23.635050+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 7
Jul  7 08:16:29 moneta 1 2021-07-07T08:16:29.664145+01:00 moneta.local dhclient 1261 - - re0 link state up -> down
Jul  7 08:16:30 moneta 1 2021-07-07T08:16:30.666141+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 12
Jul  7 08:16:33 moneta 1 2021-07-07T08:16:33.676201+01:00 moneta.local dhclient 1261 - - re0 link state down -> up
Jul  7 08:16:33 moneta 1 2021-07-07T08:16:33.676262+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 3
Jul  7 08:16:37 moneta 1 2021-07-07T08:16:37.072388+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 6
Jul  7 08:16:43 moneta 1 2021-07-07T08:16:43.107291+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 8
Jul  7 08:16:51 moneta 1 2021-07-07T08:16:51.829406+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 17
Jul  7 08:17:08 moneta 1 2021-07-07T08:17:08.841006+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 12
Jul  7 08:17:20 moneta 1 2021-07-07T08:17:20.408574+01:00 moneta.local dhclient 1261 - - re0 link state up -> down


console.log
Code:
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941705+01:00 moneta.local collectd 1790 - - write_graphite plugin: send to 192.168.1.7:2003 (tcp) failed with status 32 (Broken pipe)
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941789+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:17:46 moneta 1 2021-07-07T08:17:46.583826+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:18:08 moneta 1 2021-07-07T08:18:08.959077+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:18:53 moneta 1 2021-07-07T08:18:53.593435+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:20:13 moneta 1 2021-07-07T08:20:13.607865+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:22:56 moneta 1 2021-07-07T08:22:56.517694+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:28:16 moneta 1 2021-07-07T08:28:16.525698+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable


messages
Code:
Jul  7 08:14:22 moneta ipfw[1468]: Last message 'pullup failed' repeated 23 times, suppressed by syslog-ng on moneta.local
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.701363+01:00 moneta.local dhclient 20084 - - New IP Address (re0): 192.168.1.10
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.701995+01:00 moneta.local dhclient 20085 - - New Subnet Mask (re0): 255.255.255.0
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.702689+01:00 moneta.local dhclient 20086 - - New Broadcast Address (re0): 192.168.1.255
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.703280+01:00 moneta.local dhclient 20087 - - New Routers (re0): 192.168.1.1
Jul  7 08:14:35 moneta 1 2021-07-07T08:14:35.757130+01:00 moneta.local dhclient 20089 - - New Routers (re0): 192.168.1.1
Jul  7 08:14:39 moneta kernel: re0: watchdog timeout
Jul  7 08:14:39 moneta kernel: re0: link state changed to DOWN
Jul  7 08:14:43 moneta kernel: re0: link state changed to UP
Jul  7 08:14:54 moneta ipfw: pullup failed
Jul  7 08:15:37 moneta ipfw[1468]: Last message 'pullup failed' repeated 5 times, suppressed by syslog-ng on moneta.local
Jul  7 08:15:37 moneta kernel: re0: watchdog timeout
Jul  7 08:15:37 moneta kernel: re0: link state changed to DOWN
Jul  7 08:15:41 moneta kernel: re0: link state changed to UP
Jul  7 08:16:14 moneta ipfw: pullup failed
Jul  7 08:16:29 moneta ipfw[1468]: Last message 'pullup failed' repeated 2 times, suppressed by syslog-ng on moneta.local
Jul  7 08:16:29 moneta kernel: re0: watchdog timeout
Jul  7 08:16:29 moneta kernel: re0: link state changed to DOWN
Jul  7 08:16:33 moneta kernel: re0: link state changed to UP
Jul  7 08:16:51 moneta ipfw: pullup failed
Jul  7 08:17:20 moneta ipfw[1468]: Last message 'pullup failed' repeated 17 times, suppressed by syslog-ng on moneta.local
Jul  7 08:17:20 moneta kernel: re0: watchdog timeout
Jul  7 08:17:20 moneta kernel: re0: link state changed to DOWN
Jul  7 08:17:24 moneta kernel: re0: link state changed to UP
Jul  7 08:17:25 moneta ipfw: pullup failed
Jul  7 08:17:33 moneta ipfw[1468]: Last message 'pullup failed' repeated 2 times, suppressed by syslog-ng on moneta.local
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941705+01:00 moneta.local collectd 1790 - - write_graphite plugin: send to 192.168.1.7:2003 (tcp) failed with status 32 (Broken pipe)
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941789+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:17:46 moneta 1 2021-07-07T08:17:46.583826+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:17:53 moneta ipfw: pullup failed
 
Last edited:
Spearfoot

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,478
Jack828 said:
Update:

That didn't help - system locked up at about 8AM today :(

EDIT: i think this is networking related now?!?

daemon.log
Code:
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700612+01:00 moneta.local avahi-daemon 1796 - - Withdrawing address record for 192.168.1.10 on re0.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700655+01:00 moneta.local avahi-daemon 1796 - - Leaving mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700679+01:00 moneta.local avahi-daemon 1796 - - Interface re0.IPv4 no longer relevant for mDNS.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700692+01:00 moneta.local avahi-daemon 1796 - - Joining mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700706+01:00 moneta.local avahi-daemon 1796 - - New relevant interface re0.IPv4 for mDNS.
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.700729+01:00 moneta.local avahi-daemon 1796 - - Registering new address record for 192.168.1.10 on re0.IPv4.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.770968+01:00 moneta.local avahi-daemon 1796 - - Withdrawing address record for 192.168.1.10 on re0.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.771005+01:00 moneta.local avahi-daemon 1796 - - Leaving mDNS multicast group on interface re0.IPv4 with address 192.168.1.10.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.771032+01:00 moneta.local avahi-daemon 1796 - - Interface re0.IPv4 no longer relevant for mDNS.
Jul  7 08:14:36 moneta 1 2021-07-07T08:14:36.783940+01:00 moneta.local dhclient 1261 - - No working leases in persistent database - sleeping.
Jul  7 08:14:39 moneta 1 2021-07-07T08:14:39.332581+01:00 moneta.local dhclient 1261 - - re0 link state up -> down
Jul  7 08:14:43 moneta 1 2021-07-07T08:14:43.390965+01:00 moneta.local dhclient 1261 - - re0 link state down -> up
Jul  7 08:14:43 moneta 1 2021-07-07T08:14:43.391032+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 3
Jul  7 08:14:47 moneta 1 2021-07-07T08:14:47.003564+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 5
Jul  7 08:14:52 moneta 1 2021-07-07T08:14:52.237200+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 11
Jul  7 08:15:03 moneta 1 2021-07-07T08:15:03.240974+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 13
Jul  7 08:15:16 moneta 1 2021-07-07T08:15:16.180195+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 13
Jul  7 08:15:29 moneta 1 2021-07-07T08:15:29.732068+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 14
Jul  7 08:15:37 moneta 1 2021-07-07T08:15:37.128303+01:00 moneta.local dhclient 1261 - - re0 link state up -> down
Jul  7 08:15:41 moneta 1 2021-07-07T08:15:41.170285+01:00 moneta.local dhclient 1261 - - re0 link state down -> up
Jul  7 08:15:41 moneta 1 2021-07-07T08:15:41.170347+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 8
Jul  7 08:15:49 moneta 1 2021-07-07T08:15:49.761391+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 8
Jul  7 08:15:57 moneta 1 2021-07-07T08:15:57.837466+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 15
Jul  7 08:16:12 moneta 1 2021-07-07T08:16:12.767639+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 11
Jul  7 08:16:23 moneta 1 2021-07-07T08:16:23.635050+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 7
Jul  7 08:16:29 moneta 1 2021-07-07T08:16:29.664145+01:00 moneta.local dhclient 1261 - - re0 link state up -> down
Jul  7 08:16:30 moneta 1 2021-07-07T08:16:30.666141+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 12
Jul  7 08:16:33 moneta 1 2021-07-07T08:16:33.676201+01:00 moneta.local dhclient 1261 - - re0 link state down -> up
Jul  7 08:16:33 moneta 1 2021-07-07T08:16:33.676262+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 3
Jul  7 08:16:37 moneta 1 2021-07-07T08:16:37.072388+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 6
Jul  7 08:16:43 moneta 1 2021-07-07T08:16:43.107291+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 8
Jul  7 08:16:51 moneta 1 2021-07-07T08:16:51.829406+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 17
Jul  7 08:17:08 moneta 1 2021-07-07T08:17:08.841006+01:00 moneta.local dhclient 1261 - - DHCPDISCOVER on re0 to 255.255.255.255 port 67 interval 12
Jul  7 08:17:20 moneta 1 2021-07-07T08:17:20.408574+01:00 moneta.local dhclient 1261 - - re0 link state up -> down


console.log
Code:
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941705+01:00 moneta.local collectd 1790 - - write_graphite plugin: send to 192.168.1.7:2003 (tcp) failed with status 32 (Broken pipe)
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941789+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:17:46 moneta 1 2021-07-07T08:17:46.583826+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:18:08 moneta 1 2021-07-07T08:18:08.959077+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:18:53 moneta 1 2021-07-07T08:18:53.593435+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:20:13 moneta 1 2021-07-07T08:20:13.607865+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:22:56 moneta 1 2021-07-07T08:22:56.517694+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:28:16 moneta 1 2021-07-07T08:28:16.525698+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable


messages
Code:
Jul  7 08:14:22 moneta ipfw[1468]: Last message 'pullup failed' repeated 23 times, suppressed by syslog-ng on moneta.local
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.701363+01:00 moneta.local dhclient 20084 - - New IP Address (re0): 192.168.1.10
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.701995+01:00 moneta.local dhclient 20085 - - New Subnet Mask (re0): 255.255.255.0
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.702689+01:00 moneta.local dhclient 20086 - - New Broadcast Address (re0): 192.168.1.255
Jul  7 08:14:34 moneta 1 2021-07-07T08:14:34.703280+01:00 moneta.local dhclient 20087 - - New Routers (re0): 192.168.1.1
Jul  7 08:14:35 moneta 1 2021-07-07T08:14:35.757130+01:00 moneta.local dhclient 20089 - - New Routers (re0): 192.168.1.1
Jul  7 08:14:39 moneta kernel: re0: watchdog timeout
Jul  7 08:14:39 moneta kernel: re0: link state changed to DOWN
Jul  7 08:14:43 moneta kernel: re0: link state changed to UP
Jul  7 08:14:54 moneta ipfw: pullup failed
Jul  7 08:15:37 moneta ipfw[1468]: Last message 'pullup failed' repeated 5 times, suppressed by syslog-ng on moneta.local
Jul  7 08:15:37 moneta kernel: re0: watchdog timeout
Jul  7 08:15:37 moneta kernel: re0: link state changed to DOWN
Jul  7 08:15:41 moneta kernel: re0: link state changed to UP
Jul  7 08:16:14 moneta ipfw: pullup failed
Jul  7 08:16:29 moneta ipfw[1468]: Last message 'pullup failed' repeated 2 times, suppressed by syslog-ng on moneta.local
Jul  7 08:16:29 moneta kernel: re0: watchdog timeout
Jul  7 08:16:29 moneta kernel: re0: link state changed to DOWN
Jul  7 08:16:33 moneta kernel: re0: link state changed to UP
Jul  7 08:16:51 moneta ipfw: pullup failed
Jul  7 08:17:20 moneta ipfw[1468]: Last message 'pullup failed' repeated 17 times, suppressed by syslog-ng on moneta.local
Jul  7 08:17:20 moneta kernel: re0: watchdog timeout
Jul  7 08:17:20 moneta kernel: re0: link state changed to DOWN
Jul  7 08:17:24 moneta kernel: re0: link state changed to UP
Jul  7 08:17:25 moneta ipfw: pullup failed
Jul  7 08:17:33 moneta ipfw[1468]: Last message 'pullup failed' repeated 2 times, suppressed by syslog-ng on moneta.local
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941705+01:00 moneta.local collectd 1790 - - write_graphite plugin: send to 192.168.1.7:2003 (tcp) failed with status 32 (Broken pipe)
Jul  7 08:17:33 moneta 1 2021-07-07T08:17:33.941789+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:17:46 moneta 1 2021-07-07T08:17:46.583826+01:00 moneta.local collectd 1790 - - write_graphite plugin: Connecting to 192.168.1.7:2003 via tcp failed. The last error was: failed to connect to remote host: Network is unreachable
Jul  7 08:17:53 moneta ipfw: pullup failed
Click to expand...
Looks like your re0 network interface is losing its DHCP-assigned IP address, then never receiving a new IP assignment from your DHCP server. Try configuring a static IP address on your TrueNAS system.
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Spearfoot said:
Looks like your re0 network interface is losing its DHCP-assigned IP address, then never receiving a new IP assignment from your DHCP server. Try configuring a static IP address on your TrueNAS system.
Click to expand...

I thought I had checked this - in my router it is reserved, now I have setup the static IP for the NAS in the webui.

Maybe I have two issues at once!
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Spearfoot said:
Looks like your re0 network interface is losing its DHCP-assigned IP address, then never receiving a new IP assignment from your DHCP server. Try configuring a static IP address on your TrueNAS system.
Click to expand...

Hey so I set this and later on in the day I couldn't access any of my jails. I restarted to see if that would fix it and it didn't - I had to undo the static config as shown below and reboot.

1625739176785.png
 
tangles

tangles

Dabbler
Joined
Jan 12, 2018
Messages
33
What are the IPs listed in your Plugins/Jail sections?
There is a Netwait feature in Network —> Global config. This may help by ensuring your system‘s tcp stack is fully initialised before services kick in.
if you stop then start your plugins/jails after booting up, do they come back to life?
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
tangles said:
What are the IPs listed in your Plugins/Jail sections?
Click to expand...

Sorry for the delay mate. Here's what you asked for.

1626274003094.png


tangles said:
There is a Netwait feature in Network —> Global config. This may help by ensuring your system‘s tcp stack is fully initialised before services kick in.
if you stop then start your plugins/jails after booting up, do they come back to life?
Click to expand...
I will give this a try soon!
 
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Update....still frequently freezes. Has happened a couple times since I returned home after an extended period offline.
 
Redcoat

Redcoat

MVP
Joined
Feb 18, 2014
Messages
2,925
Last edited:
Jack828

Jack828

Dabbler
Joined
Nov 11, 2020
Messages
16
Cool, that sounds like a reasonable avenue for investigation.

I'd like to be able to confirm if it definitely is that before spending money on a new one (bit tight at the moment!)

Any ideas for confirming it is that?
 
You must log in or register to reply here.

Similar threads

B
truenas keeps powering off
Replies
8
Views
3K
beagleboys
B
Y
  • Locked
impossible to wipe disk
Replies
9
Views
22K
yaume
Y
Hugo Ochoa
  • Locked
[SOLVED] Time Machine doesn't find AFP share. Help please!!!
Replies
4
Views
5K
Hugo Ochoa
Hugo Ochoa
thijsjek
Iocage jails not auto starting on boot
Replies
11
Views
2K
sash
S
I
TrueNAS 12.0-U4 -->> BackBlaze ... unable to access/pair/sync
Replies
0
Views
881
IanMcLachlan
I
Top