Help diagnosing shutdowns every few days

[ooimo]

I've been noticing my FreeNAS system shutting itself down once or twice a day over the last week and I'm struggling to find an explaination. I've looked through the logs after it happens and not found anything that seems obvious. I've pasted the log below in case anyone can decipher it.

The log seems to just show the system shutting down, although when I do a clean shutdown I don't get anything about "Lost x pages of memory".I̶'̶v̶e̶ ̶a̶l̶s̶o̶ ̶n̶o̶t̶i̶c̶e̶d̶ ̶t̶h̶a̶t̶ ̶w̶h̶e̶n̶ ̶I̶ ̶r̶e̶s̶t̶a̶r̶t̶ ̶t̶h̶e̶ ̶s̶e̶r̶v̶e̶r̶ ̶a̶f̶t̶e̶r̶w̶a̶r̶d̶s̶ ̶S̶S̶H̶ ̶i̶s̶n̶'̶t̶ ̶r̶u̶n̶n̶i̶n̶g̶ ̶a̶n̶d̶ ̶I̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶r̶e̶s̶t̶a̶r̶t̶ ̶i̶t̶ ̶t̶h̶r̶o̶u̶g̶h̶ ̶t̶h̶e̶ ̶w̶e̶b̶u̶i̶.̶ (fixed)

A̶l̶s̶o̶ ̶j̶a̶i̶l̶s̶ ̶w̶o̶n̶'̶t̶ ̶s̶t̶a̶r̶t̶ ̶a̶f̶t̶e̶r̶ ̶r̶e̶b̶o̶o̶t̶i̶n̶g̶,̶ ̶w̶h̶e̶n̶ ̶I̶ ̶t̶y̶p̶e̶ ̶"̶w̶a̶r̶d̶e̶n̶ ̶s̶t̶a̶r̶t̶ ̶p̶l̶e̶x̶_̶1̶"̶ ̶i̶t̶ ̶s̶a̶y̶s̶ ̶i̶t̶'̶s̶ ̶a̶l̶r̶e̶a̶d̶y̶ ̶r̶u̶n̶n̶i̶n̶g̶ ̶b̶u̶t̶ ̶t̶h̶e̶ ̶w̶e̶b̶u̶i̶ ̶s̶a̶y̶s̶ ̶i̶t̶'̶s̶ ̶o̶f̶f̶ ̶a̶n̶d̶ ̶I̶ ̶c̶a̶n̶'̶t̶ ̶c̶o̶n̶n̶e̶c̶t̶ ̶t̶o̶ ̶i̶t̶.̶ (fixed)

This system has been reliable for the 363 days of uptime it had prior to this.. Any advice would be greatly appreciated :)

The system: http://au.pcpartpicker.com/p/F4cT99

/var/log/messages:

Code:

Apr 29 19:11:54 freenas kernel: delete: cannot locate 192.168.1.99
Apr 29 19:11:54 freenas kernel: epair0a: link state changed to DOWN
Apr 29 19:11:54 freenas kernel: epair0b: link state changed to DOWN
Apr 29 19:11:55 freenas upsmon[2487]: UPS [ups]: connect failed: Connection failure: Can't assign requested address
Apr 29 19:12:05 freenas last message repeated 2 times
Apr 29 19:12:06 freenas kernel: .
Apr 29 19:12:06 freenas kernel: ifa_del_loopback_route: deletion failed
Apr 29 19:12:06 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:12:06 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items).  Lost 3 pages of memory.
Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcpreass) was not empty (252 items).  Lost 3 pages of memory.
Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcptw) was not empty (50 items).  Lost 1 pages of memory.
Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:12:06 freenas kernel: Freed UMA keg (tcpcb) was not empty (16 items).  Lost 4 pages of memory.
Apr 29 19:12:06 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items).  Lost 2 pages of memory.
Apr 29 19:12:06 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Apr 29 19:12:06 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Apr 29 19:12:07 freenas kernel: jexec 2 /usr/pbi/transmission-amd64/control stop
Apr 29 19:12:15 freenas kernel: epair1a: link state changed to DOWN
Apr 29 19:12:15 freenas kernel: epair1b: link state changed to DOWN
Apr 29 19:12:19 freenas kernel: tun0: link state changed to DOWN
Apr 29 19:12:22 freenas kernel: .
Apr 29 19:12:23 freenas kernel: Done
Apr 29 19:12:23 freenas kernel: jexec 3 /usr/pbi/btsync-amd64/control stop
Apr 29 19:12:32 freenas kernel: delete: cannot locate 192.168.1.6
Apr 29 19:12:32 freenas kernel: epair2a: link state changed to DOWN
Apr 29 19:12:32 freenas kernel: epair2b: link state changed to DOWN
Apr 29 19:12:45 freenas last message repeated 2 times
Apr 29 19:12:50 freenas kernel: .
Apr 29 19:12:50 freenas kernel: ifa_del_loopback_route: deletion failed
Apr 29 19:12:50 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (90 items).  Lost 9 pages of memory.
Apr 29 19:12:50 freenas kernel: Freed UMA keg (udpcb) was not empty (672 items).  Lost 4 pages of memory.
Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcpreass) was not empty (84 items).  Lost 1 pages of memory.
Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcptw) was not empty (300 items).  Lost 6 pages of memory.
Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (80 items).  Lost 8 pages of memory.
Apr 29 19:12:50 freenas kernel: Freed UMA keg (tcpcb) was not empty (32 items).  Lost 8 pages of memory.
Apr 29 19:12:50 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items).  Lost 2 pages of memory.
Apr 29 19:12:50 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Apr 29 19:12:50 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Apr 29 19:12:51 freenas kernel: Stoppingthejail....
Apr 29 19:12:51 freenas kernel: delete: cannot locate 192.168.1.22
Apr 29 19:12:51 freenas kernel: epair3a: link state changed to DOWN
Apr 29 19:12:51 freenas kernel: epair3b: link state changed to DOWN
Apr 29 19:12:56 freenas kernel: .
Apr 29 19:12:56 freenas kernel: ifa_del_loopback_route: deletion failed
Apr 29 19:12:56 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (50 items).  Lost 5 pages of memory.
Apr 29 19:12:56 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items).  Lost 3 pages of memory.
Apr 29 19:12:56 freenas kernel: Freed UMA keg (tcpreass) was not empty (252 items).  Lost 3 pages of memory.
Apr 29 19:12:56 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:12:56 freenas kernel: Freed UMA keg (tcpcb) was not empty (20 items).  Lost 5 pages of memory.
Apr 29 19:12:56 freenas kernel: Freed UMA keg (ripcb) was not empty (10 items).  Lost 1 pages of memory.
Apr 29 19:12:56 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Apr 29 19:12:56 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Apr 29 19:12:57 freenas kernel: Stoppingthejail....
Apr 29 19:12:57 freenas kernel: delete: cannot locate 192.168.1.23
Apr 29 19:12:57 freenas kernel: epair4a: link state changed to DOWN
Apr 29 19:12:57 freenas kernel: epair4b: link state changed to DOWN
Apr 29 19:13:01 freenas kernel: .
Apr 29 19:13:01 freenas kernel: ifa_del_loopback_route: deletion failed
Apr 29 19:13:01 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:13:01 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items).  Lost 3 pages of memory.
Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcpreass) was not empty (252 items).  Lost 3 pages of memory.
Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcptw) was not empty (50 items).  Lost 1 pages of memory.
Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:13:01 freenas kernel: Freed UMA keg (tcpcb) was not empty (20 items).  Lost 5 pages of memory.
Apr 29 19:13:01 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items).  Lost 2 pages of memory.
Apr 29 19:13:01 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Apr 29 19:13:01 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Apr 29 19:13:02 freenas kernel: Stoppingthejail....
Apr 29 19:13:02 freenas kernel: delete: cannot locate 192.168.1.98
Apr 29 19:13:02 freenas kernel: epair5a: link state changed to DOWN
Apr 29 19:13:02 freenas kernel: epair5b: link state changed to DOWN
Apr 29 19:13:07 freenas kernel: .
Apr 29 19:13:07 freenas kernel: ifa_del_loopback_route: deletion failed
Apr 29 19:13:07 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:13:07 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items).  Lost 3 pages of memory.
Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcpreass) was not empty (336 items).  Lost 4 pages of memory.
Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcptw) was not empty (100 items).  Lost 2 pages of memory.
Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:13:07 freenas kernel: Freed UMA keg (tcpcb) was not empty (20 items).  Lost 5 pages of memory.
Apr 29 19:13:07 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items).  Lost 2 pages of memory.
Apr 29 19:13:07 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Apr 29 19:13:07 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Apr 29 19:13:08 freenas kernel: jexec 7 /usr/pbi/minidlna-amd64/control stop
Apr 29 19:13:08 freenas kernel: delete: cannot locate 192.168.1.26
Apr 29 19:13:08 freenas kernel: epair6a: link state changed to DOWN
Apr 29 19:13:08 freenas kernel: epair6b: link state changed to DOWN
Apr 29 19:13:15 freenas kernel: .
Apr 29 19:13:15 freenas kernel: ifa_del_loopback_route: deletion failed
Apr 29 19:13:15 freenas kernel: Freed UMA keg (udp_inpcb) was not empty (40 items).  Lost 4 pages of memory.
Apr 29 19:13:15 freenas kernel: Freed UMA keg (udpcb) was not empty (504 items).  Lost 3 pages of memory.
Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcpreass) was not empty (336 items).  Lost 4 pages of memory.
Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcptw) was not empty (250 items).  Lost 5 pages of memory.
Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcp_inpcb) was not empty (70 items).  Lost 7 pages of memory.
Apr 29 19:13:15 freenas kernel: Freed UMA keg (tcpcb) was not empty (32 items).  Lost 8 pages of memory.
Apr 29 19:13:15 freenas kernel: Freed UMA keg (ripcb) was not empty (20 items).  Lost 2 pages of memory.
Apr 29 19:13:15 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=1 cleanup required
Apr 29 19:13:15 freenas kernel: hhook_vnet_uninit: hhook_head type=1, id=0 cleanup required
Apr 29 19:13:16 freenas kernel: jexec 8 /usr/pbi/mediabrowser-amd64/control stop
Apr 29 19:13:18 freenas rc.shutdown: 90 second watchdog timeout expired. Shutdown terminated.
Apr 29 19:13:18 freenas init: /bin/sh on /etc/rc.shutdown terminated abnormally, going to single user mode
Apr 29 19:13:18 freenas syslogd: exiting on signal 15

 

[SweetAndLow]

Try searching for the error message, Freed UMA keg.

 

[ooimo]

Hmm, searched for that and found this thread: https://forums.freenas.org/index.ph...tput-problem-with-lost-pages-of-memory.22549/
I got an intel NIC about 6 months ago so I don't know if that is causing me problems. Either way I completely disabled the on board one. I'll report back here when I figure out if it's sorted.

 

[Apollo]

I think the errors messages about "freed UMA keg..... was not empty... Lost X pages of memory" are fine.
As far as I can tell, it happens when your NIC gets disconnected. Either dropped by Freenas itself or when you unplug your NIC from the rest of the network.
If is also possible you are having a jail IP conflict.
It seems you may have a UPS. If so, it could be causing more harm than none, ie bad battery, input line drooping or browning due to too excess load.

 

[ooimo]

Thanks, I checked for jail IP conflicts, don't have any. The UPS isn't under load but I might disconnect the usb cable because it isn't very reliable.
I get a lot of this kind of stuff which I can't seem to fix:

Code:

Apr 29 20:40:31 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair9b
Apr 29 20:40:42 freenas kernel: epair10a: Ethernet address: 02:25:73:00:14:0a
Apr 29 20:40:42 freenas kernel: epair10b: Ethernet address: 02:25:73:00:15:0b
Apr 29 20:40:42 freenas kernel: epair10a: link state changed to UP
Apr 29 20:40:42 freenas kernel: epair10b: link state changed to UP
Apr 29 20:40:42 freenas kernel: epair10a: promiscuous mode enabled
Apr 29 20:40:43 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair10b
Apr 29 20:40:52 freenas kernel: epair11a: Ethernet address: 02:e1:e0:00:15:0a
Apr 29 20:40:52 freenas kernel: epair11b: Ethernet address: 02:e1:e0:00:16:0b
Apr 29 20:40:52 freenas kernel: epair11a: link state changed to UP
Apr 29 20:40:52 freenas kernel: epair11b: link state changed to UP
Apr 29 20:40:52 freenas kernel: epair11a: promiscuous mode enabled
Apr 29 20:40:52 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair11b
Apr 29 20:41:12 freenas kernel: epair12a: Ethernet address: 02:56:5e:00:16:0a
Apr 29 20:41:12 freenas kernel: epair12b: Ethernet address: 02:56:5e:00:17:0b
Apr 29 20:41:12 freenas kernel: epair12a: link state changed to UP
Apr 29 20:41:12 freenas kernel: epair12b: link state changed to UP
Apr 29 20:41:12 freenas kernel: epair12a: promiscuous mode enabled
Apr 29 20:41:12 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair12b
Apr 29 20:42:53 freenas kernel: epair13a: Ethernet address: 02:8a:fa:00:17:0a
Apr 29 20:42:53 freenas kernel: epair13b: Ethernet address: 02:8a:fa:00:18:0b
Apr 29 20:42:53 freenas kernel: epair13a: link state changed to UP
Apr 29 20:42:53 freenas kernel: epair13b: link state changed to UP
Apr 29 20:42:53 freenas kernel: epair13a: promiscuous mode enabled
Apr 29 20:42:53 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair13b
Apr 29 21:04:58 freenas kernel: pid 13751 (mono-sgen), uid 989: exited on signal 6
Apr 29 23:22:47 freenas manage.py: [freeadmin.navtree:526] Couldn't retrieve http://192.168.1.15/plugins/subsonic/27/_s/treemenu: timed out
Apr 29 23:22:47 freenas manage.py: [freeadmin.navtree:526] Couldn't retrieve http://192.168.1.15/plugins/plexmediaserver/29/_s/treemenu: timed out
Apr 29 23:22:47 freenas manage.py: [freeadmin.navtree:526] Couldn't retrieve http://192.168.1.15/plugins/sickrage/30/_s/treemenu: timed out

 

[Apollo]

It's all related to your NICs. Could it be a faulty router? It seems you have something like 13 NIC. Do you have LAGG or other network redundancy scheme on Freenas?
I think it all boils down to networking issues. Either software or hardware.

 

[ooimo]

Those NICs are the virtual ones used by my jails. Thanks for the help, I'll do some googling to figure out how to fix the NIC issues