Unauthorized Reboot - 11.1-RELEASE

Scharbag · Dec 30, 2017

My system has been very stable for years. Specs are in my signature.

Last night, I got this email:

Code:

System booted at Sat Dec 30 09:02:40 2017 was not shut down properly

Email was received at 3:12AM local time. Not sure why there is a GMT shift here but that is minor.

I looked at \var\log\messages and at 3:03:52 things went south:

Code:

Dec 30 03:03:52 Zhang syslog-ng[1729]: syslog-ng starting up; version='3.7.3'
Dec 30 03:03:52 Zhang Fatal double fault
Dec 30 03:03:52 Zhang rip 0xffffffff803c4ce8 rsp 0xfffffe28f7474000 rbp 0xfffffe28f7474010
Dec 30 03:03:52 Zhang rax 0xfffffe28f74740a0 rdx 0 rbx 0
Dec 30 03:03:52 Zhang rcx 0x8 rsi 0xfffff8018af6cc00 rdi 0x3
Dec 30 03:03:52 Zhang r8 0 r9 0xffffffff8118999a r10 0xffffffff8118999a
Dec 30 03:03:52 Zhang r11 0xfffff81ab8acc100 r12 0 r13 0x8
Dec 30 03:03:52 Zhang r14 0 r15 0 rflags 0x10282
Dec 30 03:03:52 Zhang cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
Dec 30 03:03:52 Zhang fsbase 0x8006e7130 gsbase 0xffffffff821b9f80 kgsbase 0
Dec 30 03:03:52 Zhang cpuid = 1; apic id = 01
Dec 30 03:03:52 Zhang panic: double fault

The only change I made yesterday is to have the Reporting Database stored on the system dataset:

I have attached the entire log (I deleted the SAS/SATA and networking details).

If anyone has any ideas or suggestions, I would be greatly appreciative!!

Thank you.

Ericloewe · Dec 30, 2017

03:03... That seems awfully close to the daily report generation time.

stoffix · Dec 30, 2017

Weird!
I too have had a stable system, and got the same email some hours ago:

Code:

System booted at Sat Dec 30 22:37:28 2017 was not shut down properly

Although my syslog doesn't show anything out of the ordinary until the machine started up again.

Apart from you, I haven't done any changes in the config the last couple of days.
Another new thing was two extra emails sent from the machine when it booted, both regarding a couple of datasets:

Code:

Refquota exceed on dataset volume1/kristoffer.
Used 82.04% (61.67 GB of 75.16 GB)

This dataset i just saved some files to, the other one is just an archive, and shouldn't have seen any changes for at least a year. Anyways, I upped the quota on both the datasets, so they're now below 80% used.

danb35 · Dec 30, 2017

Did your system(s) actually reboot? Because I've gotten false positives with that email. uptime would tell you.

Ericloewe · Dec 30, 2017

To be honest, I'm not sure what's going on here. The log immediately jumps from an oddly verbose panic to a (truncated?) boot sequence, in the same second.

Scharbag · Dec 31, 2017

Well, it did not panic last night. So fingers crossed it was a one time glitch. If it happens again, I will post additional info.

stoffix · Dec 31, 2017

danb35 said:
Did your system(s) actually reboot? Because I've gotten false positives with that email. uptime would tell you.

according to uptime, my system has now been up about 1 day and 5 hours, which would indicate it actually rebooted.

Scharbag · Jan 1, 2018

It happened again at about the same time:

Code:

System booted at Mon Jan  1 09:02:40 2018 was not shut down properly

Similar log messages:

Code:

Jan  1 00:00:00 Zhang newsyslog[44807]: logfile turned over
Jan  1 00:00:00 Zhang syslog-ng[1729]: Configuration reload request received, reloading configuration;
Jan  1 00:30:00 Zhang afpd[55018]: Login by timemachine (AFP3.4)
Jan  1 00:30:01 Zhang afpd[55018]: afp_zzz: entering extended sleep
Jan  1 00:33:14 Zhang afpd[56298]: Login by timemachine (AFP3.4)
Jan  1 00:34:12 Zhang afpd[55018]: AFP logout by timemachine
Jan  1 00:34:12 Zhang afpd[55018]: AFP statistics: 203638.88 KB read, 191069.51 KB written
Jan  1 00:34:12 Zhang afpd[55018]: done
Jan  1 00:36:04 Zhang afpd[56298]: AFP logout by timemachine
Jan  1 00:36:04 Zhang afpd[56298]: AFP statistics: 269678.72 KB read, 326854.85 KB written
Jan  1 00:36:04 Zhang afpd[56298]: done
Jan  1 01:33:05 Zhang afpd[76591]: Login by timemachine (AFP3.4)
Jan  1 01:35:02 Zhang afpd[76591]: AFP logout by timemachine
Jan  1 01:35:02 Zhang afpd[76591]: AFP statistics: 174391.75 KB read, 273984.55 KB written
Jan  1 01:35:02 Zhang afpd[76591]: done
Jan  1 01:40:19 Zhang afpd[78959]: Login by timemachine (AFP3.4)
Jan  1 01:40:20 Zhang afpd[78959]: afp_zzz: entering extended sleep
Jan  1 01:42:40 Zhang afpd[78959]: AFP logout by timemachine
Jan  1 01:42:40 Zhang afpd[78959]: AFP statistics: 97815.74 KB read, 109172.28 KB written
Jan  1 01:42:40 Zhang afpd[78959]: done
Jan  1 02:19:56 Zhang afpd[92238]: Login by timemachine (AFP3.4)
Jan  1 02:19:57 Zhang afpd[92238]: afp_zzz: entering extended sleep
Jan  1 02:21:38 Zhang afpd[92238]: AFP logout by timemachine
Jan  1 02:21:38 Zhang afpd[92238]: AFP statistics: 74004.65 KB read, 89918.13 KB written
Jan  1 02:21:38 Zhang afpd[92238]: done
Jan  1 02:38:28 Zhang afpd[98580]: Login by timemachine (AFP3.4)
Jan  1 02:40:29 Zhang afpd[98580]: AFP logout by timemachine
Jan  1 02:40:29 Zhang afpd[98580]: AFP statistics: 158696.24 KB read, 258939.78 KB written
Jan  1 02:40:29 Zhang afpd[98580]: done
Jan  1 03:07:12 Zhang syslog-ng[1729]: syslog-ng starting up; version='3.7.3'
Jan  1 03:07:12 Zhang Fatal double fault
Jan  1 03:07:12 Zhang rip 0xffffffff803e2fbf rsp 0xfffffe28f73aeff0 rbp 0xfffffe28f73af050
Jan  1 03:07:12 Zhang rax 0 rdx 0x1 rbx 0xfffff8018cfabb18
Jan  1 03:07:12 Zhang rcx 0x1 rsi 0xfffff8018cfabb18 rdi 0x28d7
Jan  1 03:07:12 Zhang r8 0xfffff815e117ed98 r9 0 r10 0xfffff813d6f03000
Jan  1 03:07:12 Zhang r11 0xffffffff814dd75c r12 0xfffff813d6f03000 r13 0xfffff815e117edf0
Jan  1 03:07:12 Zhang r14 0x28d7 r15 0 rflags 0x10246
Jan  1 03:07:12 Zhang cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
Jan  1 03:07:12 Zhang fsbase 0x800627530 gsbase 0xffffffff821baf80 kgsbase 0
Jan  1 03:07:12 Zhang cpuid = 5; apic id = 05
Jan  1 03:07:12 Zhang panic: double fault
Jan  1 03:07:12 Zhang cpuid = 5
Jan  1 03:07:12 Zhang KDB: stack backtrace:
Jan  1 03:07:12 Zhang db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe2872558d80
Jan  1 03:07:12 Zhang vpanic() at vpanic+0x186/frame 0xfffffe2872558e00
Jan  1 03:07:12 Zhang panic() at panic+0x43/frame 0xfffffe2872558e60
Jan  1 03:07:12 Zhang dblfault_handler() at dblfault_handler+0x1de/frame 0xfffffe2872558f30
Jan  1 03:07:12 Zhang Xdblfault() at Xdblfault+0xac/frame 0xfffffe2872558f30
Jan  1 03:07:12 Zhang --- trap 0x17, rip = 0xffffffff803e2fbf, rsp = 0xfffffe28f73aeff0, rbp = 0xfffffe28f73af050 ---
Jan  1 03:07:12 Zhang dmu_zfetch() at dmu_zfetch+0x2f/frame 0xfffffe28f73af050
Jan  1 03:07:12 Zhang dbuf_read() at dbuf_read+0x177/frame 0xfffffe28f73af0e0
Jan  1 03:07:12 Zhang dnode_hold_impl() at dnode_hold_impl+0x187/frame 0xfffffe28f73af160
Jan  1 03:07:12 Zhang dmu_free_long_range() at dmu_free_long_range+0x2a/frame 0xfffffe28f73af200
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x6a/frame 0xfffffe28f73af3b0
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73af3e0
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73af410
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73af490
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73af4c0
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73af4d0
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73af500
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73af560
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73af5c0
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73af620
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73af650
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73af6d0
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73af740
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73af770
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73af830
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73af9e0
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73afa10
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73afa40
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73afac0
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73afaf0
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73afb00
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73afb30
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73afb90
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73afbf0
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73afc50
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73afc80
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73afd00
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73afd70
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73afda0
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73afe60
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73b0010
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73b0040
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b0070
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b00f0
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73b0120
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73b0130
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73b0160
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73b01c0
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73b0220
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73b0280
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b02b0
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b0330
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73b03a0
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73b03d0
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73b0490
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73b0640
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73b0670
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b06a0
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b0720
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73b0750
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73b0760
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73b0790
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73b07f0
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73b0850
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73b08b0
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b08e0
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b0960
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73b09d0
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73b0a00
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73b0ac0
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73b0c70
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73b0ca0
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b0cd0
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b0d50
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73b0d80
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73b0d90
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73b0dc0
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73b0e20
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73b0e80
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73b0ee0
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b0f10
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b0f90
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73b1000
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73b1030
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73b10f0
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73b12a0
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73b12d0
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b1300
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b1380
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73b13b0
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73b13c0
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73b13f0
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73b1450
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73b14b0
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73b1510
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b1540
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b15c0
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73b1630
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73b1660
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73b1720
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73b18d0
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73b1900
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b1930
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b19b0
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73b19e0
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73b19f0
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73b1a20
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73b1a80
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73b1ae0
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73b1b40
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b1b70
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b1bf0
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73b1c60
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73b1c90
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73b1d50
Jan  1 03:07:12 Zhang zfs_rmnode() at zfs_rmnode+0x295/frame 0xfffffe28f73b1f00
Jan  1 03:07:12 Zhang zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x46/frame 0xfffffe28f73b1f30
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b1f60
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b1fe0
Jan  1 03:07:12 Zhang vrecycle() at vrecycle+0x4d/frame 0xfffffe28f73b2010
Jan  1 03:07:12 Zhang zfs_freebsd_inactive() at zfs_freebsd_inactive+0xd/frame 0xfffffe28f73b2020
Jan  1 03:07:12 Zhang VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0x89/frame 0xfffffe28f73b2050
Jan  1 03:07:12 Zhang vinactive() at vinactive+0xf2/frame 0xfffffe28f73b20b0
Jan  1 03:07:12 Zhang vputx() at vputx+0x2c5/frame 0xfffffe28f73b2110
Jan  1 03:07:12 Zhang null_reclaim() at null_reclaim+0xf6/frame 0xfffffe28f73b2170
Jan  1 03:07:12 Zhang VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0x89/frame 0xfffffe28f73b21a0
Jan  1 03:07:12 Zhang vgonel() at vgonel+0x2a0/frame 0xfffffe28f73b2220
Jan  1 03:07:12 Zhang vnlru_free_locked() at vnlru_free_locked+0x22c/frame 0xfffffe28f73b2290
Jan  1 03:07:12 Zhang getnewvnode_reserve() at getnewvnode_reserve+0x77/frame 0xfffffe28f73b22c0
Jan  1 03:07:12 Zhang zfs_zget() at zfs_zget+0x27/frame 0xfffffe28f73b2380
Jan  1 03:07:12 Zhang zfs_dirent_lookup() at zfs_dirent_lookup+0x15d/frame 0xfffffe28f73b23d0
Jan  1 03:07:12 Zhang zfs_dirlook() at zfs_dirlook+0x77/frame 0xfffffe28f73b2410
Jan  1 03:07:12 Zhang zfs_lookup() at zfs_lookup+0x432/frame 0xfffffe28f73b2500
Jan  1 03:07:12 Zhang zfs_freebsd_lookup() at zfs_freebsd_lookup+0x6d/frame 0xfffffe28f73b2640
Jan  1 03:07:12 Zhang VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x83/frame 0xfffffe28f73b2670
Jan  1 03:07:12 Zhang vfs_cache_lookup() at vfs_cache_lookup+0xd6/frame 0xfffffe28f73b26d0
Jan  1 03:07:12 Zhang VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x83/frame 0xfffffe28f73b2700
Jan  1 03:07:12 Zhang lookup() at lookup+0x6c1/frame 0xfffffe28f73b27a0
Jan  1 03:07:12 Zhang namei() at namei+0x48f/frame 0xfffffe28f73b2870
Jan  1 03:07:12 Zhang kern_statat() at kern_statat+0x98/frame 0xfffffe28f73b2a20
Jan  1 03:07:12 Zhang sys_stat() at sys_stat+0x2d/frame 0xfffffe28f73b2ac0
Jan  1 03:07:12 Zhang amd64_syscall() at amd64_syscall+0xa4a/frame 0xfffffe28f73b2bf0
Jan  1 03:07:12 Zhang Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe28f73b2bf0
Jan  1 03:07:12 Zhang --- syscall (188, FreeBSD ELF64, sys_stat), rip = 0x8014787ba, rsp = 0x7fffffff9c88, rbp = 0x7fffffffa6f0 ---
Jan  1 03:07:12 Zhang KDB: enter: panic
Jan  1 03:07:12 Zhang Copyright (c) 1992-2017 The FreeBSD Project.
Jan  1 03:07:12 Zhang Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jan  1 03:07:12 Zhang	 The Regents of the University of California. All rights reserved.
Jan  1 03:07:12 Zhang FreeBSD is a registered trademark of The FreeBSD Foundation.
Jan  1 03:07:12 Zhang FreeBSD 11.1-STABLE #0 r321665+d4625dcee3e(freenas/11.1-stable): Wed Dec 13 16:33:42 UTC 2017
Jan  1 03:07:12 Zhang root@gauntlet:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
Jan  1 03:07:12 Zhang FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn)
Jan  1 03:07:12 Zhang CPU: Intel(R) Xeon(R) CPU		   L5638  @ 2.00GHz (2000.07-MHz K8-class CPU)
Jan  1 03:07:12 Zhang Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
Jan  1 03:07:12 Zhang Features=0x1fa3fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,DTS,MMX,FXSR,SSE,SSE2,SS,HTT>
Jan  1 03:07:12 Zhang Features2=0x83b82203<SSE3,PCLMULQDQ,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,HV>
Jan  1 03:07:12 Zhang AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
Jan  1 03:07:12 Zhang AMD Features2=0x1<LAHF>
Jan  1 03:07:12 Zhang Structured Extended Features=0x2<TSCADJ>
Jan  1 03:07:12 Zhang TSC: P-state invariant
Jan  1 03:07:12 Zhang Hypervisor: Origin = "VMwareVMware"
Jan  1 03:07:12 Zhang real memory  = 138512695296 (132096 MB)
Jan  1 03:07:12 Zhang avail memory = 133541572608 (127355 MB)
Jan  1 03:07:12 Zhang Event timer "LAPIC" quality 600
Jan  1 03:07:12 Zhang ACPI APIC Table: <PTLTD	   APIC  >
Jan  1 03:07:12 Zhang FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
Jan  1 03:07:12 Zhang FreeBSD/SMP: 4 package(s) x 2 core(s)
Jan  1 03:07:12 Zhang WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Jan  1 03:07:12 Zhang MADT: Forcing active-low polarity and level trigger for SCI
Jan  1 03:07:12 Zhang ioapic0 <Version 1.1> irqs 0-23 on motherboard
Jan  1 03:07:12 Zhang SMP: AP CPU #1 Launched!
Jan  1 03:07:12 Zhang SMP: AP CPU #7 Launched!
Jan  1 03:07:12 Zhang SMP: AP CPU #2 Launched!
Jan  1 03:07:12 Zhang SMP: AP CPU #3 Launched!
Jan  1 03:07:12 Zhang SMP: AP CPU #6 Launched!
Jan  1 03:07:12 Zhang SMP: AP CPU #5 Launched!
Jan  1 03:07:12 Zhang SMP: AP CPU #4 Launched!
Jan  1 03:07:12 Zhang random: entropy device external interface
Jan  1 03:07:12 Zhang kbd1 at kbdmux0
Jan  1 03:07:12 Zhang nexus0
Jan  1 03:07:12 Zhang cryptosoft0: <software crypto> on motherboard
Jan  1 03:07:12 Zhang aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
Jan  1 03:07:12 Zhang padlock0: No ACE support.
Jan  1 03:07:12 Zhang acpi0: <INTEL 440BX> on motherboard
Jan  1 03:07:12 Zhang acpi0: Power Button (fixed)
Jan  1 03:07:12 Zhang hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Jan  1 03:07:12 Zhang Timecounter "HPET" frequency 14318180 Hz quality 950
Jan  1 03:07:12 Zhang cpu0: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu1: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu2: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu3: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu4: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu5: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu6: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang cpu7: <ACPI CPU> numa-domain 0 on acpi0
Jan  1 03:07:12 Zhang attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Jan  1 03:07:12 Zhang Timecounter "i8254" frequency 1193182 Hz quality 0
Jan  1 03:07:12 Zhang Event timer "i8254" frequency 1193182 Hz quality 100
Jan  1 03:07:12 Zhang atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Jan  1 03:07:12 Zhang atrtc0: registered as a time-of-day clock, resolution 1.000000s
Jan  1 03:07:12 Zhang Event timer "RTC" frequency 32768 Hz quality 0
Jan  1 03:07:12 Zhang Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
Jan  1 03:07:12 Zhang acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
Jan  1 03:07:12 Zhang pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Jan  1 03:07:12 Zhang pci0: <ACPI PCI bus> on pcib0
Jan  1 03:07:12 Zhang pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Jan  1 03:07:12 Zhang pci1: <ACPI PCI bus> on pcib1
Jan  1 03:07:12 Zhang isab0: <PCI-ISA bridge> at device 7.0 on pci0
Jan  1 03:07:12 Zhang isa0: <ISA bus> on isab0
Jan  1 03:07:12 Zhang atapci0: <Intel PIIX4 UDMA33 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
Jan  1 03:07:12 Zhang ata0: <ATA channel> at channel 0 on atapci0
Jan  1 03:07:12 Zhang ata1: <ATA channel> at channel 1 on atapci0
Jan  1 03:07:12 Zhang pci0: <bridge> at device 7.3 (no driver attached)
Jan  1 03:07:12 Zhang vgapci0: <VGA-compatible display> port 0x1070-0x107f mem 0xe8000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
Jan  1 03:07:12 Zhang vgapci0: Boot video device
Jan  1 03:07:12 Zhang pcib2: <ACPI PCI-PCI bridge> at device 17.0 on pci0
Jan  1 03:07:12 Zhang pci2: <ACPI PCI bus> on pcib2
Jan  1 03:07:12 Zhang uhci0: <UHCI (generic) USB controller> port 0x2000-0x201f irq 19 at device 1.0 on pci2
Jan  1 03:07:12 Zhang usbus0 on uhci0
Jan  1 03:07:12 Zhang usbus0: 12Mbps Full Speed USB v1.0
Jan  1 03:07:12 Zhang ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfd5ff000-0xfd5fffff irq 16 at device 2.0 on pci2
Jan  1 03:07:12 Zhang usbus1: EHCI version 1.0
Jan  1 03:07:12 Zhang usbus1 on ehci0
Jan  1 03:07:12 Zhang usbus1: 480Mbps High Speed USB v2.0
Jan  1 03:07:12 Zhang pcib3: <ACPI PCI-PCI bridge> at device 21.0 on pci0
Jan  1 03:07:12 Zhang pcib3: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pci3: <ACPI PCI bus> on pcib3
Jan  1 03:07:12 Zhang mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem 0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3
Jan  1 03:07:12 Zhang mpt0: MPI Version=1.5.0.0
Jan  1 03:07:12 Zhang pcib4: <ACPI PCI-PCI bridge> at device 21.1 on pci0
Jan  1 03:07:12 Zhang pcib4: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pci4: <ACPI PCI bus> on pcib4
Jan  1 03:07:12 Zhang mps0: <Avago Technologies (LSI) SAS2008> port 0x8000-0x80ff mem 0xfd0fc000-0xfd0fffff,0xfd080000-0xfd0bffff irq 18 at device 0.0 on pci4
Jan  1 03:07:12 Zhang mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
Jan  1 03:07:12 Zhang mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Jan  1 03:07:12 Zhang pcib5: <ACPI PCI-PCI bridge> at device 21.2 on pci0
Jan  1 03:07:12 Zhang pcib5: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pcib6: <ACPI PCI-PCI bridge> at device 21.3 on pci0
Jan  1 03:07:12 Zhang pcib6: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pcib7: <ACPI PCI-PCI bridge> at device 21.4 on pci0
Jan  1 03:07:12 Zhang pcib7: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pcib8: <ACPI PCI-PCI bridge> at device 21.5 on pci0
Jan  1 03:07:12 Zhang pcib8: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pcib9: <ACPI PCI-PCI bridge> at device 21.6 on pci0
Jan  1 03:07:12 Zhang pcib9: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pcib10: <ACPI PCI-PCI bridge> at device 21.7 on pci0
Jan  1 03:07:12 Zhang pcib10: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pcib11: <ACPI PCI-PCI bridge> at device 22.0 on pci0
Jan  1 03:07:12 Zhang pcib11: [GIANT-LOCKED]
Jan  1 03:07:12 Zhang pci5: <ACPI PCI bus> on pcib11

Everything past the above is just normal boot up information.

It did reboot. I have now disabled the only setting that I had changed prior to these reboots.

I will keep my eye on this and see if it has the same issue again.

If there is anything else I can try/do/provide, please let me know.

Cheers,

Scharbag · Jan 1, 2018

danb35 said:
Did your system(s) actually reboot? Because I've gotten false positives with that email. uptime would tell you.

Yes, both nights that I received the emails, my system booted. Confirmed by looking at uptime.

And for reference, my pools are all healthy if that matters:

Code:

########## ZPool status report summary for all pools on server ZHANG ##########

+--------------+--------+------+------+------+----+----+--------+------+-----+
|Pool Name	 |Status  |Read  |Write |Cksum |Used|Frag|Scrub   |Scrub |Last |
|			  |		|Errors|Errors|Errors|	|	|Repaired|Errors|Scrub|
|			  |		|	  |	  |	  |	|	|Bytes   |	  |Age  |
+--------------+--------+------+------+------+----+----+--------+------+-----+
|backuptank	|ONLINE  |	 0|	 0|	 0| 74%| 17%|	   0|	 0|	3|
|bigtank	   |ONLINE  |	 0|	 0|	 0| 72%| 24%|	   0|	 0|	8|
|fasttank	  |ONLINE  |	 0|	 0|	 0| 33%| 46%|	   0|	 0|   11|
|freenas-boot  |ONLINE  |	 0|	 0|	 0| 10%|	|	   0|	 0|	3|
+--------------+--------+------+------+------+----+----+--------+------+-----+

Cheers,

rovan · Jan 1, 2018

I just had an unauthorised reboot as well. drives are healthy just one dataset over 80% full.

Running Freenas 11.1. @Scharbag see attached.

Scharbag · Jan 1, 2018

rovan said:
I just had an unauthorised reboot as well. drives are healthy just one dataset over 80% full.

Running Freenas 11.1.

Could you please post your var/logs/messages results for when the system rebooted?

Cheers,

rovan · Jan 2, 2018

anyone else have any ideas ^_^"

scrappy · Jan 2, 2018

This unauthorized reboot message is happening to my remote FreeNAS machine running 11.1 almost daily.

My local FreeNAS machine also running 11.1 sent me an email about unauthorized reboot once while I was actively using it. In that case I know it never actually rebooted but it sent the email saying it had.

Scharbag · Jan 2, 2018

scrappy said:
This unauthorized reboot message is happening to my remote FreeNAS machine running 11.1 almost daily.

My local FreeNAS machine also running 11.1 sent me an email about unauthorized reboot once while I was actively using it. In that case I know it never actually rebooted but it sent the email saying it had.

My uptime was reset so I think my system rebooted. It has been better the last 2 days. This is a strange one...

I-Tech · Jan 8, 2018

have also had several unauthorized reboots.. at random times..
first was the day I updated to current 11.1 (has not happened before until this version)
most recent 2 were Sunday at 12:23 PM and again at 7:50 PM (uptime confirmed)
no entries in /var/log/messages showing anything interesting prior to reboot sequence.
(only entry that is new is "daemon[3681]: ==> Newer Consul version available: 1.0.2 (currently running: 1.0.0)"
also .. on first update .. the reporting history disappeared.. thought it had something to do with the update (first time this happened too)
.. and right now the reporting history is only showing since last reboot on Sunday evening.
also have the reporting database option unselected as shown by Sharbag above.

any suggestions??

edit: zpool status ok

Code:

# zpool status
  pool: BackupZ2
 state: ONLINE
  scan: scrub repaired 0 in 0 days 03:41:58 with 0 errors on Sun Dec 31 03:41:59 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		BackupZ2										ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/c07b8cd2-2ec2-11e7-8afd-0019b9cbdf75  ONLINE	   0	 0	 0
			gptid/c25ea057-2ec2-11e7-8afd-0019b9cbdf75  ONLINE	   0	 0	 0
			gptid/c3423f93-2ec2-11e7-8afd-0019b9cbdf75  ONLINE	   0	 0	 0
			gptid/c42e75b8-2ec2-11e7-8afd-0019b9cbdf75  ONLINE	   0	 0	 0
			gptid/c55a07a1-2ec2-11e7-8afd-0019b9cbdf75  ONLINE	   0	 0	 0
			gptid/c68d4106-2ec2-11e7-8afd-0019b9cbdf75  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:08:28 with 0 errors on Thu Jan  4 03:53:28 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		freenas-boot									ONLINE	   0	 0	 0
		  mirror-0									  ONLINE	   0	 0	 0
			gptid/92fd215d-4b72-11e5-8727-00188b52cbd0  ONLINE	   0	 0	 0
			gptid/3069bf82-bb28-11e5-9301-0019b9cbdf75  ONLINE	   0	 0	 0

errors: No known data errors

DJABE · Jan 8, 2018

It is quite obvious that kernel panic (e.g. system fatal error) occurs (low level), thus resulting system to reboot. I see all of you have 11.1 version, could it be the culprit?
Something killing your memory? :) What ECC event log shows in your BIOS/Mobo/IPMI?

I-Tech · Jan 8, 2018

DJABE said:
It is quite obvious that kernel panic (e.g. system fatal error) occurs (low level), thus resulting system to reboot. I see all of you have 11.1 version, could it be the culprit?
Something killing your memory? :) What ECC event log shows in your BIOS/Mobo/IPMI?

no (relevant) entries in system event..
last one was Jan 01 06:41 .. System Board OS Watchdog: Watchdog sensor for System Board, reboot was asserted.
there were no other entries for the latest series of reboots.

baodad · Jan 8, 2018

Me too. Upgraded to 11.1. System had been up for over 80 days before I updated. Now it's rebooted twice. Glad I'm not the only one having this issue. Hope this gets attention and gets fixed. Thanks all.

vikozo6 · Jan 9, 2018

I did get the same email
System booted at Sun Jan 7 02:13:58 2018 was not shut down properly
so I will read what should have been the Problem ;-)

PhilipS · Jan 9, 2018

Is there a bug report for this? Are these reboots possibly due to this bug?: https://redmine.ixsystems.com/issues/27270

Important Announcement for the TrueNAS Community.

Unauthorized Reboot - 11.1-RELEASE

Guru

Attachments

Server Wrangler

Dabbler

Hall of Famer

Server Wrangler

Guru

Dabbler

Guru

Guru

Dabbler

Attachments

Guru

Dabbler

Patron

Guru

Dabbler

Contributor

Dabbler

Dabbler

Patron

Contributor

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Unauthorized Reboot - 11.1-RELEASE"

Similar threads