Unscheduled Reboot

[MattTaylor2]

Can anyone reccomend which log/debug dump to look in that will provide me with some information on why my FreeNAS performed an unsheduled reboot?

Casued some issues with a pool that was resilvering.

It has had a couple of unschedueld reboots over the past 2 weeks so would be good to know whatrs casuing it.

 

[sretalla]

You may find a core dump file in:

/var/db/system/cores

Separately to that /var/log/messages might have something.

 

[MattTaylor2]

You may find a core dump file in:

/var/db/system/cores

Separately to that /var/log/messages might have something.

Thanks for the reply, i'll have a look at the core dump file.

I've been through the debug dumps in /var/tmp/fndebug but can't see anything that stands out as to whats casuing it.

/var/log has has a disk error, which had been logging for a while before the unscheduled reboot, then there's a time gap of 48 minutes which then shows the logs of the NAS booting up

 

[andrewjones216]

Here's the panic text from the crash dump:

Code:

panic: Solaris(panic): error 6 from bpobj_iterate()
cpuid = 9
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe183e20d320
vpanic() at vpanic+0x17e/frame 0xfffffe183e20d380
panic() at panic+0x43/frame 0xfffffe183e20d3e0
vcmn_err() at vcmn_err+0xcf/frame 0xfffffe183e20d510
zfs_panic_recover() at zfs_panic_recover+0x5a/frame 0xfffffe183e20d570
dsl_scan_sync() at dsl_scan_sync+0x33a/frame 0xfffffe183e20d770
spa_sync() at spa_sync+0xb67/frame 0xfffffe183e20d9a0
txg_sync_thread() at txg_sync_thread+0x238/frame 0xfffffe183e20da70
fork_exit() at fork_exit+0x83/frame 0xfffffe183e20dab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe183e20dab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic

Nothing that I can understand in there sadly! Hoping one of the FreeNas/ BSD gurus can point me in the right direction.
The machine has been running for years now, only changes that regularly occur are the swapping out of degraded/ failing disks.

 

[Samuel Tai]

FreeBSD ZFS: bpobj.c File Reference

people.freebsd.org

You say this was from a crash during a resilver? ZFS ran into a block of garbage in your pool it couldn't process. I hope you have backups, because you're likely going to have to destroy and rebuild your pool from backup, because your pool is too corrupt for ZFS to repair.

 

[andrewjones216]

FreeBSD ZFS: bpobj.c File Reference

people.freebsd.org

You say this was from a crash during a resilver? ZFS ran into a block of garbage in your pool it couldn't process. I hope you have backups, because you're likely going to have to destroy and rebuild your pool from backup, because your pool is too corrupt for ZFS to repair.

Yes that's correct a resilver was taking place at the time, sounds like we've lost that pool. Luckily we also snapshot and replicate all our pools to another FreeNAS server.

Is there anything we can do to avoid messing up ZFS like this in the future?

 

[Samuel Tai]

ZFS is usually extremely robust, so it looks like you just happened unluckily to draw the short straw with respect to entropy that day.