freeaq2004
Cadet
- Joined
- Dec 26, 2017
- Messages
- 4
FreeNAS appliance has intermittently crashed twice over the past few days and I'm unsure what's causing it. In checking /var/log/messages, I noticed that there are several "stack backtrace" messages related to the network interfaces after the system has recovered from the kernel panic.
Currently running FreeNAS-11.0-U4 (54848d13b) on an iXSystems built appliance server NFSv3 to about 50 clients. Smartctl checked out OK on all hard disks and ZFS scrub sees no errors with zpool.
Code:
Dec 26 13:57:42 scratchfs-01 lock order reversal: Dec 26 13:57:42 scratchfs-01 1st 0xfffff8000df4c858 ix0:rx(7) (ix0:rx(7)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763 Dec 26 13:57:42 scratchfs-01 2nd 0xfffff8002779b608 if_lagg rmlock (if_lagg rmlock) @ /freenas-11-releng/freenas/_BE/os/sys/net/if_lagg.c:1603 Dec 26 13:57:42 scratchfs-01 stack backtrace: Dec 26 13:57:42 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70 Dec 26 13:57:42 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23 Dec 26 13:57:42 scratchfs-01 #2 0xffffffff80a85129 at _rm_rlock_debug+0x119 Dec 26 13:57:42 scratchfs-01 #3 0xffffffff80ba39c2 at lagg_input+0x42 Dec 26 13:57:42 scratchfs-01 #4 0xffffffff80b9e30c at ether_nh_input+0x1dc Dec 26 13:57:42 scratchfs-01 #5 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f Dec 26 13:57:42 scratchfs-01 #6 0xffffffff80b9d84f at ether_input+0x8f Dec 26 13:57:42 scratchfs-01 #7 0xffffffff80bfb517 at tcp_lro_flush+0x1b7 Dec 26 13:57:42 scratchfs-01 #8 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b Dec 26 13:57:42 scratchfs-01 #9 0xffffffff806ee967 at ixgbe_rxeof+0x787 Dec 26 13:57:42 scratchfs-01 #10 0xffffffff806e48e7 at ixgbe_msix_que+0x97 Dec 26 13:57:42 scratchfs-01 #11 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99 Dec 26 13:57:42 scratchfs-01 #12 0xffffffff80a4ec86 at ithread_loop+0xb6 Dec 26 13:57:42 scratchfs-01 #13 0xffffffff80a4bcf4 at fork_exit+0x84 Dec 26 13:57:42 scratchfs-01 #14 0xffffffff80f250fe at fork_trampoline+0xe Dec 26 13:57:42 scratchfs-01 lock order reversal: Dec 26 13:57:42 scratchfs-01 1st 0xfffff8000df4c858 ix0:rx(7) (ix0:rx(7)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763 Dec 26 13:57:42 scratchfs-01 2nd 0xfffffe0000dabe08 pcbinfohash (pcbinfohash) @ /freenas-11-releng/freenas/_BE/os/sys/netinet/in_pcb.c:1959 Dec 26 13:57:42 scratchfs-01 stack backtrace: Dec 26 13:57:42 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70 Dec 26 13:57:42 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23 Dec 26 13:57:42 scratchfs-01 #2 0xffffffff80a86162 at __rw_rlock+0x82 Dec 26 13:57:42 scratchfs-01 #3 0xffffffff80be571b at in_pcblookup_hash+0x3b Dec 26 13:57:42 scratchfs-01 #4 0xffffffff80bf5da1 at tcp_input+0x921 Dec 26 13:57:42 scratchfs-01 #5 0xffffffff80beb367 at ip_input+0x137 Dec 26 13:57:42 scratchfs-01 #6 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f Dec 26 13:57:42 scratchfs-01 #7 0xffffffff80b9d4ae at ether_demux+0x16e Dec 26 13:57:42 scratchfs-01 #8 0xffffffff80b9e4cc at ether_nh_input+0x39c Dec 26 13:57:42 scratchfs-01 #9 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f Dec 26 13:57:42 scratchfs-01 #10 0xffffffff80b9d84f at ether_input+0x8f Dec 26 13:57:42 scratchfs-01 #11 0xffffffff80bfb517 at tcp_lro_flush+0x1b7 Dec 26 13:57:42 scratchfs-01 #12 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b Dec 26 13:57:42 scratchfs-01 #13 0xffffffff806ee967 at ixgbe_rxeof+0x787 Dec 26 13:57:42 scratchfs-01 #14 0xffffffff806e48e7 at ixgbe_msix_que+0x97 Dec 26 13:57:42 scratchfs-01 #15 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99 Dec 26 13:57:42 scratchfs-01 #16 0xffffffff80a4ec86 at ithread_loop+0xb6 Dec 26 13:57:42 scratchfs-01 #17 0xffffffff80a4bcf4 at fork_exit+0x84 Dec 26 13:57:42 scratchfs-01 lock order reversal: Dec 26 13:57:42 scratchfs-01 1st 0xfffff8000df4c858 ix0:rx(7) (ix0:rx(7)) @ /freenas-11-releng/freenas/_BE/os/sys/dev/ixgbe/ix_txrx.c:1763 Dec 26 13:57:42 scratchfs-01 2nd 0xfffff80143fb7338 tcpinp (tcpinp) @ /freenas-11-releng/freenas/_BE/os/sys/netinet/in_pcb.c:1966 Dec 26 13:57:42 scratchfs-01 stack backtrace: Dec 26 13:57:42 scratchfs-01 #0 0xffffffff80aea690 at witness_debugger+0x70 Dec 26 13:57:42 scratchfs-01 #1 0xffffffff80aea583 at witness_checkorder+0xe23 Dec 26 13:57:42 scratchfs-01 #2 0xffffffff80a8571e at _rw_wlock_cookie+0x5e Dec 26 13:57:42 scratchfs-01 #3 0xffffffff80be57e0 at in_pcblookup_hash+0x100 Dec 26 13:57:42 scratchfs-01 #4 0xffffffff80bf5da1 at tcp_input+0x921 Dec 26 13:57:42 scratchfs-01 #5 0xffffffff80beb367 at ip_input+0x137 Dec 26 13:57:42 scratchfs-01 #6 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f Dec 26 13:57:42 scratchfs-01 #7 0xffffffff80b9d4ae at ether_demux+0x16e Dec 26 13:57:42 scratchfs-01 #8 0xffffffff80b9e4cc at ether_nh_input+0x39c Dec 26 13:57:42 scratchfs-01 #9 0xffffffff80bb8d4f at netisr_dispatch_src+0x9f Dec 26 13:57:42 scratchfs-01 #10 0xffffffff80b9d84f at ether_input+0x8f Dec 26 13:57:42 scratchfs-01 #11 0xffffffff80bfb517 at tcp_lro_flush+0x1b7 Dec 26 13:57:42 scratchfs-01 #12 0xffffffff80bfb68b at tcp_lro_flush_all+0x10b Dec 26 13:57:42 scratchfs-01 #13 0xffffffff806ee967 at ixgbe_rxeof+0x787 Dec 26 13:57:42 scratchfs-01 #14 0xffffffff806e48e7 at ixgbe_msix_que+0x97 Dec 26 13:57:42 scratchfs-01 #15 0xffffffff80a4e5e9 at intr_event_execute_handlers+0x99 Dec 26 13:57:42 scratchfs-01 #16 0xffffffff80a4ec86 at ithread_loop+0xb6 Dec 26 13:57:42 scratchfs-01 #17 0xffffffff80a4bcf4 at fork_exit+0x84
Currently running FreeNAS-11.0-U4 (54848d13b) on an iXSystems built appliance server NFSv3 to about 50 clients. Smartctl checked out OK on all hard disks and ZFS scrub sees no errors with zpool.