kernel panic & unscheduled system reboot after successful replication run

[tj1627]

Hi All,

I'm experiencing a new kernel panic on my backup system that began today.

I have 2 truenas core 13.0 boxes, tank & dozer.

I have a replication task on dozer that pulls snapshots from tank at 15 and 45 past the hour. This has been running successfully for months (Under 12.0 U7 and then now 13.0). Today I had an unexpected reboot at 9:16 and then again at 9:46, 10:16, and 10:46.

The replication task shows successful, and I can list snapshots from tank created at 10am for instance, so it appears to be replicating successfully.

When looking at /data/crash/textdump.tar.gz files, I see this at the tail end of msgbuf.txt:

Code:

cpuid = 6
time = 1654019104
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01aa5e2ce0
vpanic() at vpanic+0x17f/frame 0xfffffe01aa5e2d30
spl_panic() at spl_panic+0x3a/frame 0xfffffe01aa5e2d90
receive_object() at receive_object+0x6d0/frame 0xfffffe01aa5e2e80
receive_writer_thread() at receive_writer_thread+0x199/frame 0xfffffe01aa5e2ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe01aa5e2f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe01aa5e2f30
--- trap 0xc, rip = 0x800804e7a, rsp = 0x7fffffff8f88, rbp = 0x7fffffff8ff0 ---
KDB: enter: panic

Admittedly, I'm more familiar troubleshooting linux than freeBSD, so I'm looking for advice on how to troubleshoot this. I've disabled the replication task for now and I haven't experienced a new unexpected restart.

the backup system, dozer, is an old gaming setup, core i7-8700K, 64gb ddr4 (non-ecc), LSI controller, and drives laid out in 2 raid z2 vdevs: 6x8tb and 6x10tb

Any ideas on where to begin?

 

[Apollo]

Having the same issue while trying to run manual replication (via CLI over SSH) from local system running TrueNAS 12.0-U8.1 to remote system running TrueNAS 13.0-RELEASE:

zfs send -I NNNNNNNN | ssh -i /data/ssh/replication root@192.168.1.147 zfs receive -F NNNNNNN

Reverted to TrueNAS-12.0-U8.1 and performing ongoing replication. No issues as expected.