Hi I am having an issue where I have 2 hosts running Freenas 11.2 u7 and they are rebooting when I do snapshot deletes.
I have the 2 hosts configured for snapshots every hour and replication turned on. I noticed after testing for about a month or so that I forgot to check the
Delete Stale Snapshots on Remote System. I went in the other day and checked that flag. I checked periodically within the hour and saw the space on the remote host was going down and left it be to complete the task. I realized that the secondary host rebooted itself multiple times and also rebooted the primary overnight. Below are my specs and the output of the dump. Before enabling the "Delete Stale Snapshots on Remote System" both hosts had an up time of approximately 40 days,
Specs:
Dell PowerEdge R730xd
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz
64.00 GB DDR4
2x Samsung SSD 883 240 GB (OS)
4X Samsung SSD 883 3.8 TB (raid z1)
Solarflare Communications SFN7x02F-R2 Flareon 7000 Series 10G Adapter
<118>Tue Mar 3 18:06:53 CST 2020
panic: I/O to pool 'pool1' appears to be hung on vdev guid 8000783744409220160 at '/dev/gptid/1cf6de69-3d30-11ea-bdc3-000f53270930'.
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0f926ba800
vpanic() at vpanic+0x177/frame 0xfffffe0f926ba860
panic() at panic+0x43/frame 0xfffffe0f926ba8c0
vdev_deadman() at vdev_deadman+0x191/frame 0xfffffe0f926ba910
vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe0f926ba960
vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe0f926ba9b0
spa_deadman() at spa_deadman+0x87/frame 0xfffffe0f926ba9e0
taskqueue_run_locked() at taskqueue_run_locked+0x154/frame 0xfffffe0f926baa40
taskqueue_thread_loop() at taskqueue_thread_loop+0x98/frame 0xfffffe0f926baa70
fork_exit() at fork_exit+0x83/frame 0xfffffe0f926baab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f926baab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
I have the 2 hosts configured for snapshots every hour and replication turned on. I noticed after testing for about a month or so that I forgot to check the
Delete Stale Snapshots on Remote System. I went in the other day and checked that flag. I checked periodically within the hour and saw the space on the remote host was going down and left it be to complete the task. I realized that the secondary host rebooted itself multiple times and also rebooted the primary overnight. Below are my specs and the output of the dump. Before enabling the "Delete Stale Snapshots on Remote System" both hosts had an up time of approximately 40 days,
Specs:
Dell PowerEdge R730xd
Intel(R) Xeon(R) CPU E5-2643 v3 @ 3.40GHz
64.00 GB DDR4
2x Samsung SSD 883 240 GB (OS)
4X Samsung SSD 883 3.8 TB (raid z1)
Solarflare Communications SFN7x02F-R2 Flareon 7000 Series 10G Adapter
<118>Tue Mar 3 18:06:53 CST 2020
panic: I/O to pool 'pool1' appears to be hung on vdev guid 8000783744409220160 at '/dev/gptid/1cf6de69-3d30-11ea-bdc3-000f53270930'.
cpuid = 2
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0f926ba800
vpanic() at vpanic+0x177/frame 0xfffffe0f926ba860
panic() at panic+0x43/frame 0xfffffe0f926ba8c0
vdev_deadman() at vdev_deadman+0x191/frame 0xfffffe0f926ba910
vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe0f926ba960
vdev_deadman() at vdev_deadman+0x41/frame 0xfffffe0f926ba9b0
spa_deadman() at spa_deadman+0x87/frame 0xfffffe0f926ba9e0
taskqueue_run_locked() at taskqueue_run_locked+0x154/frame 0xfffffe0f926baa40
taskqueue_thread_loop() at taskqueue_thread_loop+0x98/frame 0xfffffe0f926baa70
fork_exit() at fork_exit+0x83/frame 0xfffffe0f926baab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0f926baab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
