Dec 18 09:11:33 truenas kernel: nvme nvme0: I/O 26 QID 0 timeout, reset controller
Dec 18 09:12:14 truenas kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Dec 18 09:12:14 truenas kernel: nvme nvme0: Abort status: 0x371
Dec 18 09:12:14 truenas kernel: nvme nvme0: Abort status: 0x371
Dec 18 09:12:14 truenas kernel: nvme nvme0: Abort status: 0x371
Dec 18 09:12:14 truenas kernel: nvme nvme0: Abort status: 0x371
Dec 18 09:12:14 truenas kernel: nvme nvme0: Abort status: 0x371
Dec 18 09:12:37 truenas kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Dec 18 09:12:37 truenas kernel: nvme nvme0: Removing after probe failure status: -19
Dec 18 09:12:59 truenas kernel: nvme nvme0: Device not ready; aborting reset, CSTS=0x1
Dec 18 09:12:59 truenas kernel: nvme0n1: detected capacity change from 488397168 to 0
Dec 18 09:12:59 truenas kernel: blk_update_request: I/O error, dev nvme0n1, sector 43069720 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Dec 18 09:12:59 truenas kernel: blk_update_request: I/O error, dev nvme0n1, sector 64012688 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Dec 18 09:12:59 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=15054610432 size=4096 flags=180880
Dec 18 09:12:59 truenas kernel: blk_update_request: I/O error, dev nvme0n1, sector 39032344 op 0x0:(READ) flags 0x0 phys_seg 3 prio class 0
Dec 18 09:12:59 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=2264674304 size=77824 flags=180880
Dec 18 09:12:59 truenas kernel: blk_update_request: I/O error, dev nvme0n1, sector 56635576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Dec 18 09:12:59 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=11277529088 size=4096 flags=180880
Dec 18 09:12:59 truenas kernel: blk_update_request: I/O error, dev nvme0n1, sector 39886144 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=4331810816 size=4096 flags=180880
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=2701819904 size=4096 flags=180880
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=15425474560 size=8192 flags=180880
Dec 18 09:13:00 truenas kernel: blk_update_request: I/O error, dev nvme0n1, sector 63990520 op 0x0:(READ) flags 0x0 phys_seg 4 prio class 0
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=6452666368 size=4096 flags=180880
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=15043260416 size=61440 flags=180880
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=11318845440 size=4096 flags=180880
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=40820453376 size=4096 flags=180880
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: zio pool=boot-pool vdev=/dev/nvme0n1p3 error=5 type=1 offset=11524669440 size=4096 flags=180880
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:00 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:13:01 truenas kernel: INFO: task asyncio_loop:2818 blocked for more than 120 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:asyncio_loop    state:D stack:    0 pid: 2818 ppid:     1 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:13:01 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:13:01 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:13:01 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:13:01 truenas kernel:  dbuf_hold_impl+0x459/0x640 [zfs]
Dec 18 09:13:01 truenas kernel:  dbuf_hold_level+0x2b/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  ? avl_add+0x46/0x90 [zavl]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x13a/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:01 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:01 truenas kernel:  __x64_sys_pread64+0x91/0xc0
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x7fbe246d9917
Dec 18 09:13:01 truenas kernel: RSP: 002b:00007fbe0f7fc230 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 0000000000000010 RCX: 00007fbe246d9917
Dec 18 09:13:01 truenas kernel: RDX: 0000000000000010 RSI: 00007fbe0f7fc2c0 RDI: 0000000000000018
Dec 18 09:13:01 truenas kernel: RBP: 0000000000000010 R08: 0000000000000000 R09: 0000000001dffb88
Dec 18 09:13:01 truenas kernel: R10: 0000000000000018 R11: 0000000000000293 R12: 00007fbe0f7fc2c0
Dec 18 09:13:01 truenas kernel: R13: 0000000000000010 R14: 00007fbdfc006840 R15: 0000000000000018
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task multus:2049089 blocked for more than 120 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:multus          state:D stack:    0 pid:2049089 ppid: 17840 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:13:01 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:13:01 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:13:01 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x38f/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  mappedread+0xe1/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x189/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:01 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:01 truenas kernel:  __x64_sys_pread64+0x91/0xc0
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x40394e
Dec 18 09:13:01 truenas kernel: RSP: 002b:000000c0000d4f78 EFLAGS: 00000202 ORIG_RAX: 0000000000000011
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 000000000040394e
Dec 18 09:13:01 truenas kernel: RDX: 0000000000000040 RSI: 000000c0000d51e8 RDI: 0000000000000004
Dec 18 09:13:01 truenas kernel: RBP: 000000c0000d4fb8 R08: 0000000000000000 R09: 0000000000000000
Dec 18 09:13:01 truenas kernel: R10: 00000000007c30de R11: 0000000000000202 R12: 0000000000000000
Dec 18 09:13:01 truenas kernel: R13: 0000000000000000 R14: 000000c0000061a0 R15: 00007f6cdf224a06
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task runc:2050980 blocked for more than 120 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:runc            state:D stack:    0 pid:2050980 ppid: 18547 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:13:01 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:13:01 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:13:01 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x38f/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  mappedread+0xe1/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x189/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:01 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:01 truenas kernel:  __x64_sys_pread64+0x91/0xc0
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x55f092c38d2a
Dec 18 09:13:01 truenas kernel: RSP: 002b:000000c0000595f0 EFLAGS: 00000206 ORIG_RAX: 0000000000000011
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 000000c00002e000 RCX: 000055f092c38d2a
Dec 18 09:13:01 truenas kernel: RDX: 0000000000000040 RSI: 000000c0000597a8 RDI: 0000000000000003
Dec 18 09:13:01 truenas kernel: RBP: 000000c000059658 R08: 0000000000000000 R09: 0000000000000000
Dec 18 09:13:01 truenas kernel: R10: 00000000005314ac R11: 0000000000000206 R12: 000000c000059558
Dec 18 09:13:01 truenas kernel: R13: 0000000000000000 R14: 000000c0000001a0 R15: 00007fd8c726cf82
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task runc:2050986 blocked for more than 120 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:runc            state:D stack:    0 pid:2050986 ppid: 18490 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  cv_wait_common+0xf5/0x130 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x3f1/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  mappedread+0xe1/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x189/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:01 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:01 truenas kernel:  __x64_sys_pread64+0x91/0xc0
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x5579388ebd2a
Dec 18 09:13:01 truenas kernel: RSP: 002b:000000c0000595f0 EFLAGS: 00000206 ORIG_RAX: 0000000000000011
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 000000c00002e000 RCX: 00005579388ebd2a
Dec 18 09:13:01 truenas kernel: RDX: 0000000000000040 RSI: 000000c0000597a8 RDI: 0000000000000003
Dec 18 09:13:01 truenas kernel: RBP: 000000c000059658 R08: 0000000000000000 R09: 0000000000000000
Dec 18 09:13:01 truenas kernel: R10: 00000000005314ac R11: 0000000000000206 R12: 000000c000059558
Dec 18 09:13:01 truenas kernel: R13: 0000000000000000 R14: 000000c0000001a0 R15: 00007fb071db5f68
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task IoThread:2052896 blocked for more than 120 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:IoThread        state:D stack:    0 pid:2052896 ppid:  6798 flags:0x00004000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:13:01 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:13:01 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:13:01 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x38f/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  ? avl_add+0x46/0x90 [zavl]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x13a/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  __kernel_read+0x149/0x2e0
Dec 18 09:13:01 truenas kernel:  bprm_execve+0x217/0x660
Dec 18 09:13:01 truenas kernel:  do_execveat_common+0x1b5/0x230
Dec 18 09:13:01 truenas kernel:  __x64_sys_execve+0x39/0x50
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x7fbe2441e087
Dec 18 09:13:01 truenas kernel: RSP: 002b:00007fbdafffdc28 EFLAGS: 00000206 ORIG_RAX: 000000000000003b
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fbe2441e087
Dec 18 09:13:01 truenas kernel: RDX: 00000000037852c0 RSI: 00007fbe0c07af30 RDI: 00007fbe0c07ad50
Dec 18 09:13:01 truenas kernel: RBP: 00007fbe0c07af30 R08: 0000000000000000 R09: 00007fbd00000030
Dec 18 09:13:01 truenas kernel: R10: fffffffffffff11a R11: 0000000000000206 R12: 0000000000000000
Dec 18 09:13:01 truenas kernel: R13: 00007fbe0c3c5290 R14: 0000000000000000 R15: 000000000000004d
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task runc:2054143 blocked for more than 121 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:runc            state:D stack:    0 pid:2054143 ppid: 16675 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  cv_wait_common+0xf5/0x130 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x3f1/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  mappedread+0xe1/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x189/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:01 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:01 truenas kernel:  __x64_sys_pread64+0x91/0xc0
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x561191345d2a
Dec 18 09:13:01 truenas kernel: RSP: 002b:000000c0000545f0 EFLAGS: 00000206 ORIG_RAX: 0000000000000011
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 000000c000030800 RCX: 0000561191345d2a
Dec 18 09:13:01 truenas kernel: RDX: 0000000000000040 RSI: 000000c0000547a8 RDI: 0000000000000003
Dec 18 09:13:01 truenas kernel: RBP: 000000c000054658 R08: 0000000000000000 R09: 0000000000000000
Dec 18 09:13:01 truenas kernel: R10: 00000000005314ac R11: 0000000000000206 R12: 000000c000054558
Dec 18 09:13:01 truenas kernel: R13: 0000000000000000 R14: 000000c0000001a0 R15: 00007fa7abacbe68
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task sensors:2057997 blocked for more than 121 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:sensors         state:D stack:    0 pid:2057997 ppid:  5032 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:13:01 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:13:01 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:13:01 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x38f/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:01 truenas kernel:  ? avl_add+0x46/0x90 [zavl]
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  zfs_read+0x13a/0x3d0 [zfs]
Dec 18 09:13:01 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:01 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:01 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:01 truenas kernel:  ksys_read+0x5f/0xe0
Dec 18 09:13:01 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:01 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:01 truenas kernel: RIP: 0033:0x7f029c8a384e
Dec 18 09:13:01 truenas kernel: RSP: 002b:00007fff677f8608 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Dec 18 09:13:01 truenas kernel: RAX: ffffffffffffffda RBX: 000055ccc09772c0 RCX: 00007f029c8a384e
Dec 18 09:13:01 truenas kernel: RDX: 0000000000002000 RSI: 000055ccc0977a80 RDI: 0000000000000003
Dec 18 09:13:01 truenas kernel: RBP: 00007f029c983480 R08: 0000000000000003 R09: 00007f029c986be0
Dec 18 09:13:01 truenas kernel: R10: 0000000040000000 R11: 0000000000000246 R12: 0000000000002000
Dec 18 09:13:01 truenas kernel: R13: 000055ccc0977a80 R14: 0000000000000d68 R15: 00007f029c982880
Dec 18 09:13:01 truenas kernel:  </TASK>
Dec 18 09:13:01 truenas kernel: INFO: task zfs:2060155 blocked for more than 121 seconds.
Dec 18 09:13:01 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:13:01 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:13:01 truenas kernel: task:zfs             state:D stack:    0 pid:2060155 ppid:2060154 flags:0x00000000
Dec 18 09:13:01 truenas kernel: Call Trace:
Dec 18 09:13:01 truenas kernel:  <TASK>
Dec 18 09:13:01 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:13:01 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:13:01 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:13:01 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:13:01 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:13:01 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:13:01 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:13:01 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:13:01 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:13:01 truenas kernel:  dbuf_hold_impl+0x459/0x640 [zfs]
Dec 18 09:13:01 truenas kernel:  dbuf_hold_level+0x2b/0x60 [zfs]
Dec 18 09:13:01 truenas kernel:  dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs]
Dec 18 09:13:01 truenas kernel:  ? get_page_from_freelist+0x1126/0x13b0
Dec 18 09:13:01 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:13:02 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:13:02 truenas kernel:  mappedread+0xe1/0x110 [zfs]
Dec 18 09:13:02 truenas kernel:  zfs_read+0x189/0x3d0 [zfs]
Dec 18 09:13:02 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:13:02 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:13:02 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:13:02 truenas kernel:  ksys_read+0x5f/0xe0
Dec 18 09:13:02 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:13:02 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:13:02 truenas kernel: RIP: 0033:0x7f0f85f9c84e
Dec 18 09:13:02 truenas kernel: RSP: 002b:00007ffe44668858 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Dec 18 09:13:02 truenas kernel: RAX: ffffffffffffffda RBX: 0000000000000d68 RCX: 00007f0f85f9c84e
Dec 18 09:13:02 truenas kernel: RDX: 0000000000000181 RSI: 0000556b67ba65a0 RDI: 0000000000000003
Dec 18 09:13:02 truenas kernel: RBP: 0000556b67ba5c90 R08: 0000000000000003 R09: 00007f0f8607fbe0
Dec 18 09:13:02 truenas kernel: R10: 0000000000000070 R11: 0000000000000246 R12: 00000000002e6181
Dec 18 09:13:02 truenas kernel: R13: 0000000000000181 R14: 00007f0f8607b880 R15: 0000000000000001
Dec 18 09:13:02 truenas kernel:  </TASK>
Dec 18 09:13:02 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:14:00 truenas systemd-udevd[2122340]: nvme0n1p3: Spawned process '/lib/udev/vdev_id -d nvme0n1p3' [2122348] is taking longer than 59s to complete
Dec 18 09:14:00 truenas systemd-udevd[2122347]: nvme0n1p4: Spawned process '/lib/udev/vdev_id -d nvme0n1p4' [2122351] is taking longer than 59s to complete
Dec 18 09:14:00 truenas systemd-udevd[2122341]: nvme0n1p1: Spawned process '/lib/udev/vdev_id -d nvme0n1p1' [2122346] is taking longer than 59s to complete
Dec 18 09:14:00 truenas systemd-udevd[835]: nvme0n1p1: Worker [2122341] processing SEQNUM=3817 is taking a long time
Dec 18 09:14:00 truenas systemd-udevd[835]: nvme0n1p4: Worker [2122347] processing SEQNUM=3820 is taking a long time
Dec 18 09:14:00 truenas systemd-udevd[835]: nvme0n1p3: Worker [2122340] processing SEQNUM=3819 is taking a long time
Dec 18 09:14:00 truenas systemd-udevd[835]: nvme0n1p2: Worker [2122344] processing SEQNUM=3818 is taking a long time
Dec 18 09:14:00 truenas systemd-udevd[2122344]: nvme0n1p2: Spawned process '/lib/udev/vdev_id -d nvme0n1p2' [2122350] is taking longer than 59s to complete
Dec 18 09:15:02 truenas kernel: INFO: task asyncio_loop:2818 blocked for more than 241 seconds.
Dec 18 09:15:02 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:15:02 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:15:02 truenas kernel: task:asyncio_loop    state:D stack:    0 pid: 2818 ppid:     1 flags:0x00000000
Dec 18 09:15:02 truenas kernel: Call Trace:
Dec 18 09:15:02 truenas kernel:  <TASK>
Dec 18 09:15:02 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:15:02 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:15:02 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:15:02 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:15:02 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:15:02 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:15:02 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:15:02 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:15:02 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:15:02 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:15:02 truenas kernel:  dbuf_hold_impl+0x459/0x640 [zfs]
Dec 18 09:15:02 truenas kernel:  dbuf_hold_level+0x2b/0x60 [zfs]
Dec 18 09:15:02 truenas kernel:  dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs]
Dec 18 09:15:02 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:15:02 truenas kernel:  ? avl_add+0x46/0x90 [zavl]
Dec 18 09:15:02 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:15:02 truenas kernel:  zfs_read+0x13a/0x3d0 [zfs]
Dec 18 09:15:02 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:15:02 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:15:02 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:15:02 truenas kernel:  __x64_sys_pread64+0x91/0xc0
Dec 18 09:15:02 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:15:02 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:15:02 truenas kernel: RIP: 0033:0x7fbe246d9917
Dec 18 09:15:02 truenas kernel: RSP: 002b:00007fbe0f7fc230 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
Dec 18 09:15:02 truenas kernel: RAX: ffffffffffffffda RBX: 0000000000000010 RCX: 00007fbe246d9917
Dec 18 09:15:02 truenas kernel: RDX: 0000000000000010 RSI: 00007fbe0f7fc2c0 RDI: 0000000000000018
Dec 18 09:15:02 truenas kernel: RBP: 0000000000000010 R08: 0000000000000000 R09: 0000000001dffb88
Dec 18 09:15:02 truenas kernel: R10: 0000000000000018 R11: 0000000000000293 R12: 00007fbe0f7fc2c0
Dec 18 09:15:02 truenas kernel: R13: 0000000000000010 R14: 00007fbdfc006840 R15: 0000000000000018
Dec 18 09:15:02 truenas kernel:  </TASK>
Dec 18 09:15:02 truenas kernel: INFO: task zed:6308 blocked for more than 120 seconds.
Dec 18 09:15:02 truenas kernel:       Tainted: P           OE     5.15.79+truenas #1
Dec 18 09:15:02 truenas kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 18 09:15:02 truenas kernel: task:zed             state:D stack:    0 pid: 6308 ppid:     1 flags:0x00000000
Dec 18 09:15:02 truenas kernel: Call Trace:
Dec 18 09:15:02 truenas kernel:  <TASK>
Dec 18 09:15:02 truenas kernel:  __schedule+0x2f0/0x950
Dec 18 09:15:02 truenas kernel:  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
Dec 18 09:15:02 truenas kernel:  schedule+0x5b/0xd0
Dec 18 09:15:02 truenas kernel:  schedule_timeout+0x88/0x140
Dec 18 09:15:02 truenas kernel:  ? __bpf_trace_tick_stop+0x10/0x10
Dec 18 09:15:02 truenas kernel:  io_schedule_timeout+0x4c/0x80
Dec 18 09:15:02 truenas kernel:  __cv_timedwait_common+0x128/0x160 [spl]
Dec 18 09:15:02 truenas kernel:  ? finish_wait+0x90/0x90
Dec 18 09:15:02 truenas kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 18 09:15:02 truenas kernel:  zio_wait+0x109/0x220 [zfs]
Dec 18 09:15:02 truenas kernel:  dmu_buf_hold_array_by_dnode+0x38f/0x530 [zfs]
Dec 18 09:15:02 truenas kernel:  dmu_read_uio_dnode+0x47/0x100 [zfs]
Dec 18 09:15:02 truenas kernel:  ? avl_add+0x46/0x90 [zavl]
Dec 18 09:15:02 truenas kernel:  dmu_read_uio_dbuf+0x42/0x60 [zfs]
Dec 18 09:15:02 truenas kernel:  zfs_read+0x13a/0x3d0 [zfs]
Dec 18 09:15:02 truenas kernel:  zpl_iter_read+0xa3/0x110 [zfs]
Dec 18 09:15:02 truenas kernel:  new_sync_read+0x119/0x1b0
Dec 18 09:15:02 truenas kernel:  vfs_read+0xf6/0x190
Dec 18 09:15:02 truenas kernel:  ksys_read+0x5f/0xe0
Dec 18 09:15:02 truenas kernel:  do_syscall_64+0x3b/0xc0
Dec 18 09:15:02 truenas kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Dec 18 09:15:02 truenas kernel: RIP: 0033:0x7fbb2d120a14
Dec 18 09:15:02 truenas kernel: RSP: 002b:00007fff45d31e08 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
Dec 18 09:15:02 truenas kernel: RAX: ffffffffffffffda RBX: 00005561f72779d0 RCX: 00007fbb2d120a14
Dec 18 09:15:02 truenas kernel: RDX: 0000000000000e00 RSI: 00005561f72803a0 RDI: 0000000000000010
Dec 18 09:15:02 truenas kernel: RBP: 00007fbb2d1fb480 R08: 0000000000000010 R09: 00007fbb2d1febe0
Dec 18 09:15:02 truenas kernel: R10: 0000000000000800 R11: 0000000000000246 R12: 000000000000002c
Dec 18 09:15:02 truenas kernel: R13: 00007fbb2d1fa880 R14: 0000000000000d68 R15: 0000000000000d68
Dec 18 09:15:02 truenas kernel:  </TASK>
Dec 18 09:16:00 truenas systemd-udevd[2122340]: nvme0n1p3: Spawned process '/lib/udev/vdev_id -d nvme0n1p3' [2122348] timed out after 2min 59s, killing
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p1: Worker [2122341] processing SEQNUM=3817 killed
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p4: Worker [2122347] processing SEQNUM=3820 killed
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p3: Worker [2122340] processing SEQNUM=3819 killed
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p2: Worker [2122344] processing SEQNUM=3818 killed
Dec 18 09:16:00 truenas systemd-udevd[2122344]: nvme0n1p2: Spawned process '/lib/udev/vdev_id -d nvme0n1p2' [2122350] timed out after 2min 59s, killing
Dec 18 09:16:00 truenas systemd-udevd[835]: Worker [2122340] terminated by signal 9 (KILL)
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p3: Worker [2122340] failed
Dec 18 09:16:00 truenas systemd-udevd[835]: Worker [2122341] terminated by signal 9 (KILL)
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p1: Worker [2122341] failed
Dec 18 09:16:00 truenas systemd-udevd[835]: Worker [2122344] terminated by signal 9 (KILL)
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p2: Worker [2122344] failed
Dec 18 09:16:00 truenas systemd-udevd[835]: Worker [2122347] terminated by signal 9 (KILL)
Dec 18 09:16:00 truenas systemd-udevd[835]: nvme0n1p4: Worker [2122347] failed
Dec 18 09:17:01 truenas kernel: WARNING: Pool 'boot-pool' has encountered an uncorrectable I/O failure and has been suspended.
Dec 18 09:17:01 truenas systemd-udevd[2122349]: nvme0n1: Spawned process '/lib/udev/vdev_id -d nvme0n1' [2200328] is taking longer than 59s to complete
Dec 18 09:17:01 truenas systemd-udevd[835]: nvme0n1: Worker [2122349] processing SEQNUM=3822 is taking a long time
Dec 18 09:19:01 truenas systemd-udevd[835]: nvme0n1: Worker [2122349] processing SEQNUM=3822 killed
Dec 18 09:19:01 truenas systemd-udevd[835]: Worker [2122349] terminated by signal 9 (KILL)
Dec 18 09:19:01 truenas systemd-udevd[835]: nvme0n1: Worker [2122349] failed