INFO: task txg_sync:4873 blocked for more than 120 seconds.

[arnabiitr]

Get this error when doing resilvering or replacing a disk , not able to figure out the reason .Pasted the log from that timstamp below .Also trunsnas lost a connection for few minutes again restores it back .

Code:

[  264.770176]  sdh: sdh1
[  265.808743]  sdh: sdh1 sdh2
[ 2176.092633] INFO: task txg_sync:4873 blocked for more than 120 seconds.
[ 2176.092821]       Tainted: P           OE     5.15.131+truenas #1
[ 2176.092980] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2176.093189] task:txg_sync        state:D stack:    0 pid: 4873 ppid:     2 flags:0x00004000
[ 2176.093413] Call Trace:
[ 2176.093478]  <TASK>
[ 2176.093535]  __schedule+0x2f0/0x950
[ 2176.093632]  ? __raw_spin_unlock+0x5/0x10 [zfs]
[ 2176.094142]  schedule+0x5b/0xd0
[ 2176.094226]  cv_wait_common+0xf5/0x130 [spl]
[ 2176.094360]  ? finish_wait+0x90/0x90
[ 2176.094454]  arc_read+0x1eb/0x14b0 [zfs]
[ 2176.094820]  ? arc_evict_cb_check+0x10/0x10 [zfs]
[ 2176.095207]  ? arc_buf_fill+0x143/0x480 [zfs]
[ 2176.095614]  dsl_scan_visitbp+0x2c7/0xcd0 [zfs]
[ 2176.096086]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2176.096528]  dsl_scan_visitbp+0x5f1/0xcd0 [zfs]
[ 2176.096984]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2176.097436]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2176.097875]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2176.098308]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2176.098752]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2176.099234]  dsl_scan_visitbp+0x82b/0xcd0 [zfs]
[ 2176.102849]  dsl_scan_visit_rootbp+0xe7/0x160 [zfs]
[ 2176.106500]  dsl_scan_visitds+0x148/0x460 [zfs]
[ 2176.110169]  ? _flat_send_IPI_mask+0x21/0x40
[ 2176.113563]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2176.116961]  ? try_to_wake_up+0x34b/0x500
[ 2176.120334]  ? __cond_resched+0x16/0x50
[ 2176.123764]  ? __cond_resched+0x16/0x50
[ 2176.127157]  ? __kmalloc_node+0x68/0x490
[ 2176.130542]  ? __raw_spin_unlock+0x5/0x10 [spl]
[ 2176.134003]  ? tsd_hash_search+0x75/0xa0 [spl]
[ 2176.137476]  ? __list_add+0x12/0x30 [spl]
[ 2176.140971]  ? __raw_spin_unlock+0x5/0x10 [spl]
[ 2176.144481]  ? tsd_hash_add+0x122/0x150 [spl]
[ 2176.148049]  dsl_scan_visit+0x1a7/0x2c0 [zfs]
[ 2176.151931]  dsl_scan_sync+0x444/0x980 [zfs]
[ 2176.155805]  ? kfree+0x1fc/0x250
[ 2176.159374]  spa_sync_iterate_to_convergence+0x11f/0x1e0 [zfs]
[ 2176.163432]  spa_sync+0x2eb/0x5d0 [zfs]
[ 2176.167467]  txg_sync_thread+0x229/0x2a0 [zfs]
[ 2176.171542]  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 2176.175547]  thread_generic_wrapper+0x59/0x70 [spl]
[ 2176.179349]  ? __thread_exit+0x20/0x20 [spl]
[ 2176.183194]  kthread+0x127/0x150
[ 2176.187052]  ? set_kthread_struct+0x50/0x50
[ 2176.190988]  ret_from_fork+0x22/0x30
[ 2176.194800]  </TASK>
[ 2659.420689] INFO: task txg_sync:4873 blocked for more than 120 seconds.
[ 2659.423841]       Tainted: P           OE     5.15.131+truenas #1
[ 2659.427097] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2659.430502] task:txg_sync        state:D stack:    0 pid: 4873 ppid:     2 flags:0x00004000
[ 2659.434054] Call Trace:
[ 2659.437494]  <TASK>
[ 2659.440866]  __schedule+0x2f0/0x950
[ 2659.444250]  schedule+0x5b/0xd0
[ 2659.447622]  cv_wait_common+0xf5/0x130 [spl]
[ 2659.451106]  ? finish_wait+0x90/0x90
[ 2659.454616]  arc_read+0x1eb/0x14b0 [zfs]
[ 2659.458496]  ? arc_evict_cb_check+0x10/0x10 [zfs]
[ 2659.462420]  ? spl_kmem_cache_free+0xbc/0x120 [spl]
[ 2659.466200]  dsl_scan_visitbp+0x2c7/0xcd0 [zfs]
[ 2659.470348]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2659.474505]  dsl_scan_visitbp+0x5f1/0xcd0 [zfs]
[ 2659.478673]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2659.482782]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2659.486835]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2659.490806]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2659.494744]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2659.498608]  dsl_scan_visitbp+0x82b/0xcd0 [zfs]
[ 2659.502435]  dsl_scan_visit_rootbp+0xe7/0x160 [zfs]
[ 2659.506217]  dsl_scan_visitds+0x148/0x460 [zfs]
[ 2659.510005]  ? _flat_send_IPI_mask+0x21/0x40
[ 2659.513576]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2659.517117]  ? try_to_wake_up+0x34b/0x500
[ 2659.520649]  ? __cond_resched+0x16/0x50
[ 2659.524155]  ? __cond_resched+0x16/0x50
[ 2659.527619]  ? __kmalloc_node+0x68/0x490
[ 2659.531057]  ? __raw_spin_unlock+0x5/0x10 [spl]
[ 2659.534517]  ? tsd_hash_search+0x75/0xa0 [spl]
[ 2659.538000]  ? __list_add+0x12/0x30 [spl]
[ 2659.541472]  ? __raw_spin_unlock+0x5/0x10 [spl]
[ 2659.544975]  ? tsd_hash_add+0x122/0x150 [spl]
[ 2659.548486]  dsl_scan_visit+0x1a7/0x2c0 [zfs]
[ 2659.552129]  dsl_scan_sync+0x444/0x980 [zfs]
[ 2659.555789]  ? kfree+0x1fc/0x250
[ 2659.559322]  spa_sync_iterate_to_convergence+0x11f/0x1e0 [zfs]
[ 2659.563133]  spa_sync+0x2eb/0x5d0 [zfs]
[ 2659.566960]  txg_sync_thread+0x229/0x2a0 [zfs]
[ 2659.570803]  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 2659.574686]  thread_generic_wrapper+0x59/0x70 [spl]
[ 2659.578480]  ? __thread_exit+0x20/0x20 [spl]
[ 2659.582324]  kthread+0x127/0x150
[ 2659.586156]  ? set_kthread_struct+0x50/0x50
[ 2659.590085]  ret_from_fork+0x22/0x30
[ 2659.594051]  </TASK>
[ 2901.084715] INFO: task txg_sync:4873 blocked for more than 120 seconds.
[ 2901.087885]       Tainted: P           OE     5.15.131+truenas #1
[ 2901.091209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 2901.094666] task:txg_sync        state:D stack:    0 pid: 4873 ppid:     2 flags:0x00004000
[ 2901.098227] Call Trace:
[ 2901.101709]  <TASK>
[ 2901.105132]  __schedule+0x2f0/0x950
[ 2901.108559]  ? __raw_spin_unlock+0x5/0x10 [zfs]
[ 2901.112413]  schedule+0x5b/0xd0
[ 2901.115905]  cv_wait_common+0xf5/0x130 [spl]
[ 2901.119515]  ? finish_wait+0x90/0x90
[ 2901.123148]  arc_read+0x1eb/0x14b0 [zfs]
[ 2901.127116]  ? arc_evict_cb_check+0x10/0x10 [zfs]
[ 2901.131183]  dsl_scan_visitbp+0x524/0xcd0 [zfs]
[ 2901.135370]  ? dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2901.139644]  ? kfree+0x220/0x250
[ 2901.143645]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2901.147996]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2901.152294]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2901.156511]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2901.160691]  dsl_scan_visitbp+0x35c/0xcd0 [zfs]
[ 2901.164795]  dsl_scan_visitbp+0x82b/0xcd0 [zfs]
[ 2901.168776]  dsl_scan_visit_rootbp+0xe7/0x160 [zfs]
[ 2901.172793]  dsl_scan_visitds+0x148/0x460 [zfs]
[ 2901.176757]  ? _flat_send_IPI_mask+0x21/0x40
[ 2901.180541]  ? ttwu_queue_wakelist+0xf2/0x110
[ 2901.184328]  ? try_to_wake_up+0x34b/0x500
[ 2901.188096]  ? __cond_resched+0x16/0x50
[ 2901.191842]  ? __cond_resched+0x16/0x50
[ 2901.195516]  ? __kmalloc_node+0x68/0x490
[ 2901.199184]  ? __raw_spin_unlock+0x5/0x10 [spl]
[ 2901.202886]  ? tsd_hash_search+0x75/0xa0 [spl]
[ 2901.206602]  ? __list_add+0x12/0x30 [spl]
[ 2901.210311]  ? __raw_spin_unlock+0x5/0x10 [spl]
[ 2901.214057]  ? tsd_hash_add+0x122/0x150 [spl]
[ 2901.217811]  dsl_scan_visit+0x1a7/0x2c0 [zfs]
[ 2901.221687]  dsl_scan_sync+0x444/0x980 [zfs]
[ 2901.225579]  ? kfree+0x1fc/0x250
[ 2901.229351]  spa_sync_iterate_to_convergence+0x11f/0x1e0 [zfs]
[ 2901.233393]  spa_sync+0x2eb/0x5d0 [zfs]
[ 2901.237473]  txg_sync_thread+0x229/0x2a0 [zfs]
[ 2901.241543]  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[ 2901.245654]  thread_generic_wrapper+0x59/0x70 [spl]
[ 2901.249689]  ? __thread_exit+0x20/0x20 [spl]
[ 2901.253743]  kthread+0x127/0x150
[ 2901.257815]  ? set_kthread_struct+0x50/0x50
[ 2901.261978]  ret_from_fork+0x22/0x30
[ 2901.266179]  </TASK>
[ 5438.556894] INFO: task z_rd_iss:3533 blocked for more than 120 seconds.
[ 5438.560288]       Tainted: P           OE     5.15.131+truenas #1
[ 5438.563797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.567414] task:z_rd_iss        state:D stack:    0 pid: 3533 ppid:     2 flags:0x00004000
[ 5438.571156] Call Trace:
[ 5438.574811]  <TASK>
[ 5438.578416]  __schedule+0x2f0/0x950
[ 5438.582030]  schedule+0x5b/0xd0
[ 5438.585602]  schedule_preempt_disabled+0xa/0x10
[ 5438.589191]  __mutex_lock.constprop.0+0x210/0x400
[ 5438.592789]  ? ktime_get_raw_ts64+0x3e/0xe0
[ 5438.596388]  vdev_queue_io+0x87/0x190 [zfs]
[ 5438.600267]  zio_vdev_io_start+0x96/0x2b0 [zfs]
[ 5438.604157]  zio_nowait+0x97/0x120 [zfs]
[ 5438.608097]  vdev_raidz_io_start_read+0x129/0x1b0 [zfs]
[ 5438.612157]  vdev_raidz_io_start+0x4d/0x70 [zfs]
[ 5438.616242]  zio_vdev_io_start+0xd5/0x2b0 [zfs]
[ 5438.620366]  zio_nowait+0x97/0x120 [zfs]
[ 5438.624481]  vdev_mirror_io_start+0xa2/0x230 [zfs]
[ 5438.628658]  zio_vdev_io_start+0x244/0x2b0 [zfs]
[ 5438.632833]  zio_execute+0x90/0xe0 [zfs]
[ 5438.636987]  taskq_thread+0x1ff/0x3c0 [spl]
[ 5438.641058]  ? wake_up_q+0x90/0x90
[ 5438.645150]  ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs]
[ 5438.649553]  ? taskq_thread_spawn+0x60/0x60 [spl]
[ 5438.653813]  kthread+0x127/0x150
[ 5438.658040]  ? set_kthread_struct+0x50/0x50
[ 5438.662281]  ret_from_fork+0x22/0x30
[ 5438.666524]  </TASK>
[ 5438.880876] INFO: task nmbd:8056 blocked for more than 121 seconds.
[ 5438.885251]       Tainted: P           OE     5.15.131+truenas #1
[ 5438.889669] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5438.894221] task:nmbd            state:D stack:    0 pid: 8056 ppid:     1 flags:0x00000000
[ 5438.898915] Call Trace:
[ 5438.903521]  <TASK>
[ 5438.908102]  __schedule+0x2f0/0x950
[ 5438.912723]  ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs]
[ 5438.917667]  schedule+0x5b/0xd0
[ 5438.922419]  schedule_preempt_disabled+0xa/0x10
[ 5438.927250]  __mutex_lock.constprop.0+0x210/0x400
[ 5438.932157]  ? __cond_resched+0x16/0x50
[ 5438.937040]  arc_buf_access+0x7f/0x200 [zfs]
[ 5438.942105]  dbuf_hold_impl+0xa1/0x640 [zfs]
[ 5438.947080]  dbuf_hold_level+0x2b/0x60 [zfs]
[ 5438.952074]  dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs]
[ 5438.957157]  dmu_read_impl+0xa8/0x180 [zfs]
[ 5438.962284]  dmu_read+0x56/0xa0 [zfs]
[ 5438.967434]  zfs_fillpage+0x7a/0x180 [zfs]
[ 5438.972493]  zfs_getpage+0x50/0xf0 [zfs]
[ 5438.977418]  zpl_readpage_common+0x29/0x50 [zfs]
[ 5438.982265]  filemap_read_page+0x36/0xf0
[ 5438.986862]  filemap_fault+0x98d/0xa80
[ 5438.991357]  __do_fault+0x36/0x120
[ 5438.995862]  __handle_mm_fault+0xce7/0x1580
[ 5439.000423]  handle_mm_fault+0xcf/0x2b0
[ 5439.004993]  do_user_addr_fault+0x1be/0x670
[ 5439.009585]  exc_page_fault+0x72/0x150
[ 5439.014130]  asm_exc_page_fault+0x22/0x30
[ 5439.018679] RIP: 0033:0x7f4b100420e0
[ 5439.023217] RSP: 002b:00007fffa867d798 EFLAGS: 00010206
[ 5439.027809] RAX: 0000000000000000 RBX: 00007fffa867d7fc RCX: 0000000000000000
[ 5439.032490] RDX: 0000000000000240 RSI: 00007fffa867d880 RDI: 000000000000000f
[ 5439.037114] RBP: 00007fffa867d800 R08: 00007fffa867d800 R09: 00007fffa867d7fc
[ 5439.041772] R10: 0000000000002710 R11: 0000000000000246 R12: 0000000000000000
[ 5439.046443] R13: 0000000000000240 R14: 00007fffa867d880 R15: 000000000000000f
[ 5439.051084]  </TASK>
[ 5439.212867] INFO: task smbd:18590 blocked for more than 121 seconds.
[ 5439.217592]       Tainted: P           OE     5.15.131+truenas #1
[ 5439.222353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5439.227244] task:smbd            state:D stack:    0 pid:18590 ppid:  6000 flags:0x00000000
[ 5439.232260] Call Trace:
[ 5439.237141]  <TASK>
[ 5439.241913]  __schedule+0x2f0/0x950
[ 5439.246705]  ? __dev_queue_xmit+0x350/0xb20
[ 5439.251509]  schedule+0x5b/0xd0
[ 5439.256282]  schedule_preempt_disabled+0xa/0x10
[ 5439.261102]  __mutex_lock.constprop.0+0x210/0x400
[ 5439.265895]  dbuf_find+0xb1/0x190 [zfs]
[ 5439.270791]  ? gethrtime+0x1c/0x50 [zfs]
[ 5439.275548]  dbuf_hold_impl+0x60/0x640 [zfs]
[ 5439.280173]  dbuf_hold_level+0x2b/0x60 [zfs]
[ 5439.284706]  dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs]
[ 5439.289356]  dmu_read_impl+0xa8/0x180 [zfs]
[ 5439.294010]  dmu_read+0x56/0xa0 [zfs]
[ 5439.298596]  zfs_fillpage+0x7a/0x180 [zfs]
[ 5439.303159]  zfs_getpage+0x50/0xf0 [zfs]
[ 5439.307748]  zpl_readpage_common+0x29/0x50 [zfs]
[ 5439.312332]  filemap_read_page+0x36/0xf0
[ 5439.316737]  filemap_fault+0x98d/0xa80
[ 5439.321125]  __do_fault+0x36/0x120
[ 5439.325530]  __handle_mm_fault+0xce7/0x1580
[ 5439.329955]  handle_mm_fault+0xcf/0x2b0
[ 5439.334366]  do_user_addr_fault+0x1be/0x670
[ 5439.338749]  exc_page_fault+0x72/0x150
[ 5439.343161]  asm_exc_page_fault+0x22/0x30
[ 5439.347582] RIP: 0033:0x7f084bd580f0
[ 5439.351985] RSP: 002b:00007fffe43f20b8 EFLAGS: 00010206
[ 5439.356463] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000086e9e
[ 5439.361013] RDX: 00007f084bdae998 RSI: 00007f084bdaecb3 RDI: 000055c5b9d8ff00
[ 5439.365601] RBP: 000055c5b9e52030 R08: 000055c5b9d8ff00 R09: 00007f084ba07be0
[ 5439.370243] R10: 000055c5b9d80110 R11: 0000000000000000 R12: 000055c5b9d80110
[ 5439.374895] R13: 000055c5b9e52058 R14: 0000000065a811ce R15: 0000000000086e9e
[ 5439.379534]  </TASK>
[ 5439.720862] INFO: task smbd:1168290 blocked for more than 121 seconds.
[ 5439.725603]       Tainted: P           OE     5.15.131+truenas #1
[ 5439.730406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5439.735342] task:smbd            state:D stack:    0 pid:1168290 ppid:  6000 flags:0x00000000
[ 5439.740421] Call Trace:
[ 5439.745336]  <TASK>
[ 5439.750239]  __schedule+0x2f0/0x950
[ 5439.755159]  schedule+0x5b/0xd0
[ 5439.760019]  schedule_preempt_disabled+0xa/0x10
[ 5439.764941]  __mutex_lock.constprop.0+0x210/0x400
[ 5439.769797]  dbuf_find+0x4e/0x190 [zfs]
[ 5439.774789]  ? zio_create+0x42c/0x4e0 [zfs]
[ 5439.779686]  dbuf_hold_impl+0x60/0x640 [zfs]
[ 5439.784454]  dbuf_hold_level+0x2b/0x60 [zfs]
[ 5439.789073]  dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs]
[ 5439.793684]  dmu_read_impl+0xa8/0x180 [zfs]
[ 5439.798344]  dmu_read+0x56/0xa0 [zfs]
[ 5439.802973]  zfs_fillpage+0x7a/0x180 [zfs]
[ 5439.807643]  zfs_getpage+0x50/0xf0 [zfs]
[ 5439.812297]  zpl_readpage_common+0x29/0x50 [zfs]
[ 5439.816971]  filemap_read_page+0x36/0xf0
[ 5439.821494]  filemap_fault+0x98d/0xa80
[ 5439.825987]  __do_fault+0x36/0x120
[ 5439.830437]  __handle_mm_fault+0xce7/0x1580
[ 5439.834922]  handle_mm_fault+0xcf/0x2b0
[ 5439.839433]  do_user_addr_fault+0x1be/0x670
[ 5439.843911]  exc_page_fault+0x72/0x150
[ 5439.848377]  asm_exc_page_fault+0x22/0x30
[ 5439.852863] RIP: 0033:0x7f084bd6edb0
[ 5439.857330] RSP: 002b:00007fffe43f1678 EFLAGS: 00010206
[ 5439.861869] RAX: 00000000000001bb RBX: 000055c5b9d9f5e0 RCX: 00007fffe43f1770
[ 5439.866518] RDX: 00000000000001bb RSI: 000055c5b9e33420 RDI: 00007fffe43f16a0
[ 5439.871117] RBP: 00007fffe43f1ac0 R08: 0000000000002717 R09: 0000000000000000
[ 5439.875772] R10: 1214b52100000000 R11: 1214b520ad495bad R12: 0000000000000018
[ 5439.880439] R13: 000055c5b9e33420 R14: 00007f084bd6edb0 R15: 00007fffe43f1770
[ 5439.885169]  </TASK>
[ 5440.120865] INFO: task z_rd_iss:1336082 blocked for more than 122 seconds.
[ 5440.125728]       Tainted: P           OE     5.15.131+truenas #1
[ 5440.130641] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5440.135691] task:z_rd_iss        state:D stack:    0 pid:1336082 ppid:     2 flags:0x00004000
[ 5440.140887] Call Trace:
[ 5440.145951]  <TASK>
[ 5440.150930]  __schedule+0x2f0/0x950
[ 5440.155911]  schedule+0x5b/0xd0
[ 5440.160858]  schedule_preempt_disabled+0xa/0x10
[ 5440.165801]  __mutex_lock.constprop.0+0x210/0x400
[ 5440.170771]  ? ktime_get_raw_ts64+0x3e/0xe0
[ 5440.175662]  vdev_queue_io+0x87/0x190 [zfs]
[ 5440.180709]  zio_vdev_io_start+0x96/0x2b0 [zfs]
[ 5440.185626]  zio_nowait+0x97/0x120 [zfs]
[ 5440.190446]  vdev_raidz_io_start_read+0x129/0x1b0 [zfs]
[ 5440.195207]  vdev_raidz_io_start+0x4d/0x70 [zfs]
[ 5440.199876]  zio_vdev_io_start+0xd5/0x2b0 [zfs]
[ 5440.204601]  zio_nowait+0x97/0x120 [zfs]
[ 5440.209327]  vdev_mirror_io_start+0xa2/0x230 [zfs]
[ 5440.214109]  zio_vdev_io_start+0x244/0x2b0 [zfs]
[ 5440.218926]  zio_execute+0x90/0xe0 [zfs]
[ 5440.223701]  taskq_thread+0x1ff/0x3c0 [spl]
[ 5440.228321]  ? wake_up_q+0x90/0x90
[ 5440.232927]  ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs]
[ 5440.237803]  ? taskq_thread_spawn+0x60/0x60 [spl]
[ 5440.242575]  kthread+0x127/0x150
[ 5440.247292]  ? set_kthread_struct+0x50/0x50
[ 5440.252015]  ret_from_fork+0x22/0x30
[ 5440.256749]  </TASK>
[ 5440.572487] INFO: task z_rd_iss:1336114 blocked for more than 122 seconds.
[ 5440.577398]       Tainted: P           OE     5.15.131+truenas #1
[ 5440.582323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5440.587378] task:z_rd_iss        state:D stack:    0 pid:1336114 ppid:     2 flags:0x00004000
[ 5440.592588] Call Trace:
[ 5440.597720]  <TASK>
[ 5440.602801]  __schedule+0x2f0/0x950
[ 5440.607923]  schedule+0x5b/0xd0
[ 5440.613020]  schedule_preempt_disabled+0xa/0x10
[ 5440.618119]  __mutex_lock.constprop.0+0x210/0x400
[ 5440.623250]  ? ktime_get_raw_ts64+0x3e/0xe0
[ 5440.628379]  vdev_queue_io+0x87/0x190 [zfs]
[ 5440.633715]  zio_vdev_io_start+0x96/0x2b0 [zfs]
[ 5440.638940]  zio_nowait+0x97/0x120 [zfs]
[ 5440.644185]  vdev_raidz_io_start_read+0x129/0x1b0 [zfs]
[ 5440.649482]  vdev_raidz_io_start+0x4d/0x70 [zfs]
[ 5440.654697]  zio_vdev_io_start+0xd5/0x2b0 [zfs]
[ 5440.659891]  zio_nowait+0x97/0x120 [zfs]
[ 5440.664991]  vdev_mirror_io_start+0xa2/0x230 [zfs]
[ 5440.670061]  zio_vdev_io_start+0x244/0x2b0 [zfs]
[ 5440.675067]  zio_execute+0x90/0xe0 [zfs]
[ 5440.679967]  taskq_thread+0x1ff/0x3c0 [spl]
[ 5440.684614]  ? wake_up_q+0x90/0x90
[ 5440.689181]  ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs]
[ 5440.694059]  ? taskq_thread_spawn+0x60/0x60 [spl]
[ 5440.698819]  kthread+0x127/0x150
[ 5440.703582]  ? set_kthread_struct+0x50/0x50
[ 5440.708364]  ret_from_fork+0x22/0x30
[ 5440.713122]  </TASK>
[ 5440.980849] INFO: task z_rd_iss:1336116 blocked for more than 123 seconds.
[ 5440.985740]       Tainted: P           OE     5.15.131+truenas #1
[ 5440.990662] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 5440.995720] task:z_rd_iss        state:D stack:    0 pid:1336116 ppid:     2 flags:0x00004000
[ 5441.000905] Call Trace:
[ 5441.006003]  <TASK>
[ 5441.011040]  __schedule+0x2f0/0x950
[ 5441.016090]  schedule+0x5b/0xd0
[ 5441.021093]  schedule_preempt_disabled+0xa/0x10
[ 5441.026107]  __mutex_lock.constprop.0+0x210/0x400
[ 5441.031142]  ? ktime_get_raw_ts64+0x3e/0xe0
[ 5441.036174]  vdev_queue_io+0x87/0x190 [zfs]
[ 5441.041402]  zio_vdev_io_start+0x96/0x2b0 [zfs]
[ 5441.046535]  zio_nowait+0x97/0x120 [zfs]
[ 5441.051604]  vdev_raidz_io_start_read+0x129/0x1b0 [zfs]
[ 5441.056724]  vdev_raidz_io_start+0x4d/0x70 [zfs]
[ 5441.061812]  zio_vdev_io_start+0xd5/0x2b0 [zfs]
[ 5441.066903]  zio_nowait+0x97/0x120 [zfs]
[ 5441.071983]  vdev_mirror_io_start+0xa2/0x230 [zfs]
[ 5441.077035]  zio_vdev_io_start+0x244/0x2b0 [zfs]
[ 5441.082050]  zio_execute+0x90/0xe0 [zfs]
[ 5441.086937]  taskq_thread+0x1ff/0x3c0 [spl]
[ 5441.091599]  ? wake_up_q+0x90/0x90
[ 5441.096157]  ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs]
[ 5441.101051]  ? taskq_thread_spawn+0x60/0x60 [spl]
[ 5441.105803]  kthread+0x127/0x150
[ 5441.110530]  ? set_kthread_struct+0x50/0x50
[ 5441.115308]  ret_from_fork+0x22/0x30
[ 5441.120060]  </TASK>
[ 5533.937060] alx 0000:02:00.0 enp2s0: Link Down
[ 5579.677910] alx 0000:02:00.0 enp2s0: NIC Up: 1 Gbps Full
[ 5602.524712] systemd-coredump[1336127]: Process 5439 (systemd-journal) of user 0 dumped core.
[ 5602.527744] systemd-coredump[1336127]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.5c4b896f2ba641f5b6ca67b21c8f41d0.5439.1705513759000000.zst
[ 5602.534348] systemd-coredump[1336127]: Stack trace of thread 5439:
[ 5602.537878] systemd-coredump[1336127]: #0  0x00007f18a74bee5b journal_file_find_data_object_with_hash (libsystemd-shared-252.so + 0x216e5b)
[ 5602.545516] systemd-coredump[1336127]: #1  0x00007f18a74bf180 n/a (libsystemd-shared-252.so + 0x217180)
[ 5602.549743] systemd-coredump[1336127]: #2  0x00007f18a74bfb67 journal_file_append_entry (libsystemd-shared-252.so + 0x217b67)
[ 5602.554097] systemd-coredump[1336127]: #3  0x0000563f263f5871 n/a (systemd-journald + 0x12871)
[ 5602.558521] systemd-coredump[1336127]: #4  0x0000563f263f6eed n/a (systemd-journald + 0x13eed)
[ 5602.562954] systemd-coredump[1336127]: #5  0x00007f18a74e45f0 n/a (libsystemd-shared-252.so + 0x23c5f0)
[ 5602.567449] systemd-coredump[1336127]: #6  0x00007f18a74e48fd sd_event_dispatch (libsystemd-shared-252.so + 0x23c8fd)
[ 5602.572043] systemd-coredump[1336127]: #7  0x00007f18a74e74d0 sd_event_run (libsystemd-shared-252.so + 0x23f4d0)
[ 5602.576665] systemd-coredump[1336127]: #8  0x0000563f263eb54d n/a (systemd-journald + 0x854d)
[ 5602.581455] systemd-coredump[1336127]: #9  0x00007f18a708bd0a __libc_start_main (libc.so.6 + 0x23d0a)
[ 5602.586263] systemd-coredump[1336127]: #10 0x0000563f263ebc3a n/a (systemd-journald + 0x8c3a)
[ 5602.591228] systemd-coredump[1336127]: ELF object binary architecture: AMD x86-64
[ 5602.600408] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT
[ 5602.605655] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[ 5602.611538] systemd[1]: systemd-journald.service: Consumed 2min 36.733s CPU time.
[ 5602.618369] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[ 5602.624003] systemd[1]: Stopped systemd-journald.service - Journal Service.
[ 5602.629440] systemd[1]: systemd-journald.service: Consumed 2min 36.733s CPU time.
[ 5603.406345] systemd[1]: Starting systemd-journald.service - Journal Service...
[ 5603.547481] systemd-journald[1343717]: File /run/log/journal/7f241b3ec338497fbb5c906cca394ff9/system.journal corrupted or uncleanly shut down, renaming and replacing.