arnabiitr
Cadet
- Joined
- Oct 23, 2017
- Messages
- 2
Get this error when doing resilvering or replacing a disk , not able to figure out the reason .Pasted the log from that timstamp below .Also trunsnas lost a connection for few minutes again restores it back .
Code:
[ 264.770176] sdh: sdh1 [ 265.808743] sdh: sdh1 sdh2 [ 2176.092633] INFO: task txg_sync:4873 blocked for more than 120 seconds. [ 2176.092821] Tainted: P OE 5.15.131+truenas #1 [ 2176.092980] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2176.093189] task:txg_sync state:D stack: 0 pid: 4873 ppid: 2 flags:0x00004000 [ 2176.093413] Call Trace: [ 2176.093478] <TASK> [ 2176.093535] __schedule+0x2f0/0x950 [ 2176.093632] ? __raw_spin_unlock+0x5/0x10 [zfs] [ 2176.094142] schedule+0x5b/0xd0 [ 2176.094226] cv_wait_common+0xf5/0x130 [spl] [ 2176.094360] ? finish_wait+0x90/0x90 [ 2176.094454] arc_read+0x1eb/0x14b0 [zfs] [ 2176.094820] ? arc_evict_cb_check+0x10/0x10 [zfs] [ 2176.095207] ? arc_buf_fill+0x143/0x480 [zfs] [ 2176.095614] dsl_scan_visitbp+0x2c7/0xcd0 [zfs] [ 2176.096086] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2176.096528] dsl_scan_visitbp+0x5f1/0xcd0 [zfs] [ 2176.096984] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2176.097436] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2176.097875] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2176.098308] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2176.098752] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2176.099234] dsl_scan_visitbp+0x82b/0xcd0 [zfs] [ 2176.102849] dsl_scan_visit_rootbp+0xe7/0x160 [zfs] [ 2176.106500] dsl_scan_visitds+0x148/0x460 [zfs] [ 2176.110169] ? _flat_send_IPI_mask+0x21/0x40 [ 2176.113563] ? ttwu_queue_wakelist+0xf2/0x110 [ 2176.116961] ? try_to_wake_up+0x34b/0x500 [ 2176.120334] ? __cond_resched+0x16/0x50 [ 2176.123764] ? __cond_resched+0x16/0x50 [ 2176.127157] ? __kmalloc_node+0x68/0x490 [ 2176.130542] ? __raw_spin_unlock+0x5/0x10 [spl] [ 2176.134003] ? tsd_hash_search+0x75/0xa0 [spl] [ 2176.137476] ? __list_add+0x12/0x30 [spl] [ 2176.140971] ? __raw_spin_unlock+0x5/0x10 [spl] [ 2176.144481] ? tsd_hash_add+0x122/0x150 [spl] [ 2176.148049] dsl_scan_visit+0x1a7/0x2c0 [zfs] [ 2176.151931] dsl_scan_sync+0x444/0x980 [zfs] [ 2176.155805] ? kfree+0x1fc/0x250 [ 2176.159374] spa_sync_iterate_to_convergence+0x11f/0x1e0 [zfs] [ 2176.163432] spa_sync+0x2eb/0x5d0 [zfs] [ 2176.167467] txg_sync_thread+0x229/0x2a0 [zfs] [ 2176.171542] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs] [ 2176.175547] thread_generic_wrapper+0x59/0x70 [spl] [ 2176.179349] ? __thread_exit+0x20/0x20 [spl] [ 2176.183194] kthread+0x127/0x150 [ 2176.187052] ? set_kthread_struct+0x50/0x50 [ 2176.190988] ret_from_fork+0x22/0x30 [ 2176.194800] </TASK> [ 2659.420689] INFO: task txg_sync:4873 blocked for more than 120 seconds. [ 2659.423841] Tainted: P OE 5.15.131+truenas #1 [ 2659.427097] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2659.430502] task:txg_sync state:D stack: 0 pid: 4873 ppid: 2 flags:0x00004000 [ 2659.434054] Call Trace: [ 2659.437494] <TASK> [ 2659.440866] __schedule+0x2f0/0x950 [ 2659.444250] schedule+0x5b/0xd0 [ 2659.447622] cv_wait_common+0xf5/0x130 [spl] [ 2659.451106] ? finish_wait+0x90/0x90 [ 2659.454616] arc_read+0x1eb/0x14b0 [zfs] [ 2659.458496] ? arc_evict_cb_check+0x10/0x10 [zfs] [ 2659.462420] ? spl_kmem_cache_free+0xbc/0x120 [spl] [ 2659.466200] dsl_scan_visitbp+0x2c7/0xcd0 [zfs] [ 2659.470348] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2659.474505] dsl_scan_visitbp+0x5f1/0xcd0 [zfs] [ 2659.478673] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2659.482782] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2659.486835] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2659.490806] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2659.494744] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2659.498608] dsl_scan_visitbp+0x82b/0xcd0 [zfs] [ 2659.502435] dsl_scan_visit_rootbp+0xe7/0x160 [zfs] [ 2659.506217] dsl_scan_visitds+0x148/0x460 [zfs] [ 2659.510005] ? _flat_send_IPI_mask+0x21/0x40 [ 2659.513576] ? ttwu_queue_wakelist+0xf2/0x110 [ 2659.517117] ? try_to_wake_up+0x34b/0x500 [ 2659.520649] ? __cond_resched+0x16/0x50 [ 2659.524155] ? __cond_resched+0x16/0x50 [ 2659.527619] ? __kmalloc_node+0x68/0x490 [ 2659.531057] ? __raw_spin_unlock+0x5/0x10 [spl] [ 2659.534517] ? tsd_hash_search+0x75/0xa0 [spl] [ 2659.538000] ? __list_add+0x12/0x30 [spl] [ 2659.541472] ? __raw_spin_unlock+0x5/0x10 [spl] [ 2659.544975] ? tsd_hash_add+0x122/0x150 [spl] [ 2659.548486] dsl_scan_visit+0x1a7/0x2c0 [zfs] [ 2659.552129] dsl_scan_sync+0x444/0x980 [zfs] [ 2659.555789] ? kfree+0x1fc/0x250 [ 2659.559322] spa_sync_iterate_to_convergence+0x11f/0x1e0 [zfs] [ 2659.563133] spa_sync+0x2eb/0x5d0 [zfs] [ 2659.566960] txg_sync_thread+0x229/0x2a0 [zfs] [ 2659.570803] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs] [ 2659.574686] thread_generic_wrapper+0x59/0x70 [spl] [ 2659.578480] ? __thread_exit+0x20/0x20 [spl] [ 2659.582324] kthread+0x127/0x150 [ 2659.586156] ? set_kthread_struct+0x50/0x50 [ 2659.590085] ret_from_fork+0x22/0x30 [ 2659.594051] </TASK> [ 2901.084715] INFO: task txg_sync:4873 blocked for more than 120 seconds. [ 2901.087885] Tainted: P OE 5.15.131+truenas #1 [ 2901.091209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 2901.094666] task:txg_sync state:D stack: 0 pid: 4873 ppid: 2 flags:0x00004000 [ 2901.098227] Call Trace: [ 2901.101709] <TASK> [ 2901.105132] __schedule+0x2f0/0x950 [ 2901.108559] ? __raw_spin_unlock+0x5/0x10 [zfs] [ 2901.112413] schedule+0x5b/0xd0 [ 2901.115905] cv_wait_common+0xf5/0x130 [spl] [ 2901.119515] ? finish_wait+0x90/0x90 [ 2901.123148] arc_read+0x1eb/0x14b0 [zfs] [ 2901.127116] ? arc_evict_cb_check+0x10/0x10 [zfs] [ 2901.131183] dsl_scan_visitbp+0x524/0xcd0 [zfs] [ 2901.135370] ? dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2901.139644] ? kfree+0x220/0x250 [ 2901.143645] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2901.147996] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2901.152294] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2901.156511] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2901.160691] dsl_scan_visitbp+0x35c/0xcd0 [zfs] [ 2901.164795] dsl_scan_visitbp+0x82b/0xcd0 [zfs] [ 2901.168776] dsl_scan_visit_rootbp+0xe7/0x160 [zfs] [ 2901.172793] dsl_scan_visitds+0x148/0x460 [zfs] [ 2901.176757] ? _flat_send_IPI_mask+0x21/0x40 [ 2901.180541] ? ttwu_queue_wakelist+0xf2/0x110 [ 2901.184328] ? try_to_wake_up+0x34b/0x500 [ 2901.188096] ? __cond_resched+0x16/0x50 [ 2901.191842] ? __cond_resched+0x16/0x50 [ 2901.195516] ? __kmalloc_node+0x68/0x490 [ 2901.199184] ? __raw_spin_unlock+0x5/0x10 [spl] [ 2901.202886] ? tsd_hash_search+0x75/0xa0 [spl] [ 2901.206602] ? __list_add+0x12/0x30 [spl] [ 2901.210311] ? __raw_spin_unlock+0x5/0x10 [spl] [ 2901.214057] ? tsd_hash_add+0x122/0x150 [spl] [ 2901.217811] dsl_scan_visit+0x1a7/0x2c0 [zfs] [ 2901.221687] dsl_scan_sync+0x444/0x980 [zfs] [ 2901.225579] ? kfree+0x1fc/0x250 [ 2901.229351] spa_sync_iterate_to_convergence+0x11f/0x1e0 [zfs] [ 2901.233393] spa_sync+0x2eb/0x5d0 [zfs] [ 2901.237473] txg_sync_thread+0x229/0x2a0 [zfs] [ 2901.241543] ? txg_dispatch_callbacks+0xf0/0xf0 [zfs] [ 2901.245654] thread_generic_wrapper+0x59/0x70 [spl] [ 2901.249689] ? __thread_exit+0x20/0x20 [spl] [ 2901.253743] kthread+0x127/0x150 [ 2901.257815] ? set_kthread_struct+0x50/0x50 [ 2901.261978] ret_from_fork+0x22/0x30 [ 2901.266179] </TASK> [ 5438.556894] INFO: task z_rd_iss:3533 blocked for more than 120 seconds. [ 5438.560288] Tainted: P OE 5.15.131+truenas #1 [ 5438.563797] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5438.567414] task:z_rd_iss state:D stack: 0 pid: 3533 ppid: 2 flags:0x00004000 [ 5438.571156] Call Trace: [ 5438.574811] <TASK> [ 5438.578416] __schedule+0x2f0/0x950 [ 5438.582030] schedule+0x5b/0xd0 [ 5438.585602] schedule_preempt_disabled+0xa/0x10 [ 5438.589191] __mutex_lock.constprop.0+0x210/0x400 [ 5438.592789] ? ktime_get_raw_ts64+0x3e/0xe0 [ 5438.596388] vdev_queue_io+0x87/0x190 [zfs] [ 5438.600267] zio_vdev_io_start+0x96/0x2b0 [zfs] [ 5438.604157] zio_nowait+0x97/0x120 [zfs] [ 5438.608097] vdev_raidz_io_start_read+0x129/0x1b0 [zfs] [ 5438.612157] vdev_raidz_io_start+0x4d/0x70 [zfs] [ 5438.616242] zio_vdev_io_start+0xd5/0x2b0 [zfs] [ 5438.620366] zio_nowait+0x97/0x120 [zfs] [ 5438.624481] vdev_mirror_io_start+0xa2/0x230 [zfs] [ 5438.628658] zio_vdev_io_start+0x244/0x2b0 [zfs] [ 5438.632833] zio_execute+0x90/0xe0 [zfs] [ 5438.636987] taskq_thread+0x1ff/0x3c0 [spl] [ 5438.641058] ? wake_up_q+0x90/0x90 [ 5438.645150] ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs] [ 5438.649553] ? taskq_thread_spawn+0x60/0x60 [spl] [ 5438.653813] kthread+0x127/0x150 [ 5438.658040] ? set_kthread_struct+0x50/0x50 [ 5438.662281] ret_from_fork+0x22/0x30 [ 5438.666524] </TASK> [ 5438.880876] INFO: task nmbd:8056 blocked for more than 121 seconds. [ 5438.885251] Tainted: P OE 5.15.131+truenas #1 [ 5438.889669] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5438.894221] task:nmbd state:D stack: 0 pid: 8056 ppid: 1 flags:0x00000000 [ 5438.898915] Call Trace: [ 5438.903521] <TASK> [ 5438.908102] __schedule+0x2f0/0x950 [ 5438.912723] ? dbuf_read_impl.constprop.0+0x284/0x380 [zfs] [ 5438.917667] schedule+0x5b/0xd0 [ 5438.922419] schedule_preempt_disabled+0xa/0x10 [ 5438.927250] __mutex_lock.constprop.0+0x210/0x400 [ 5438.932157] ? __cond_resched+0x16/0x50 [ 5438.937040] arc_buf_access+0x7f/0x200 [zfs] [ 5438.942105] dbuf_hold_impl+0xa1/0x640 [zfs] [ 5438.947080] dbuf_hold_level+0x2b/0x60 [zfs] [ 5438.952074] dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs] [ 5438.957157] dmu_read_impl+0xa8/0x180 [zfs] [ 5438.962284] dmu_read+0x56/0xa0 [zfs] [ 5438.967434] zfs_fillpage+0x7a/0x180 [zfs] [ 5438.972493] zfs_getpage+0x50/0xf0 [zfs] [ 5438.977418] zpl_readpage_common+0x29/0x50 [zfs] [ 5438.982265] filemap_read_page+0x36/0xf0 [ 5438.986862] filemap_fault+0x98d/0xa80 [ 5438.991357] __do_fault+0x36/0x120 [ 5438.995862] __handle_mm_fault+0xce7/0x1580 [ 5439.000423] handle_mm_fault+0xcf/0x2b0 [ 5439.004993] do_user_addr_fault+0x1be/0x670 [ 5439.009585] exc_page_fault+0x72/0x150 [ 5439.014130] asm_exc_page_fault+0x22/0x30 [ 5439.018679] RIP: 0033:0x7f4b100420e0 [ 5439.023217] RSP: 002b:00007fffa867d798 EFLAGS: 00010206 [ 5439.027809] RAX: 0000000000000000 RBX: 00007fffa867d7fc RCX: 0000000000000000 [ 5439.032490] RDX: 0000000000000240 RSI: 00007fffa867d880 RDI: 000000000000000f [ 5439.037114] RBP: 00007fffa867d800 R08: 00007fffa867d800 R09: 00007fffa867d7fc [ 5439.041772] R10: 0000000000002710 R11: 0000000000000246 R12: 0000000000000000 [ 5439.046443] R13: 0000000000000240 R14: 00007fffa867d880 R15: 000000000000000f [ 5439.051084] </TASK> [ 5439.212867] INFO: task smbd:18590 blocked for more than 121 seconds. [ 5439.217592] Tainted: P OE 5.15.131+truenas #1 [ 5439.222353] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5439.227244] task:smbd state:D stack: 0 pid:18590 ppid: 6000 flags:0x00000000 [ 5439.232260] Call Trace: [ 5439.237141] <TASK> [ 5439.241913] __schedule+0x2f0/0x950 [ 5439.246705] ? __dev_queue_xmit+0x350/0xb20 [ 5439.251509] schedule+0x5b/0xd0 [ 5439.256282] schedule_preempt_disabled+0xa/0x10 [ 5439.261102] __mutex_lock.constprop.0+0x210/0x400 [ 5439.265895] dbuf_find+0xb1/0x190 [zfs] [ 5439.270791] ? gethrtime+0x1c/0x50 [zfs] [ 5439.275548] dbuf_hold_impl+0x60/0x640 [zfs] [ 5439.280173] dbuf_hold_level+0x2b/0x60 [zfs] [ 5439.284706] dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs] [ 5439.289356] dmu_read_impl+0xa8/0x180 [zfs] [ 5439.294010] dmu_read+0x56/0xa0 [zfs] [ 5439.298596] zfs_fillpage+0x7a/0x180 [zfs] [ 5439.303159] zfs_getpage+0x50/0xf0 [zfs] [ 5439.307748] zpl_readpage_common+0x29/0x50 [zfs] [ 5439.312332] filemap_read_page+0x36/0xf0 [ 5439.316737] filemap_fault+0x98d/0xa80 [ 5439.321125] __do_fault+0x36/0x120 [ 5439.325530] __handle_mm_fault+0xce7/0x1580 [ 5439.329955] handle_mm_fault+0xcf/0x2b0 [ 5439.334366] do_user_addr_fault+0x1be/0x670 [ 5439.338749] exc_page_fault+0x72/0x150 [ 5439.343161] asm_exc_page_fault+0x22/0x30 [ 5439.347582] RIP: 0033:0x7f084bd580f0 [ 5439.351985] RSP: 002b:00007fffe43f20b8 EFLAGS: 00010206 [ 5439.356463] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000086e9e [ 5439.361013] RDX: 00007f084bdae998 RSI: 00007f084bdaecb3 RDI: 000055c5b9d8ff00 [ 5439.365601] RBP: 000055c5b9e52030 R08: 000055c5b9d8ff00 R09: 00007f084ba07be0 [ 5439.370243] R10: 000055c5b9d80110 R11: 0000000000000000 R12: 000055c5b9d80110 [ 5439.374895] R13: 000055c5b9e52058 R14: 0000000065a811ce R15: 0000000000086e9e [ 5439.379534] </TASK> [ 5439.720862] INFO: task smbd:1168290 blocked for more than 121 seconds. [ 5439.725603] Tainted: P OE 5.15.131+truenas #1 [ 5439.730406] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5439.735342] task:smbd state:D stack: 0 pid:1168290 ppid: 6000 flags:0x00000000 [ 5439.740421] Call Trace: [ 5439.745336] <TASK> [ 5439.750239] __schedule+0x2f0/0x950 [ 5439.755159] schedule+0x5b/0xd0 [ 5439.760019] schedule_preempt_disabled+0xa/0x10 [ 5439.764941] __mutex_lock.constprop.0+0x210/0x400 [ 5439.769797] dbuf_find+0x4e/0x190 [zfs] [ 5439.774789] ? zio_create+0x42c/0x4e0 [zfs] [ 5439.779686] dbuf_hold_impl+0x60/0x640 [zfs] [ 5439.784454] dbuf_hold_level+0x2b/0x60 [zfs] [ 5439.789073] dmu_buf_hold_array_by_dnode+0x107/0x530 [zfs] [ 5439.793684] dmu_read_impl+0xa8/0x180 [zfs] [ 5439.798344] dmu_read+0x56/0xa0 [zfs] [ 5439.802973] zfs_fillpage+0x7a/0x180 [zfs] [ 5439.807643] zfs_getpage+0x50/0xf0 [zfs] [ 5439.812297] zpl_readpage_common+0x29/0x50 [zfs] [ 5439.816971] filemap_read_page+0x36/0xf0 [ 5439.821494] filemap_fault+0x98d/0xa80 [ 5439.825987] __do_fault+0x36/0x120 [ 5439.830437] __handle_mm_fault+0xce7/0x1580 [ 5439.834922] handle_mm_fault+0xcf/0x2b0 [ 5439.839433] do_user_addr_fault+0x1be/0x670 [ 5439.843911] exc_page_fault+0x72/0x150 [ 5439.848377] asm_exc_page_fault+0x22/0x30 [ 5439.852863] RIP: 0033:0x7f084bd6edb0 [ 5439.857330] RSP: 002b:00007fffe43f1678 EFLAGS: 00010206 [ 5439.861869] RAX: 00000000000001bb RBX: 000055c5b9d9f5e0 RCX: 00007fffe43f1770 [ 5439.866518] RDX: 00000000000001bb RSI: 000055c5b9e33420 RDI: 00007fffe43f16a0 [ 5439.871117] RBP: 00007fffe43f1ac0 R08: 0000000000002717 R09: 0000000000000000 [ 5439.875772] R10: 1214b52100000000 R11: 1214b520ad495bad R12: 0000000000000018 [ 5439.880439] R13: 000055c5b9e33420 R14: 00007f084bd6edb0 R15: 00007fffe43f1770 [ 5439.885169] </TASK> [ 5440.120865] INFO: task z_rd_iss:1336082 blocked for more than 122 seconds. [ 5440.125728] Tainted: P OE 5.15.131+truenas #1 [ 5440.130641] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5440.135691] task:z_rd_iss state:D stack: 0 pid:1336082 ppid: 2 flags:0x00004000 [ 5440.140887] Call Trace: [ 5440.145951] <TASK> [ 5440.150930] __schedule+0x2f0/0x950 [ 5440.155911] schedule+0x5b/0xd0 [ 5440.160858] schedule_preempt_disabled+0xa/0x10 [ 5440.165801] __mutex_lock.constprop.0+0x210/0x400 [ 5440.170771] ? ktime_get_raw_ts64+0x3e/0xe0 [ 5440.175662] vdev_queue_io+0x87/0x190 [zfs] [ 5440.180709] zio_vdev_io_start+0x96/0x2b0 [zfs] [ 5440.185626] zio_nowait+0x97/0x120 [zfs] [ 5440.190446] vdev_raidz_io_start_read+0x129/0x1b0 [zfs] [ 5440.195207] vdev_raidz_io_start+0x4d/0x70 [zfs] [ 5440.199876] zio_vdev_io_start+0xd5/0x2b0 [zfs] [ 5440.204601] zio_nowait+0x97/0x120 [zfs] [ 5440.209327] vdev_mirror_io_start+0xa2/0x230 [zfs] [ 5440.214109] zio_vdev_io_start+0x244/0x2b0 [zfs] [ 5440.218926] zio_execute+0x90/0xe0 [zfs] [ 5440.223701] taskq_thread+0x1ff/0x3c0 [spl] [ 5440.228321] ? wake_up_q+0x90/0x90 [ 5440.232927] ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs] [ 5440.237803] ? taskq_thread_spawn+0x60/0x60 [spl] [ 5440.242575] kthread+0x127/0x150 [ 5440.247292] ? set_kthread_struct+0x50/0x50 [ 5440.252015] ret_from_fork+0x22/0x30 [ 5440.256749] </TASK> [ 5440.572487] INFO: task z_rd_iss:1336114 blocked for more than 122 seconds. [ 5440.577398] Tainted: P OE 5.15.131+truenas #1 [ 5440.582323] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5440.587378] task:z_rd_iss state:D stack: 0 pid:1336114 ppid: 2 flags:0x00004000 [ 5440.592588] Call Trace: [ 5440.597720] <TASK> [ 5440.602801] __schedule+0x2f0/0x950 [ 5440.607923] schedule+0x5b/0xd0 [ 5440.613020] schedule_preempt_disabled+0xa/0x10 [ 5440.618119] __mutex_lock.constprop.0+0x210/0x400 [ 5440.623250] ? ktime_get_raw_ts64+0x3e/0xe0 [ 5440.628379] vdev_queue_io+0x87/0x190 [zfs] [ 5440.633715] zio_vdev_io_start+0x96/0x2b0 [zfs] [ 5440.638940] zio_nowait+0x97/0x120 [zfs] [ 5440.644185] vdev_raidz_io_start_read+0x129/0x1b0 [zfs] [ 5440.649482] vdev_raidz_io_start+0x4d/0x70 [zfs] [ 5440.654697] zio_vdev_io_start+0xd5/0x2b0 [zfs] [ 5440.659891] zio_nowait+0x97/0x120 [zfs] [ 5440.664991] vdev_mirror_io_start+0xa2/0x230 [zfs] [ 5440.670061] zio_vdev_io_start+0x244/0x2b0 [zfs] [ 5440.675067] zio_execute+0x90/0xe0 [zfs] [ 5440.679967] taskq_thread+0x1ff/0x3c0 [spl] [ 5440.684614] ? wake_up_q+0x90/0x90 [ 5440.689181] ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs] [ 5440.694059] ? taskq_thread_spawn+0x60/0x60 [spl] [ 5440.698819] kthread+0x127/0x150 [ 5440.703582] ? set_kthread_struct+0x50/0x50 [ 5440.708364] ret_from_fork+0x22/0x30 [ 5440.713122] </TASK> [ 5440.980849] INFO: task z_rd_iss:1336116 blocked for more than 123 seconds. [ 5440.985740] Tainted: P OE 5.15.131+truenas #1 [ 5440.990662] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 5440.995720] task:z_rd_iss state:D stack: 0 pid:1336116 ppid: 2 flags:0x00004000 [ 5441.000905] Call Trace: [ 5441.006003] <TASK> [ 5441.011040] __schedule+0x2f0/0x950 [ 5441.016090] schedule+0x5b/0xd0 [ 5441.021093] schedule_preempt_disabled+0xa/0x10 [ 5441.026107] __mutex_lock.constprop.0+0x210/0x400 [ 5441.031142] ? ktime_get_raw_ts64+0x3e/0xe0 [ 5441.036174] vdev_queue_io+0x87/0x190 [zfs] [ 5441.041402] zio_vdev_io_start+0x96/0x2b0 [zfs] [ 5441.046535] zio_nowait+0x97/0x120 [zfs] [ 5441.051604] vdev_raidz_io_start_read+0x129/0x1b0 [zfs] [ 5441.056724] vdev_raidz_io_start+0x4d/0x70 [zfs] [ 5441.061812] zio_vdev_io_start+0xd5/0x2b0 [zfs] [ 5441.066903] zio_nowait+0x97/0x120 [zfs] [ 5441.071983] vdev_mirror_io_start+0xa2/0x230 [zfs] [ 5441.077035] zio_vdev_io_start+0x244/0x2b0 [zfs] [ 5441.082050] zio_execute+0x90/0xe0 [zfs] [ 5441.086937] taskq_thread+0x1ff/0x3c0 [spl] [ 5441.091599] ? wake_up_q+0x90/0x90 [ 5441.096157] ? zio_execute_stack_check.constprop.0+0x10/0x10 [zfs] [ 5441.101051] ? taskq_thread_spawn+0x60/0x60 [spl] [ 5441.105803] kthread+0x127/0x150 [ 5441.110530] ? set_kthread_struct+0x50/0x50 [ 5441.115308] ret_from_fork+0x22/0x30 [ 5441.120060] </TASK> [ 5533.937060] alx 0000:02:00.0 enp2s0: Link Down [ 5579.677910] alx 0000:02:00.0 enp2s0: NIC Up: 1 Gbps Full [ 5602.524712] systemd-coredump[1336127]: Process 5439 (systemd-journal) of user 0 dumped core. [ 5602.527744] systemd-coredump[1336127]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.5c4b896f2ba641f5b6ca67b21c8f41d0.5439.1705513759000000.zst [ 5602.534348] systemd-coredump[1336127]: Stack trace of thread 5439: [ 5602.537878] systemd-coredump[1336127]: #0 0x00007f18a74bee5b journal_file_find_data_object_with_hash (libsystemd-shared-252.so + 0x216e5b) [ 5602.545516] systemd-coredump[1336127]: #1 0x00007f18a74bf180 n/a (libsystemd-shared-252.so + 0x217180) [ 5602.549743] systemd-coredump[1336127]: #2 0x00007f18a74bfb67 journal_file_append_entry (libsystemd-shared-252.so + 0x217b67) [ 5602.554097] systemd-coredump[1336127]: #3 0x0000563f263f5871 n/a (systemd-journald + 0x12871) [ 5602.558521] systemd-coredump[1336127]: #4 0x0000563f263f6eed n/a (systemd-journald + 0x13eed) [ 5602.562954] systemd-coredump[1336127]: #5 0x00007f18a74e45f0 n/a (libsystemd-shared-252.so + 0x23c5f0) [ 5602.567449] systemd-coredump[1336127]: #6 0x00007f18a74e48fd sd_event_dispatch (libsystemd-shared-252.so + 0x23c8fd) [ 5602.572043] systemd-coredump[1336127]: #7 0x00007f18a74e74d0 sd_event_run (libsystemd-shared-252.so + 0x23f4d0) [ 5602.576665] systemd-coredump[1336127]: #8 0x0000563f263eb54d n/a (systemd-journald + 0x854d) [ 5602.581455] systemd-coredump[1336127]: #9 0x00007f18a708bd0a __libc_start_main (libc.so.6 + 0x23d0a) [ 5602.586263] systemd-coredump[1336127]: #10 0x0000563f263ebc3a n/a (systemd-journald + 0x8c3a) [ 5602.591228] systemd-coredump[1336127]: ELF object binary architecture: AMD x86-64 [ 5602.600408] systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT [ 5602.605655] systemd[1]: systemd-journald.service: Failed with result 'watchdog'. [ 5602.611538] systemd[1]: systemd-journald.service: Consumed 2min 36.733s CPU time. [ 5602.618369] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 5602.624003] systemd[1]: Stopped systemd-journald.service - Journal Service. [ 5602.629440] systemd[1]: systemd-journald.service: Consumed 2min 36.733s CPU time. [ 5603.406345] systemd[1]: Starting systemd-journald.service - Journal Service... [ 5603.547481] systemd-journald[1343717]: File /run/log/journal/7f241b3ec338497fbb5c906cca394ff9/system.journal corrupted or uncleanly shut down, renaming and replacing.
