Jira Ticket: NAS-110481
I'm running TrueNAS SCALE 21.04 that I updated from 21.02 when it came out. This issues seems to have just started in the last few hours. Right now my webgui still "works" but whenever I try to make a change or do anything it freezes and just spins forever. I have tested this on the latest Chrome and Firefox.
With the webgui frozen, how can I dump the debug info from command line?
I think I need to reboot the machine but I have several XCP-ng VMs remotely running on it plus some CIFS/NFS datasets that are being used by the those VMs so it's hard to just migrate them to local storage temporarily.
Here is what I grabbed from DMESG that shows some issues.
I'm running TrueNAS SCALE 21.04 that I updated from 21.02 when it came out. This issues seems to have just started in the last few hours. Right now my webgui still "works" but whenever I try to make a change or do anything it freezes and just spins forever. I have tested this on the latest Chrome and Firefox.
With the webgui frozen, how can I dump the debug info from command line?
I think I need to reboot the machine but I have several XCP-ng VMs remotely running on it plus some CIFS/NFS datasets that are being used by the those VMs so it's hard to just migrate them to local storage temporarily.
Here is what I grabbed from DMESG that shows some issues.
Code:
[140418.906989] INFO: task io_wqe_worker-0:3653922 blocked for more than 120 seconds. [140418.914893] Tainted: P OE 5.10.18+truenas #1 [140418.921275] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140418.929496] task:io_wqe_worker-0 state:D stack: 0 pid:3653922 ppid: 2 flags:0x00004000 [140418.938427] Call Trace: [140418.941281] __schedule+0x282/0x870 [140418.945130] ? spl_kmem_cache_free+0xe9/0x1b0 [spl] [140418.950363] schedule+0x46/0xb0 [140418.953843] cv_wait_common+0x14a/0x290 [spl] [140418.958540] ? add_wait_queue_exclusive+0x70/0x70 [140418.963631] dmu_tx_wait+0xbb/0x2c0 [zfs] [140418.968149] dmu_tx_assign+0x70/0x150 [zfs] [140418.972766] zfs_write+0x42b/0xef0 [zfs] [140418.977139] zpl_iter_write+0xda/0x140 [zfs] [140418.981784] io_write+0xee/0x2f0 [140418.985401] ? unfreeze_partials+0x115/0x190 [140418.990116] ? do_softirq_own_stack+0x37/0x40 [140418.994978] ? irq_exit_rcu+0x3e/0xc0 [140418.999077] ? sysvec_apic_timer_interrupt+0x36/0x80 [140419.004429] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [140419.010103] io_issue_sqe+0x793/0xf20 [140419.014116] ? kmem_cache_free+0x3e5/0x410 [140419.018566] io_wq_submit_work+0x50/0x1a0 [140419.022959] io_worker_handle_work+0x1a1/0x570 [140419.027767] io_wqe_worker+0x2a9/0x350 [140419.031879] ? io_worker_handle_work+0x570/0x570 [140419.036883] kthread+0x11b/0x140 [140419.040486] ? __kthread_bind_mask+0x60/0x60 [140419.045151] ret_from_fork+0x22/0x30 [140539.713296] INFO: task io_wqe_worker-0:3653922 blocked for more than 241 seconds. [140539.721812] Tainted: P OE 5.10.18+truenas #1 [140539.728530] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140539.736960] task:io_wqe_worker-0 state:D stack: 0 pid:3653922 ppid: 2 flags:0x00004000 [140539.746012] Call Trace: [140539.749088] __schedule+0x282/0x870 [140539.753121] ? spl_kmem_cache_free+0xe9/0x1b0 [spl] [140539.758551] schedule+0x46/0xb0 [140539.762358] cv_wait_common+0x14a/0x290 [spl] [140539.767333] ? add_wait_queue_exclusive+0x70/0x70 [140539.772592] dmu_tx_wait+0xbb/0x2c0 [zfs] [140539.777193] dmu_tx_assign+0x70/0x150 [zfs] [140539.781942] zfs_write+0x42b/0xef0 [zfs] [140539.786458] zpl_iter_write+0xda/0x140 [zfs] [140539.791250] io_write+0xee/0x2f0 [140539.795009] ? unfreeze_partials+0x115/0x190 [140539.799792] ? do_softirq_own_stack+0x37/0x40 [140539.804663] ? irq_exit_rcu+0x3e/0xc0 [140539.808859] ? sysvec_apic_timer_interrupt+0x36/0x80 [140539.814354] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [140539.820186] io_issue_sqe+0x793/0xf20 [140539.824357] ? kmem_cache_free+0x3e5/0x410 [140539.828965] io_wq_submit_work+0x50/0x1a0 [140539.833471] io_worker_handle_work+0x1a1/0x570 [140539.838404] io_wqe_worker+0x2a9/0x350 [140539.842647] ? io_worker_handle_work+0x570/0x570 [140539.847762] kthread+0x11b/0x140 [140539.851480] ? __kthread_bind_mask+0x60/0x60 [140539.856242] ret_from_fork+0x22/0x30 [140660.522947] INFO: task smbd:3653908 blocked for more than 120 seconds. [140660.530058] Tainted: P OE 5.10.18+truenas #1 [140660.536598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140660.544940] task:smbd state:D stack: 0 pid:3653908 ppid:453577 flags:0x00004000 [140660.553984] Call Trace: [140660.556965] __schedule+0x282/0x870 [140660.561039] ? __raw_spin_unlock+0x5/0x10 [zfs] [140660.566104] schedule+0x46/0xb0 [140660.569780] cv_wait_common+0x14a/0x290 [spl] [140660.574659] ? add_wait_queue_exclusive+0x70/0x70 [140660.579920] dmu_tx_wait+0xbb/0x2c0 [zfs] [140660.584514] zfs_create+0x24a/0x940 [zfs] [140660.589109] zpl_create+0xae/0x1c0 [zfs] [140660.593562] path_openat+0xdb7/0x1100 [140660.598103] do_filp_open+0x88/0x130 [140660.602189] ? getname_flags.part.0+0x29/0x1a0 [140660.607138] ? __check_object_size+0x136/0x150 [140660.612098] do_sys_openat2+0x97/0x150 [140660.616376] __x64_sys_openat+0x54/0x90 [140660.620995] do_syscall_64+0x33/0x80 [140660.625137] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [140660.630687] RIP: 0033:0x7ff4de909d62 [140660.634756] RSP: 002b:00007ffd92652110 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [140660.642888] RAX: ffffffffffffffda RBX: 00005562a41ed360 RCX: 00007ff4de909d62 [140660.650558] RDX: 00000000000208c2 RSI: 00005562a40a9590 RDI: 00000000ffffff9c [140660.658192] RBP: 00005562a40a9590 R08: 00000000000208c2 R09: 00000000000001e4 [140660.665812] R10: 00000000000001e4 R11: 0000000000000246 R12: 00000000000208c2 [140660.673428] R13: 0000000000000000 R14: 00005562a40ddda0 R15: 00000000000001e4 [140660.681069] INFO: task io_wqe_worker-0:3653922 blocked for more than 362 seconds. [140660.698009] Tainted: P OE 5.10.18+truenas #1 [140660.710165] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140660.725454] task:io_wqe_worker-0 state:D stack: 0 pid:3653922 ppid: 2 flags:0x00004000 [140660.734570] Call Trace: [140660.737635] __schedule+0x282/0x870 [140660.741673] ? spl_kmem_cache_free+0xe9/0x1b0 [spl] [140660.747089] schedule+0x46/0xb0 [140660.750757] cv_wait_common+0x14a/0x290 [spl] [140660.755635] ? add_wait_queue_exclusive+0x70/0x70 [140660.760915] dmu_tx_wait+0xbb/0x2c0 [zfs] [140660.765504] dmu_tx_assign+0x70/0x150 [zfs] [140660.770278] zfs_write+0x42b/0xef0 [zfs] [140660.774789] zpl_iter_write+0xda/0x140 [zfs] [140660.779581] io_write+0xee/0x2f0 [140660.783334] ? unfreeze_partials+0x115/0x190 [140660.788127] ? do_softirq_own_stack+0x37/0x40 [140660.793006] ? irq_exit_rcu+0x3e/0xc0 [140660.793009] ? sysvec_apic_timer_interrupt+0x36/0x80 [140660.793011] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [140660.793014] io_issue_sqe+0x793/0xf20 [140660.812705] ? kmem_cache_free+0x3e5/0x410 [140660.812707] io_wq_submit_work+0x50/0x1a0 [140660.812709] io_worker_handle_work+0x1a1/0x570 [140660.826776] io_wqe_worker+0x2a9/0x350 [140660.826779] ? io_worker_handle_work+0x570/0x570 [140660.836128] kthread+0x11b/0x140 [140660.836130] ? __kthread_bind_mask+0x60/0x60 [140660.844627] ret_from_fork+0x22/0x30 [140781.324404] INFO: task smbd:3653908 blocked for more than 241 seconds. [140781.331810] Tainted: P OE 5.10.18+truenas #1 [140781.338584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140781.346931] task:smbd state:D stack: 0 pid:3653908 ppid:453577 flags:0x00004000 [140781.355998] Call Trace: [140781.358986] __schedule+0x282/0x870 [140781.363068] ? __raw_spin_unlock+0x5/0x10 [zfs] [140781.368136] schedule+0x46/0xb0 [140781.371805] cv_wait_common+0x14a/0x290 [spl] [140781.376691] ? add_wait_queue_exclusive+0x70/0x70 [140781.381977] dmu_tx_wait+0xbb/0x2c0 [zfs] [140781.386585] zfs_create+0x24a/0x940 [zfs] [140781.391196] zpl_create+0xae/0x1c0 [zfs] [140781.395646] path_openat+0xdb7/0x1100 [140781.399840] do_filp_open+0x88/0x130 [140781.403973] ? getname_flags.part.0+0x29/0x1a0 [140781.408928] ? __check_object_size+0x136/0x150 [140781.413898] do_sys_openat2+0x97/0x150 [140781.418158] __x64_sys_openat+0x54/0x90 [140781.422520] do_syscall_64+0x33/0x80 [140781.426602] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [140781.432149] RIP: 0033:0x7ff4de909d62 [140781.436208] RSP: 002b:00007ffd92652110 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [140781.444263] RAX: ffffffffffffffda RBX: 00005562a41ed360 RCX: 00007ff4de909d62 [140781.451879] RDX: 00000000000208c2 RSI: 00005562a40a9590 RDI: 00000000ffffff9c [140781.459517] RBP: 00005562a40a9590 R08: 00000000000208c2 R09: 00000000000001e4 [140781.467164] R10: 00000000000001e4 R11: 0000000000000246 R12: 00000000000208c2 [140781.474831] R13: 0000000000000000 R14: 00005562a40ddda0 R15: 00000000000001e4 [140781.482490] INFO: task io_wqe_worker-0:3653922 blocked for more than 483 seconds. [140781.499265] Tainted: P OE 5.10.18+truenas #1 [140781.511412] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140781.526878] task:io_wqe_worker-0 state:D stack: 0 pid:3653922 ppid: 2 flags:0x00004000 [140781.535990] Call Trace: [140781.538995] __schedule+0x282/0x870 [140781.543016] ? spl_kmem_cache_free+0xe9/0x1b0 [spl] [140781.548430] schedule+0x46/0xb0 [140781.552093] cv_wait_common+0x14a/0x290 [spl] [140781.556984] ? add_wait_queue_exclusive+0x70/0x70 [140781.562276] dmu_tx_wait+0xbb/0x2c0 [zfs] [140781.566869] dmu_tx_assign+0x70/0x150 [zfs] [140781.571644] zfs_write+0x42b/0xef0 [zfs] [140781.576163] zpl_iter_write+0xda/0x140 [zfs] [140781.580955] io_write+0xee/0x2f0 [140781.584708] ? unfreeze_partials+0x115/0x190 [140781.589498] ? do_softirq_own_stack+0x37/0x40 [140781.594385] ? irq_exit_rcu+0x3e/0xc0 [140781.594388] ? sysvec_apic_timer_interrupt+0x36/0x80 [140781.594391] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [140781.594396] io_issue_sqe+0x793/0xf20 [140781.614460] ? kmem_cache_free+0x3e5/0x410 [140781.614463] io_wq_submit_work+0x50/0x1a0 [140781.614467] io_worker_handle_work+0x1a1/0x570 [140781.629064] io_wqe_worker+0x2a9/0x350 [140781.629065] ? io_worker_handle_work+0x570/0x570 [140781.629068] kthread+0x11b/0x140 [140781.629069] ? __kthread_bind_mask+0x60/0x60 [140781.629071] ret_from_fork+0x22/0x30 [140781.629716] INFO: task io_wqe_worker-0:1058974 blocked for more than 120 seconds. [140781.666758] Tainted: P OE 5.10.18+truenas #1 [140781.666759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140781.666762] task:io_wqe_worker-0 state:D stack: 0 pid:1058974 ppid: 2 flags:0x00004000 [140781.666766] Call Trace: [140781.666781] __schedule+0x282/0x870 [140781.666785] schedule+0x46/0xb0 [140781.666805] cv_wait_common+0x14a/0x290 [spl] [140781.666810] ? add_wait_queue_exclusive+0x70/0x70 [140781.666957] zfs_rangelock_enter_writer+0x43/0x1a0 [zfs] [140781.667071] zfs_rangelock_enter_impl+0x104/0x180 [zfs] [140781.733322] zfs_write+0xad3/0xef0 [zfs] [140781.733391] ? update_group_capacity+0x25/0x1d0 [140781.751981] ? cpumask_next_and+0x1a/0x20 [140781.751984] ? aa_file_perm+0x113/0x480 [140781.752061] zpl_iter_write+0xda/0x140 [zfs] [140781.752065] io_write+0xee/0x2f0 [140781.752067] ? cpumask_next_and+0x1a/0x20 [140781.752070] ? load_balance+0x1b2/0xd40 [140781.752072] ? update_load_avg+0x7a/0x5e0 [140781.752073] ? dequeue_entity+0xc6/0x450 [140781.752075] io_issue_sqe+0x793/0xf20 [140781.752079] ? __switch_to_asm+0x42/0x70 [140781.752081] ? __switch_to+0x114/0x450 [140781.752082] io_wq_submit_work+0x50/0x1a0 [140781.752084] io_worker_handle_work+0x1a1/0x570 [140781.752086] io_wqe_worker+0x2a9/0x350 [140781.752087] ? io_worker_handle_work+0x570/0x570 [140781.752089] kthread+0x11b/0x140 [140781.752091] ? __kthread_bind_mask+0x60/0x60 [140781.752092] ret_from_fork+0x22/0x30 [140902.129562] INFO: task smbd:3653908 blocked for more than 362 seconds. [140902.136846] Tainted: P OE 5.10.18+truenas #1 [140902.143556] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140902.151876] task:smbd state:D stack: 0 pid:3653908 ppid:453577 flags:0x00004000 [140902.160897] Call Trace: [140902.163841] __schedule+0x282/0x870 [140902.167886] ? __raw_spin_unlock+0x5/0x10 [zfs] [140902.172910] schedule+0x46/0xb0 [140902.176545] cv_wait_common+0x14a/0x290 [spl] [140902.181394] ? add_wait_queue_exclusive+0x70/0x70 [140902.186786] dmu_tx_wait+0xbb/0x2c0 [zfs] [140902.191380] zfs_create+0x24a/0x940 [zfs] [140902.196068] zpl_create+0xae/0x1c0 [zfs] [140902.200479] path_openat+0xdb7/0x1100 [140902.204619] do_filp_open+0x88/0x130 [140902.208676] ? getname_flags.part.0+0x29/0x1a0 [140902.213600] ? __check_object_size+0x136/0x150 [140902.218515] do_sys_openat2+0x97/0x150 [140902.222734] __x64_sys_openat+0x54/0x90 [140902.227033] do_syscall_64+0x33/0x80 [140902.231064] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [140902.236556] RIP: 0033:0x7ff4de909d62 [140902.240578] RSP: 002b:00007ffd92652110 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 [140902.248605] RAX: ffffffffffffffda RBX: 00005562a41ed360 RCX: 00007ff4de909d62 [140902.256203] RDX: 00000000000208c2 RSI: 00005562a40a9590 RDI: 00000000ffffff9c [140902.263805] RBP: 00005562a40a9590 R08: 00000000000208c2 R09: 00000000000001e4 [140902.271405] R10: 00000000000001e4 R11: 0000000000000246 R12: 00000000000208c2 [140902.279004] R13: 0000000000000000 R14: 00005562a40ddda0 R15: 00000000000001e4 [140902.286610] INFO: task io_wqe_worker-0:3653922 blocked for more than 604 seconds. [140902.303250] Tainted: P OE 5.10.18+truenas #1 [140902.315222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140902.330723] task:io_wqe_worker-0 state:D stack: 0 pid:3653922 ppid: 2 flags:0x00004000 [140902.339994] Call Trace: [140902.342979] __schedule+0x282/0x870 [140902.346985] ? spl_kmem_cache_free+0xe9/0x1b0 [spl] [140902.352375] schedule+0x46/0xb0 [140902.356022] cv_wait_common+0x14a/0x290 [spl] [140902.360892] ? add_wait_queue_exclusive+0x70/0x70 [140902.366169] dmu_tx_wait+0xbb/0x2c0 [zfs] [140902.370738] dmu_tx_assign+0x70/0x150 [zfs] [140902.375505] zfs_write+0x42b/0xef0 [zfs] [140902.380007] zpl_iter_write+0xda/0x140 [zfs] [140902.384805] io_write+0xee/0x2f0 [140902.388552] ? unfreeze_partials+0x115/0x190 [140902.393339] ? do_softirq_own_stack+0x37/0x40 [140902.398208] ? irq_exit_rcu+0x3e/0xc0 [140902.402390] ? sysvec_apic_timer_interrupt+0x36/0x80 [140902.402394] ? asm_sysvec_apic_timer_interrupt+0x12/0x20 [140902.402397] io_issue_sqe+0x793/0xf20 [140902.402399] ? kmem_cache_free+0x3e5/0x410 [140902.402400] io_wq_submit_work+0x50/0x1a0 [140902.402404] io_worker_handle_work+0x1a1/0x570 [140902.402412] io_wqe_worker+0x2a9/0x350 [140902.436840] ? io_worker_handle_work+0x570/0x570 [140902.436842] kthread+0x11b/0x140 [140902.436843] ? __kthread_bind_mask+0x60/0x60 [140902.436848] ret_from_fork+0x22/0x30 [140902.456204] INFO: task io_wqe_worker-0:1058974 blocked for more than 241 seconds. [140902.471732] Tainted: P OE 5.10.18+truenas #1 [140902.471733] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [140902.471735] task:io_wqe_worker-0 state:D stack: 0 pid:1058974 ppid: 2 flags:0x00004000 [140902.471739] Call Trace: [140902.471747] __schedule+0x282/0x870 [140902.512822] schedule+0x46/0xb0 [140902.512831] cv_wait_common+0x14a/0x290 [spl] [140902.521828] ? add_wait_queue_exclusive+0x70/0x70 [140902.521897] zfs_rangelock_enter_writer+0x43/0x1a0 [zfs] [140902.521960] zfs_rangelock_enter_impl+0x104/0x180 [zfs] [140902.539177] zfs_write+0xad3/0xef0 [zfs] [140902.539181] ? update_group_capacity+0x25/0x1d0 [140902.549050] ? cpumask_next_and+0x1a/0x20 [140902.549053] ? aa_file_perm+0x113/0x480 [140902.549116] zpl_iter_write+0xda/0x140 [zfs] [140902.563264] io_write+0xee/0x2f0 [140902.563265] ? cpumask_next_and+0x1a/0x20 [140902.563266] ? load_balance+0x1b2/0xd40 [140902.563268] ? update_load_avg+0x7a/0x5e0 [140902.563270] ? dequeue_entity+0xc6/0x450 [140902.563273] io_issue_sqe+0x793/0xf20 [140902.589314] ? __switch_to_asm+0x42/0x70 [140902.589317] ? __switch_to+0x114/0x450 [140902.589319] io_wq_submit_work+0x50/0x1a0 [140902.589322] io_worker_handle_work+0x1a1/0x570 [140902.607627] io_wqe_worker+0x2a9/0x350 [140902.607629] ? io_worker_handle_work+0x570/0x570 [140902.607631] kthread+0x11b/0x140 [140902.607635] ? __kthread_bind_mask+0x60/0x60 [140902.625652] ret_from_fork+0x22/0x30
Attachments
Last edited:
