WebUI temporarily frozen

cyrus104

Explorer
Joined
Feb 7, 2021
Messages
70
Jira Ticket: NAS-110481

I'm running TrueNAS SCALE 21.04 that I updated from 21.02 when it came out. This issues seems to have just started in the last few hours. Right now my webgui still "works" but whenever I try to make a change or do anything it freezes and just spins forever. I have tested this on the latest Chrome and Firefox.

With the webgui frozen, how can I dump the debug info from command line?

I think I need to reboot the machine but I have several XCP-ng VMs remotely running on it plus some CIFS/NFS datasets that are being used by the those VMs so it's hard to just migrate them to local storage temporarily.

Here is what I grabbed from DMESG that shows some issues.
Code:
[140418.906989] INFO: task io_wqe_worker-0:3653922 blocked for more than 120 seconds.
[140418.914893]       Tainted: P           OE     5.10.18+truenas #1
[140418.921275] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140418.929496] task:io_wqe_worker-0 state:D stack:    0 pid:3653922 ppid:     2 flags:0x00004000
[140418.938427] Call Trace:
[140418.941281]  __schedule+0x282/0x870
[140418.945130]  ? spl_kmem_cache_free+0xe9/0x1b0 [spl]
[140418.950363]  schedule+0x46/0xb0
[140418.953843]  cv_wait_common+0x14a/0x290 [spl]
[140418.958540]  ? add_wait_queue_exclusive+0x70/0x70
[140418.963631]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140418.968149]  dmu_tx_assign+0x70/0x150 [zfs]
[140418.972766]  zfs_write+0x42b/0xef0 [zfs]
[140418.977139]  zpl_iter_write+0xda/0x140 [zfs]
[140418.981784]  io_write+0xee/0x2f0
[140418.985401]  ? unfreeze_partials+0x115/0x190
[140418.990116]  ? do_softirq_own_stack+0x37/0x40
[140418.994978]  ? irq_exit_rcu+0x3e/0xc0
[140418.999077]  ? sysvec_apic_timer_interrupt+0x36/0x80
[140419.004429]  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
[140419.010103]  io_issue_sqe+0x793/0xf20
[140419.014116]  ? kmem_cache_free+0x3e5/0x410
[140419.018566]  io_wq_submit_work+0x50/0x1a0
[140419.022959]  io_worker_handle_work+0x1a1/0x570
[140419.027767]  io_wqe_worker+0x2a9/0x350
[140419.031879]  ? io_worker_handle_work+0x570/0x570
[140419.036883]  kthread+0x11b/0x140
[140419.040486]  ? __kthread_bind_mask+0x60/0x60
[140419.045151]  ret_from_fork+0x22/0x30
[140539.713296] INFO: task io_wqe_worker-0:3653922 blocked for more than 241 seconds.
[140539.721812]       Tainted: P           OE     5.10.18+truenas #1
[140539.728530] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140539.736960] task:io_wqe_worker-0 state:D stack:    0 pid:3653922 ppid:     2 flags:0x00004000
[140539.746012] Call Trace:
[140539.749088]  __schedule+0x282/0x870
[140539.753121]  ? spl_kmem_cache_free+0xe9/0x1b0 [spl]
[140539.758551]  schedule+0x46/0xb0
[140539.762358]  cv_wait_common+0x14a/0x290 [spl]
[140539.767333]  ? add_wait_queue_exclusive+0x70/0x70
[140539.772592]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140539.777193]  dmu_tx_assign+0x70/0x150 [zfs]
[140539.781942]  zfs_write+0x42b/0xef0 [zfs]
[140539.786458]  zpl_iter_write+0xda/0x140 [zfs]
[140539.791250]  io_write+0xee/0x2f0
[140539.795009]  ? unfreeze_partials+0x115/0x190
[140539.799792]  ? do_softirq_own_stack+0x37/0x40
[140539.804663]  ? irq_exit_rcu+0x3e/0xc0
[140539.808859]  ? sysvec_apic_timer_interrupt+0x36/0x80
[140539.814354]  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
[140539.820186]  io_issue_sqe+0x793/0xf20
[140539.824357]  ? kmem_cache_free+0x3e5/0x410
[140539.828965]  io_wq_submit_work+0x50/0x1a0
[140539.833471]  io_worker_handle_work+0x1a1/0x570
[140539.838404]  io_wqe_worker+0x2a9/0x350
[140539.842647]  ? io_worker_handle_work+0x570/0x570
[140539.847762]  kthread+0x11b/0x140
[140539.851480]  ? __kthread_bind_mask+0x60/0x60
[140539.856242]  ret_from_fork+0x22/0x30
[140660.522947] INFO: task smbd:3653908 blocked for more than 120 seconds.
[140660.530058]       Tainted: P           OE     5.10.18+truenas #1
[140660.536598] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140660.544940] task:smbd            state:D stack:    0 pid:3653908 ppid:453577 flags:0x00004000
[140660.553984] Call Trace:
[140660.556965]  __schedule+0x282/0x870
[140660.561039]  ? __raw_spin_unlock+0x5/0x10 [zfs]
[140660.566104]  schedule+0x46/0xb0
[140660.569780]  cv_wait_common+0x14a/0x290 [spl]
[140660.574659]  ? add_wait_queue_exclusive+0x70/0x70
[140660.579920]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140660.584514]  zfs_create+0x24a/0x940 [zfs]
[140660.589109]  zpl_create+0xae/0x1c0 [zfs]
[140660.593562]  path_openat+0xdb7/0x1100
[140660.598103]  do_filp_open+0x88/0x130
[140660.602189]  ? getname_flags.part.0+0x29/0x1a0
[140660.607138]  ? __check_object_size+0x136/0x150
[140660.612098]  do_sys_openat2+0x97/0x150
[140660.616376]  __x64_sys_openat+0x54/0x90
[140660.620995]  do_syscall_64+0x33/0x80
[140660.625137]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[140660.630687] RIP: 0033:0x7ff4de909d62
[140660.634756] RSP: 002b:00007ffd92652110 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[140660.642888] RAX: ffffffffffffffda RBX: 00005562a41ed360 RCX: 00007ff4de909d62
[140660.650558] RDX: 00000000000208c2 RSI: 00005562a40a9590 RDI: 00000000ffffff9c
[140660.658192] RBP: 00005562a40a9590 R08: 00000000000208c2 R09: 00000000000001e4
[140660.665812] R10: 00000000000001e4 R11: 0000000000000246 R12: 00000000000208c2
[140660.673428] R13: 0000000000000000 R14: 00005562a40ddda0 R15: 00000000000001e4
[140660.681069] INFO: task io_wqe_worker-0:3653922 blocked for more than 362 seconds.
[140660.698009]       Tainted: P           OE     5.10.18+truenas #1
[140660.710165] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140660.725454] task:io_wqe_worker-0 state:D stack:    0 pid:3653922 ppid:     2 flags:0x00004000
[140660.734570] Call Trace:
[140660.737635]  __schedule+0x282/0x870
[140660.741673]  ? spl_kmem_cache_free+0xe9/0x1b0 [spl]
[140660.747089]  schedule+0x46/0xb0
[140660.750757]  cv_wait_common+0x14a/0x290 [spl]
[140660.755635]  ? add_wait_queue_exclusive+0x70/0x70
[140660.760915]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140660.765504]  dmu_tx_assign+0x70/0x150 [zfs]
[140660.770278]  zfs_write+0x42b/0xef0 [zfs]
[140660.774789]  zpl_iter_write+0xda/0x140 [zfs]
[140660.779581]  io_write+0xee/0x2f0
[140660.783334]  ? unfreeze_partials+0x115/0x190
[140660.788127]  ? do_softirq_own_stack+0x37/0x40
[140660.793006]  ? irq_exit_rcu+0x3e/0xc0
[140660.793009]  ? sysvec_apic_timer_interrupt+0x36/0x80
[140660.793011]  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
[140660.793014]  io_issue_sqe+0x793/0xf20
[140660.812705]  ? kmem_cache_free+0x3e5/0x410
[140660.812707]  io_wq_submit_work+0x50/0x1a0
[140660.812709]  io_worker_handle_work+0x1a1/0x570
[140660.826776]  io_wqe_worker+0x2a9/0x350
[140660.826779]  ? io_worker_handle_work+0x570/0x570
[140660.836128]  kthread+0x11b/0x140
[140660.836130]  ? __kthread_bind_mask+0x60/0x60
[140660.844627]  ret_from_fork+0x22/0x30
[140781.324404] INFO: task smbd:3653908 blocked for more than 241 seconds.
[140781.331810]       Tainted: P           OE     5.10.18+truenas #1
[140781.338584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140781.346931] task:smbd            state:D stack:    0 pid:3653908 ppid:453577 flags:0x00004000
[140781.355998] Call Trace:
[140781.358986]  __schedule+0x282/0x870
[140781.363068]  ? __raw_spin_unlock+0x5/0x10 [zfs]
[140781.368136]  schedule+0x46/0xb0
[140781.371805]  cv_wait_common+0x14a/0x290 [spl]
[140781.376691]  ? add_wait_queue_exclusive+0x70/0x70
[140781.381977]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140781.386585]  zfs_create+0x24a/0x940 [zfs]
[140781.391196]  zpl_create+0xae/0x1c0 [zfs]
[140781.395646]  path_openat+0xdb7/0x1100
[140781.399840]  do_filp_open+0x88/0x130
[140781.403973]  ? getname_flags.part.0+0x29/0x1a0
[140781.408928]  ? __check_object_size+0x136/0x150
[140781.413898]  do_sys_openat2+0x97/0x150
[140781.418158]  __x64_sys_openat+0x54/0x90
[140781.422520]  do_syscall_64+0x33/0x80
[140781.426602]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[140781.432149] RIP: 0033:0x7ff4de909d62
[140781.436208] RSP: 002b:00007ffd92652110 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[140781.444263] RAX: ffffffffffffffda RBX: 00005562a41ed360 RCX: 00007ff4de909d62
[140781.451879] RDX: 00000000000208c2 RSI: 00005562a40a9590 RDI: 00000000ffffff9c
[140781.459517] RBP: 00005562a40a9590 R08: 00000000000208c2 R09: 00000000000001e4
[140781.467164] R10: 00000000000001e4 R11: 0000000000000246 R12: 00000000000208c2
[140781.474831] R13: 0000000000000000 R14: 00005562a40ddda0 R15: 00000000000001e4
[140781.482490] INFO: task io_wqe_worker-0:3653922 blocked for more than 483 seconds.
[140781.499265]       Tainted: P           OE     5.10.18+truenas #1
[140781.511412] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140781.526878] task:io_wqe_worker-0 state:D stack:    0 pid:3653922 ppid:     2 flags:0x00004000
[140781.535990] Call Trace:
[140781.538995]  __schedule+0x282/0x870
[140781.543016]  ? spl_kmem_cache_free+0xe9/0x1b0 [spl]
[140781.548430]  schedule+0x46/0xb0
[140781.552093]  cv_wait_common+0x14a/0x290 [spl]
[140781.556984]  ? add_wait_queue_exclusive+0x70/0x70
[140781.562276]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140781.566869]  dmu_tx_assign+0x70/0x150 [zfs]
[140781.571644]  zfs_write+0x42b/0xef0 [zfs]
[140781.576163]  zpl_iter_write+0xda/0x140 [zfs]
[140781.580955]  io_write+0xee/0x2f0
[140781.584708]  ? unfreeze_partials+0x115/0x190
[140781.589498]  ? do_softirq_own_stack+0x37/0x40
[140781.594385]  ? irq_exit_rcu+0x3e/0xc0
[140781.594388]  ? sysvec_apic_timer_interrupt+0x36/0x80
[140781.594391]  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
[140781.594396]  io_issue_sqe+0x793/0xf20
[140781.614460]  ? kmem_cache_free+0x3e5/0x410
[140781.614463]  io_wq_submit_work+0x50/0x1a0
[140781.614467]  io_worker_handle_work+0x1a1/0x570
[140781.629064]  io_wqe_worker+0x2a9/0x350
[140781.629065]  ? io_worker_handle_work+0x570/0x570
[140781.629068]  kthread+0x11b/0x140
[140781.629069]  ? __kthread_bind_mask+0x60/0x60
[140781.629071]  ret_from_fork+0x22/0x30
[140781.629716] INFO: task io_wqe_worker-0:1058974 blocked for more than 120 seconds.
[140781.666758]       Tainted: P           OE     5.10.18+truenas #1
[140781.666759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140781.666762] task:io_wqe_worker-0 state:D stack:    0 pid:1058974 ppid:     2 flags:0x00004000
[140781.666766] Call Trace:
[140781.666781]  __schedule+0x282/0x870
[140781.666785]  schedule+0x46/0xb0
[140781.666805]  cv_wait_common+0x14a/0x290 [spl]
[140781.666810]  ? add_wait_queue_exclusive+0x70/0x70
[140781.666957]  zfs_rangelock_enter_writer+0x43/0x1a0 [zfs]
[140781.667071]  zfs_rangelock_enter_impl+0x104/0x180 [zfs]
[140781.733322]  zfs_write+0xad3/0xef0 [zfs]
[140781.733391]  ? update_group_capacity+0x25/0x1d0
[140781.751981]  ? cpumask_next_and+0x1a/0x20
[140781.751984]  ? aa_file_perm+0x113/0x480
[140781.752061]  zpl_iter_write+0xda/0x140 [zfs]
[140781.752065]  io_write+0xee/0x2f0
[140781.752067]  ? cpumask_next_and+0x1a/0x20
[140781.752070]  ? load_balance+0x1b2/0xd40
[140781.752072]  ? update_load_avg+0x7a/0x5e0
[140781.752073]  ? dequeue_entity+0xc6/0x450
[140781.752075]  io_issue_sqe+0x793/0xf20
[140781.752079]  ? __switch_to_asm+0x42/0x70
[140781.752081]  ? __switch_to+0x114/0x450
[140781.752082]  io_wq_submit_work+0x50/0x1a0
[140781.752084]  io_worker_handle_work+0x1a1/0x570
[140781.752086]  io_wqe_worker+0x2a9/0x350
[140781.752087]  ? io_worker_handle_work+0x570/0x570
[140781.752089]  kthread+0x11b/0x140
[140781.752091]  ? __kthread_bind_mask+0x60/0x60
[140781.752092]  ret_from_fork+0x22/0x30
[140902.129562] INFO: task smbd:3653908 blocked for more than 362 seconds.
[140902.136846]       Tainted: P           OE     5.10.18+truenas #1
[140902.143556] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140902.151876] task:smbd            state:D stack:    0 pid:3653908 ppid:453577 flags:0x00004000
[140902.160897] Call Trace:
[140902.163841]  __schedule+0x282/0x870
[140902.167886]  ? __raw_spin_unlock+0x5/0x10 [zfs]
[140902.172910]  schedule+0x46/0xb0
[140902.176545]  cv_wait_common+0x14a/0x290 [spl]
[140902.181394]  ? add_wait_queue_exclusive+0x70/0x70
[140902.186786]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140902.191380]  zfs_create+0x24a/0x940 [zfs]
[140902.196068]  zpl_create+0xae/0x1c0 [zfs]
[140902.200479]  path_openat+0xdb7/0x1100
[140902.204619]  do_filp_open+0x88/0x130
[140902.208676]  ? getname_flags.part.0+0x29/0x1a0
[140902.213600]  ? __check_object_size+0x136/0x150
[140902.218515]  do_sys_openat2+0x97/0x150
[140902.222734]  __x64_sys_openat+0x54/0x90
[140902.227033]  do_syscall_64+0x33/0x80
[140902.231064]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[140902.236556] RIP: 0033:0x7ff4de909d62
[140902.240578] RSP: 002b:00007ffd92652110 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
[140902.248605] RAX: ffffffffffffffda RBX: 00005562a41ed360 RCX: 00007ff4de909d62
[140902.256203] RDX: 00000000000208c2 RSI: 00005562a40a9590 RDI: 00000000ffffff9c
[140902.263805] RBP: 00005562a40a9590 R08: 00000000000208c2 R09: 00000000000001e4
[140902.271405] R10: 00000000000001e4 R11: 0000000000000246 R12: 00000000000208c2
[140902.279004] R13: 0000000000000000 R14: 00005562a40ddda0 R15: 00000000000001e4
[140902.286610] INFO: task io_wqe_worker-0:3653922 blocked for more than 604 seconds.
[140902.303250]       Tainted: P           OE     5.10.18+truenas #1
[140902.315222] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140902.330723] task:io_wqe_worker-0 state:D stack:    0 pid:3653922 ppid:     2 flags:0x00004000
[140902.339994] Call Trace:
[140902.342979]  __schedule+0x282/0x870
[140902.346985]  ? spl_kmem_cache_free+0xe9/0x1b0 [spl]
[140902.352375]  schedule+0x46/0xb0
[140902.356022]  cv_wait_common+0x14a/0x290 [spl]
[140902.360892]  ? add_wait_queue_exclusive+0x70/0x70
[140902.366169]  dmu_tx_wait+0xbb/0x2c0 [zfs]
[140902.370738]  dmu_tx_assign+0x70/0x150 [zfs]
[140902.375505]  zfs_write+0x42b/0xef0 [zfs]
[140902.380007]  zpl_iter_write+0xda/0x140 [zfs]
[140902.384805]  io_write+0xee/0x2f0
[140902.388552]  ? unfreeze_partials+0x115/0x190
[140902.393339]  ? do_softirq_own_stack+0x37/0x40
[140902.398208]  ? irq_exit_rcu+0x3e/0xc0
[140902.402390]  ? sysvec_apic_timer_interrupt+0x36/0x80
[140902.402394]  ? asm_sysvec_apic_timer_interrupt+0x12/0x20
[140902.402397]  io_issue_sqe+0x793/0xf20
[140902.402399]  ? kmem_cache_free+0x3e5/0x410
[140902.402400]  io_wq_submit_work+0x50/0x1a0
[140902.402404]  io_worker_handle_work+0x1a1/0x570
[140902.402412]  io_wqe_worker+0x2a9/0x350
[140902.436840]  ? io_worker_handle_work+0x570/0x570
[140902.436842]  kthread+0x11b/0x140
[140902.436843]  ? __kthread_bind_mask+0x60/0x60
[140902.436848]  ret_from_fork+0x22/0x30
[140902.456204] INFO: task io_wqe_worker-0:1058974 blocked for more than 241 seconds.
[140902.471732]       Tainted: P           OE     5.10.18+truenas #1
[140902.471733] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[140902.471735] task:io_wqe_worker-0 state:D stack:    0 pid:1058974 ppid:     2 flags:0x00004000
[140902.471739] Call Trace:
[140902.471747]  __schedule+0x282/0x870
[140902.512822]  schedule+0x46/0xb0
[140902.512831]  cv_wait_common+0x14a/0x290 [spl]
[140902.521828]  ? add_wait_queue_exclusive+0x70/0x70
[140902.521897]  zfs_rangelock_enter_writer+0x43/0x1a0 [zfs]
[140902.521960]  zfs_rangelock_enter_impl+0x104/0x180 [zfs]
[140902.539177]  zfs_write+0xad3/0xef0 [zfs]
[140902.539181]  ? update_group_capacity+0x25/0x1d0
[140902.549050]  ? cpumask_next_and+0x1a/0x20
[140902.549053]  ? aa_file_perm+0x113/0x480
[140902.549116]  zpl_iter_write+0xda/0x140 [zfs]
[140902.563264]  io_write+0xee/0x2f0
[140902.563265]  ? cpumask_next_and+0x1a/0x20
[140902.563266]  ? load_balance+0x1b2/0xd40
[140902.563268]  ? update_load_avg+0x7a/0x5e0
[140902.563270]  ? dequeue_entity+0xc6/0x450
[140902.563273]  io_issue_sqe+0x793/0xf20
[140902.589314]  ? __switch_to_asm+0x42/0x70
[140902.589317]  ? __switch_to+0x114/0x450
[140902.589319]  io_wq_submit_work+0x50/0x1a0
[140902.589322]  io_worker_handle_work+0x1a1/0x570
[140902.607627]  io_wqe_worker+0x2a9/0x350
[140902.607629]  ? io_worker_handle_work+0x570/0x570
[140902.607631]  kthread+0x11b/0x140
[140902.607635]  ? __kthread_bind_mask+0x60/0x60
[140902.625652]  ret_from_fork+0x22/0x30
 

Attachments

  • scale-tasks.JPG
    scale-tasks.JPG
    50.6 KB · Views: 185
Last edited:

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
It would be good to get background info:

What version of software? Did this happen after upgrade?
Is it the same from multiple browsers?
 

cyrus104

Explorer
Joined
Feb 7, 2021
Messages
70
@morganL you are absolutely right, I had a baby on my lap and was in a rush. Sorry about leaving out details, updating the first post.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
@cyrus104 You should file a Jira ticket and/or report this to the OpenZFS Github ;-)

I don't think we've had any similar reports. (please chime in if any one else has seen this)

I'd recommend that reboot be done 1st and check that hardware is in a good state (Is it ECC RAM?).
 

cyrus104

Explorer
Joined
Feb 7, 2021
Messages
70
Because my nfs shares are still working, no errors from the hosted vms I have put off rebooting so far.
This system does have 128gb of ddr4 ecc memory.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Its reasonable for you to keep going. However, its only after reboot that we would know if the problem is reproducable. Perhaps just lets us know after the next reboot.. whenever it is. If anyone else experiences the same problem, they might read this. However, I would recommend the thread title be changed to something more specific like:

"Frozen WebUI and io_wqe_worker blocked"
 

cyrus104

Explorer
Joined
Feb 7, 2021
Messages
70
I have more info on the Jira ticket but after something finished... the webui seems to be working fairly normal. I was able to get it to complete a debug download. I'm happy to change the name.. I was hoping for one I'd understand. ;-)
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
I changed it to "WebUi temporarily frozen".... please change if you think of something better.
 

oumpa31

Patron
Joined
Apr 7, 2015
Messages
253
I actually have mine frozen im about to do a clean install of it on my test machine to see if that fixes it.
 
Top