Random Shutdown - Not crashing?

-cj- · Apr 20, 2023

System:
TrueNAS Scale 22.12.2 with 12 cores assigned (Host has 48 logical cores, is not busy)
VMWare ESX VM, 64GB ECC assigned, 2x LSI2008 and 2x Mellanox ConnectX5 SR-IOV, 1x VMXNET3, 1x RDMA adapter

This setup was upgraded from TrueNAS CORE, which ran flawlessly without crashes for many years.

I have configured the system to use 50GB of ram for the L2 Cache.

When I login via the web interface, more often than not it will cause a graceful shut down of the VM. When I do this, my SSH session gets terminated and I see the VM going through the shut down stages on the console. I managed to capture some quick outputs before it dropped me from SSH (listed below).

I logged in twice to the UI after the system returned and it shut down twice in a row. It did this the other day as well. It's not consistent. I logged in with a private web browsing window and it did not shut down, but frankly I'm afraid to test the limits without understanding this better.

When it gracefully shuts down, I need to power up the VM again. It doesn't SEEM to be memory related, since I have a considerable amount of available RAM in the VM (7.4GB as per the cut and paste below), not to mention tons of unused swap.

Only thing that stands out is " /bin/sh /etc/init.d/scst stop" process being executed.

Any tips?
Edit: I had to delete a bunch of the processes in the ps aux output since the forum was warning me about 40k+ line limit. I deleted random kernel processes from the output.

Code:

user@nas:/$ ps auxw | grep init
root           1  0.2  0.0 169836 12912 ?        Ss   12:33   0:04 /sbin/init
root       59047  0.0  0.0      0     0 ?        S<   12:35   0:00 [scst_initd]
root     1446861  0.0  0.0   2484  1572 ?        Ss   13:02   0:00 /bin/sh /etc/init.d/scst stop
user   1597603  0.0  0.0   5136  2240 pts/0    S+   13:02   0:00 grep init
user@nas:/$ free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        54Gi       7.7Gi        21Mi       230Mi       7.4Gi
Swap:            9Gi          0B         9Gi
user@nas:/$ ps auxw
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.2  0.0 169836 12912 ?        Ss   12:33   0:04 /sbin/init
root           2  7.5  0.0      0     0 ?        S    12:33   2:13 [kthreadd]
root           3  0.0  0.0      0     0 ?        I<   12:33   0:00 [rcu_gp]
root           4  0.0  0.0      0     0 ?        I<   12:33   0:00 [rcu_par_gp]
root           5  0.0  0.0      0     0 ?        I<   12:33   0:00 [slub_flushwq]
root           6  0.0  0.0      0     0 ?        I<   12:33   0:00 [netns]
root           8  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/0:0H-events_highpri]
root          10  0.0  0.0      0     0 ?        I<   12:33   0:00 [mm_percpu_wq]
root          11  0.0  0.0      0     0 ?        S    12:33   0:00 [rcu_tasks_rude_]
root          12  0.0  0.0      0     0 ?        S    12:33   0:00 [rcu_tasks_trace]
root          13  0.0  0.0      0     0 ?        S    12:33   0:01 [ksoftirqd/0]
root          14  0.0  0.0      0     0 ?        I    12:33   0:01 [rcu_sched]
root          15  0.0  0.0      0     0 ?        S    12:33   0:00 [migration/0]
root          17  0.0  0.0      0     0 ?        S    12:33   0:00 [cpuhp/0]
root          18  0.0  0.0      0     0 ?        S    12:33   0:00 [cpuhp/1]
root          19  0.0  0.0      0     0 ?        S    12:33   0:00 [migration/1]
root          20  0.0  0.0      0     0 ?        S    12:33   0:01 [ksoftirqd/1]
root          22  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/1:0H-events_highpri]
root          23  0.0  0.0      0     0 ?        S    12:33   0:00 [cpuhp/2]
root          24  0.0  0.0      0     0 ?        S    12:33   0:00 [migration/2]
root          25  0.1  0.0      0     0 ?        S    12:33   0:02 [ksoftirqd/2]

root         351  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5_cmd_0000:1]
root         352  0.0  0.0      0     0 ?        S    12:33   0:00 [scsi_eh_4]
root         353  0.0  0.0      0     0 ?        I<   12:33   0:00 [scsi_tmf_4]
root         354  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5_events]
root         355  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5_fw_reset_e]
root         356  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5_hv_vhca]
root         357  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5_fc]
root         358  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5e]
root         359  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5e_arfs]
root         360  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5e]
root         361  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5e_arfs]
root         362  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5e]
root         363  0.0  0.0      0     0 ?        I<   12:33   0:00 [mlx5e_arfs]
root         364  0.0  0.0      0     0 ?        I<   12:33   0:00 [poll_mpt2sas0_s]
root         365  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/2:1H-kblockd]
root         366  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/8:1H-kblockd]
root         367  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/9:1H-kblockd]
root         368  0.0  0.0      0     0 ?        I<   12:33   0:00 [poll_mpt2sas1_s]
root         371  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib-comp-wq]
root         372  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib-comp-unb-wq]
root         373  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib_mcast]
root         374  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib_nl_sa_wq]
root         389  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/11:1H-kblockd]
root         391  0.0  0.0      0     0 ?        I<   12:33   0:00 [kworker/3:1H-kblockd]
root         417  0.0  0.0      0     0 ?        I    12:33   0:00 [kworker/7:2-events]
root         428  0.0  0.0      0     0 ?        I<   12:33   0:00 [md]
root         440  0.0  0.0      0     0 ?        I    12:33   0:00 [kworker/9:2-events]
root         462  0.0  0.0      0     0 ?        S    12:33   0:00 [0:0:16:0]
root         465  0.0  0.0      0     0 ?        S    12:33   0:00 [3:0:8:0]
root         487  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib_mad1]
root         493  0.0  0.0      0     0 ?        I<   12:33   0:00 [mkey_cache]
root         499  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib_mad1]
root         500  0.0  0.0      0     0 ?        I<   12:33   0:00 [mkey_cache]
root         501  0.0  0.0      0     0 ?        I<   12:33   0:00 [ib_mad1]
root         503  0.0  0.0      0     0 ?        I<   12:33   0:00 [mkey_cache]
root         543  0.0  0.0      0     0 ?        S    12:33   0:00 [md127_raid1]
root         548  0.0  0.0      0     0 ?        S    12:33   0:00 [md126_raid1]
root         556  0.0  0.0      0     0 ?        S    12:33   0:00 [md124_raid1]
root         557  0.0  0.0      0     0 ?        S    12:33   0:00 [md123_raid1]
root         580  0.0  0.0      0     0 ?        S    12:33   0:00 [md125_raid1]
root         597  0.0  0.0      0     0 ?        I<   12:33   0:00 [raid5wq]
root         620  0.0  0.0      0     0 ?        S<   12:33   0:00 [spl_system_task]
root         621  0.0  0.0      0     0 ?        S<   12:33   0:00 [spl_delay_taskq]
root         622  3.1  0.0      0     0 ?        S<   12:33   0:55 [spl_dynamic_tas]
root         623  0.3  0.0      0     0 ?        S<   12:33   0:05 [spl_kmem_cache]
root         624  0.0  0.0      0     0 ?        S<   12:33   0:00 [zvol]
root         625  0.0  0.0      0     0 ?        S    12:33   0:00 [arc_prune]
root         626  0.0  0.0      0     0 ?        S    12:33   0:01 [arc_evict]
root         627  0.0  0.0      0     0 ?        SN   12:33   0:00 [arc_reap]
root         628  0.0  0.0      0     0 ?        S    12:33   0:00 [dbu_evict]
root         629  0.2  0.0      0     0 ?        SN   12:33   0:05 [dbuf_evict]
root         630  0.0  0.0      0     0 ?        SN   12:33   0:00 [z_vdev_file]
root         631  0.2  0.0      0     0 ?        S    12:33   0:03 [l2arc_feed]
root         738  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_null_iss]
root         739  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_null_int]
root         740  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_rd_iss]
root         741  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_rd_int_0]
root         742  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_rd_int_1]
root         743  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_wr_iss]
root         744  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_wr_iss_h]
root         745  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_wr_int_0]
root         746  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_wr_int_1]
root         747  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_wr_int_h]
root         748  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_fr_iss_0]
root         749  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_fr_iss_1]
root         750  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_fr_int]
root         751  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_cl_iss]
root         752  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_cl_int]
root         753  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_ioctl_iss]
root         754  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_ioctl_int]
root         755  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_trim_iss]
root         756  0.0  0.0      0     0 ?        S<   12:33   0:00 [z_trim_int]
root         757  0.0  0.0      0     0 ?        S    12:33   0:00 [z_zvol]
root         758  0.0  0.0      0     0 ?        S    12:33   0:00 [z_prefetch]
root         759  0.0  0.0      0     0 ?        S    12:33   0:00 [z_upgrade]
root         762  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         763  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         764  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         765  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         766  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         767  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         768  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         769  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         770  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_sync_taskq]
root         771  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         772  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         773  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         774  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         775  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         776  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         777  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         778  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         779  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         780  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         781  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         782  0.0  0.0      0     0 ?        SN   12:33   0:00 [dp_zil_clean_ta]
root         783  0.0  0.0      0     0 ?        S    12:33   0:00 [z_zrele]
root         784  0.0  0.0      0     0 ?        S    12:33   0:00 [z_unlinked_drai]
root         788  0.0  0.0      0     0 ?        S<   12:33   0:00 [metaslab_group_]
root         789  0.0  0.0      0     0 ?        S<   12:33   0:00 [metaslab_group_]
root         830  0.0  0.0      0     0 ?        S    12:33   0:00 [txg_quiesce]
root         831  0.0  0.0      0     0 ?        S    12:33   0:00 [txg_sync]
root         832  0.0  0.0      0     0 ?        S    12:33   0:00 [mmp]
root         893  0.0  0.0      0     0 ?        SN   12:33   0:00 [z_indirect_cond]
root         894  0.0  0.0      0     0 ?        SN   12:33   0:00 [z_livelist_dest]
root         895  0.0  0.0      0     0 ?        SN   12:33   0:00 [z_livelist_cond]
root         896  0.0  0.0      0     0 ?        SN   12:33   0:00 [z_checkpoint_di]
message+    1129  0.0  0.0   8572  3316 ?        Ss   12:34   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd
root        1136  0.0  0.0      0     0 ?        I<   12:34   0:00 [rpciod]
root        1137  0.0  0.0      0     0 ?        I<   12:34   0:00 [xprtiod]
root        1298  0.0  0.0  22820  5588 ?        Ss   12:34   0:00 /lib/systemd/systemd-udevd
root        2288  0.0  0.0      0     0 ?        I    12:34   0:01 [kworker/5:2-ata_sff]
root        2667  0.0  0.0      0     0 ?        I<   12:34   0:00 [ttm_swap]
root        2670  0.0  0.0      0     0 ?        S    12:34   0:00 [irq/16-vmwgfx]
root        2685  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc0]
root        2686  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc1]
root        2687  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc2]
root        2688  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc3]
root        2689  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc4]
root        2692  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc5]
root        2693  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc6]
root        2694  0.0  0.0      0     0 ?        S    12:34   0:00 [card0-crtc7]
root        7453  0.0  0.0      0     0 ?        I<   12:34   0:00 [kdmflush]
root        7471  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd_io/253:]
root        7472  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd/253:0]
root        7473  0.0  0.0      0     0 ?        S    12:34   0:00 [dmcrypt_write/2]
root        7530  0.0  0.0      0     0 ?        I<   12:34   0:00 [kdmflush]
root        7531  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd_io/253:]
root        7532  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd/253:1]
root        7533  0.0  0.0      0     0 ?        S    12:34   0:00 [dmcrypt_write/2]
root        7553  0.0  0.0      0     0 ?        I<   12:34   0:00 [kdmflush]
root        7554  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd_io/253:]
root        7555  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd/253:2]
root        7556  0.0  0.0      0     0 ?        S    12:34   0:00 [dmcrypt_write/2]
root        7571  0.0  0.0      0     0 ?        I<   12:34   0:00 [kdmflush]
root        7572  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd_io/253:]
root        7573  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd/253:3]
root        7574  0.0  0.0      0     0 ?        S    12:34   0:00 [dmcrypt_write/2]
root        7624  0.0  0.0      0     0 ?        I<   12:34   0:00 [kdmflush]
root        7625  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd_io/253:]
root        7626  0.0  0.0      0     0 ?        I<   12:34   0:00 [kcryptd/253:4]
root        7627  0.0  0.0      0     0 ?        S    12:34   0:00 [dmcrypt_write/2]
root        7644  0.1  0.0      0     0 ?        I    12:34   0:03 [kworker/8:2-mpt_poll_0]
root        7645  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_null_iss]
root        7646  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_null_int]
root        7647  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_iss]
root        7648  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_int_0]
root        7649  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_int_1]
root        7650  0.1  0.0      0     0 ?        S<   12:34   0:02 [z_wr_iss]
root        7651  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_iss_h]
root        7652  0.0  0.0      0     0 ?        S<   12:34   0:01 [z_wr_int_0]
root        7653  0.0  0.0      0     0 ?        S<   12:34   0:01 [z_wr_int_1]
root        7654  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_int_h]
root        7655  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_iss_0]
root        7656  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_iss_1]
root        7657  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_int]
root        7658  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_cl_iss]
root        7659  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_cl_int]
root        7660  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_ioctl_iss]
root        7661  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_ioctl_int]
root        7662  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_trim_iss]
root        7663  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_trim_int]
root        7664  0.0  0.0      0     0 ?        S    12:34   0:00 [z_zvol]
root        7665  0.0  0.0      0     0 ?        S    12:34   0:00 [z_prefetch]
root        7666  0.0  0.0      0     0 ?        S    12:34   0:00 [z_upgrade]

root        7699  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]
root        7700  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]
root        7701  0.0  0.0      0     0 ?        S    12:34   0:00 [z_zrele]
root        7702  0.0  0.0      0     0 ?        S    12:34   0:00 [z_unlinked_drai]
root        7717  0.0  0.0      0     0 ?        S<   12:34   0:00 [metaslab_group_]
root        7719  0.0  0.0      0     0 ?        S<   12:34   0:00 [metaslab_group_]
root        7720  0.0  0.0      0     0 ?        S<   12:34   0:00 [metaslab_group_]
root        7862  0.0  0.0      0     0 ?        S    12:34   0:00 [txg_quiesce]
root        7863  0.0  0.0      0     0 ?        S    12:34   0:01 [txg_sync]
root        7864  0.0  0.0      0     0 ?        S    12:34   0:00 [mmp]
root        8043  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_indirect_cond]
root        8044  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_livelist_dest]
root        8045  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_livelist_cond]
root        8046  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_checkpoint_di]
root        9799  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_null_iss]
root        9800  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_null_int]
root        9801  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_iss]
root        9802  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_int_0]
root        9803  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_int_1]
root        9804  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_iss]
root        9805  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_iss_h]
root        9806  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_int_0]
root        9807  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_int_1]
root        9808  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_int_h]
root        9809  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_iss_0]
root        9810  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_iss_1]
root        9811  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_int]
root        9812  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_cl_iss]
root        9813  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_cl_int]
root        9814  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_ioctl_iss]
root        9815  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_ioctl_int]
root        9816  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_trim_iss]
root        9817  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_trim_int]
root        9818  0.0  0.0      0     0 ?        S    12:34   0:00 [z_zvol]
root        9819  0.0  0.0      0     0 ?        S    12:34   0:00 [z_prefetch]
root        9820  0.0  0.0      0     0 ?        S    12:34   0:00 [z_upgrade]
root        9832  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_sync_taskq]
root        9833  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_sync_taskq]

root        9841  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]
root        9842  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]

root        9850  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]
root        9851  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]
root        9852  0.0  0.0      0     0 ?        SN   12:34   0:00 [dp_zil_clean_ta]
root        9853  0.0  0.0      0     0 ?        S    12:34   0:00 [z_zrele]
root        9854  0.0  0.0      0     0 ?        S    12:34   0:00 [z_unlinked_drai]
root        9874  0.0  0.0      0     0 ?        S<   12:34   0:00 [metaslab_group_]
root        9875  0.0  0.0      0     0 ?        S<   12:34   0:00 [metaslab_group_]
root       10168  0.0  0.0      0     0 ?        S    12:34   0:00 [txg_quiesce]
root       10169  0.0  0.0      0     0 ?        S    12:34   0:00 [txg_sync]
root       10170  0.0  0.0      0     0 ?        S    12:34   0:00 [mmp]
root       10272  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_indirect_cond]
root       10273  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_livelist_dest]
root       10274  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_livelist_cond]
root       10275  0.0  0.0      0     0 ?        SN   12:34   0:00 [z_checkpoint_di]
root       10943  0.0  0.0      0     0 ?        S<   12:34   0:01 [z_null_iss]
root       10944  0.4  0.0      0     0 ?        S<   12:34   0:07 [z_null_int]
root       10946  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_rd_iss]
root       10948 12.8  0.0      0     0 ?        R<   12:34   3:39 [z_rd_int_0]
root       10950 12.8  0.0      0     0 ?        S<   12:34   3:39 [z_rd_int_1]
root       10952  0.3  0.0      0     0 ?        S<   12:34   0:06 [z_wr_iss]
root       10954  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_iss_h]
root       10956  2.1  0.0      0     0 ?        S<   12:34   0:36 [z_wr_int_0]
root       10957  2.1  0.0      0     0 ?        S<   12:34   0:36 [z_wr_int_1]
root       10958  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_wr_int_h]
root       10959  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_iss_0]
root       10960  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_iss_1]
root       10961  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_fr_int]
root       10962  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_cl_iss]
root       10965  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_cl_int]
root       10966  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_ioctl_iss]
root       10967  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_ioctl_int]
root       10968  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_trim_iss]
root       10970  0.0  0.0      0     0 ?        S<   12:34   0:00 [z_trim_int]
root       10971  0.0  0.0      0     0 ?        S    12:34   0:00 [z_zvol]
root       10972  0.0  0.0      0     0 ?        S    12:34   0:00 [z_prefetch]
root       10974  0.0  0.0      0     0 ?        S    12:34   0:00 [z_upgrade]
root       11016  0.7  0.0      0     0 ?        SN   12:34   0:13 [dp_sync_taskq]
root       11017  0.2  0.0      0     0 ?        SN   12:34   0:04 [dp_sync_taskq]
root       11018  0.1  0.0      0     0 ?        SN   12:34   0:02 [dp_sync_taskq]
root       11019  0.3  0.0      0     0 ?        SN   12:34   0:06 [dp_sync_taskq]

root       51001  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_sync_taskq]
root       51002  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_sync_taskq]
root       51003  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_sync_taskq]
root       51004  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_sync_taskq]
root       51005  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51006  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51007  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51008  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51009  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51010  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51011  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51012  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51013  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51014  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51015  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51016  0.0  0.0      0     0 ?        SN   12:35   0:00 [dp_zil_clean_ta]
root       51017  0.0  0.0      0     0 ?        S    12:35   0:00 [z_zrele]
root       51018  0.0  0.0      0     0 ?        S    12:35   0:00 [z_unlinked_drai]
root       51031  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51032  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51033  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51034  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51035  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51036  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51037  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51038  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51114  0.0  0.0      0     0 ?        S    12:35   0:00 [txg_quiesce]
root       51115  0.0  0.0      0     0 ?        S    12:35   0:00 [txg_sync]
root       51116  0.0  0.0      0     0 ?        S    12:35   0:00 [mmp]
root       51117  0.0  0.0      0     0 ?        SN   12:35   0:00 [z_indirect_cond]
root       51118  0.0  0.0      0     0 ?        SN   12:35   0:00 [z_livelist_dest]
root       51119  0.0  0.0      0     0 ?        SN   12:35   0:00 [z_livelist_cond]
root       51120  0.0  0.0      0     0 ?        SN   12:35   0:00 [z_checkpoint_di]
root       51121  0.0  0.0      0     0 ?        S<   12:35   0:00 [vdev_autotrim]
root       51122  0.0  0.0      0     0 ?        S<   12:35   0:00 [vdev_autotrim]
root       51123  0.0  0.0      0     0 ?        S<   12:35   0:00 [vdev_autotrim]
root       51124  0.0  0.0      0     0 ?        S<   12:35   0:00 [vdev_autotrim]
root       51149  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51150  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       51155  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       55411  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_iss]
root       55412  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_iss]
root       55414  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_iss]
root       55415  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_iss]
root       55417  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_iss]
root       55442  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_int_0]
root       55443  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_int_0]
root       55445  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_int_1]
root       55446  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_wr_int_1]
root       55462  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       55465  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_rd_int_1]
root       55467  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_rd_int_1]
root       55470  0.0  0.0      0     0 ?        S<   12:35   0:00 [z_rd_int_0]
root       55471  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       57749  0.0  0.0  25948 16564 ?        Ss   12:35   0:00 /lib/systemd/systemd-journald
_rpc       57851  0.0  0.0   7904  3256 ?        Ss   12:35   0:00 /sbin/rpcbind -f -w
root       57911  0.0  0.0      0     0 ?        I<   12:35   0:00 [kworker/u25:1]
root       58812  0.0  0.0      0     0 ?        I<   12:35   0:00 [scst_release_ac]
root       58817  0.0  0.0      0     0 ?        I<   12:35   0:00 [scst_event_wq]
root       58819  0.0  0.0      0     0 ?        S<   12:35   0:00 [scst_uid]
root       59035  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd0]
root       59036  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd1]
root       59037  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd2]
root       59038  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd3]
root       59039  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd4]
root       59040  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd5]
root       59041  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd6]
root       59042  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd7]
root       59043  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd8]
root       59044  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd9]
root       59045  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd10]
root       59046  0.0  0.0      0     0 ?        S    12:35   0:00 [scstd11]
root       59047  0.0  0.0      0     0 ?        S<   12:35   0:00 [scst_initd]
root       59048  0.0  0.0      0     0 ?        S<   12:35   0:00 [scsi_tm]
root       59049  0.0  0.0      0     0 ?        S<   12:35   0:01 [scst_mgmtd]
root       59879  0.0  0.0      0     0 ?        S    12:35   0:00 [vm-storage0_0]
root       61673  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       61675  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       61676  0.0  0.0      0     0 ?        S<   12:35   0:00 [metaslab_group_]
root       63851  0.0  0.0  13648  7036 ?        Ss   12:36   0:00 sshd: user [priv]
user     64764  0.0  0.0  13960  5832 ?        R    12:36   0:00 sshd: user@pts/0
user     64854  0.0  0.0   7208  4708 pts/0    Ss   12:36   0:00 -bash
root       82897  0.0  0.0      0     0 ?        S    12:36   0:00 [vm-storage0_0]
root       82900  0.0  0.0      0     0 ?        S    12:36   0:00 [vm-storage0_0]
root       82903  0.0  0.0      0     0 ?        S    12:36   0:00 [vm-storage0_0]
root       83260  0.0  0.0      0     0 ?        S    12:36   0:00 [vm-storage0_0]
root       96269 11.0  0.0      0     0 ?        SN   12:37   2:52 [dsl_scan_iss]
root       96270 10.9  0.0      0     0 ?        RN   12:37   2:52 [dsl_scan_iss]
root      250394  0.0  0.0      0     0 ?        I    12:56   0:00 [kworker/4:0-events]
root      261115  0.1  0.0      0     0 ?        I    12:56   0:00 [kworker/6:2-events]
root      477565  0.0  0.0      0     0 ?        I    12:57   0:00 [kworker/1:1-events]
root      512531  0.0  0.0      0     0 ?        I    12:57   0:00 [kworker/5:1-ata_sff]
root      771793  0.1  0.0      0     0 ?        I    12:58   0:00 [kworker/5:3-events]
root     1021415  0.0  0.0      0     0 ?        I    13:00   0:00 [kworker/8:0-events]
root     1112567  0.0  0.0      0     0 ?        S<   13:00   0:00 [metaslab_group_]
root     1136890  0.0  0.0      0     0 ?        I    13:00   0:00 [kworker/2:1-cgroup_destroy]
root     1137073  0.1  0.0      0     0 ?        I    13:00   0:00 [kworker/u24:0-flush-253:3]
root     1298193  0.0  0.0      0     0 ?        S<   13:01   0:00 [metaslab_group_]
root     1379998  0.0  0.0      0     0 ?        S<   13:01   0:00 [metaslab_group_]
root     1401179  0.0  0.0      0     0 ?        S<   13:01   0:00 [metaslab_group_]
root     1413816  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1413817  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1414310  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1414311  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1414312  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1414313  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1414314  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1414795  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_int_1]
root     1428649  0.0  0.0      0     0 ?        S<   13:02   0:00 [metaslab_group_]
root     1436599  0.0  0.0      0     0 ?        S<   13:02   0:00 [metaslab_group_]
root     1446529  0.0  0.0      0     0 ?        S<   13:02   0:00 [zvol]
root     1446530  0.0  0.0      0     0 ?        S<   13:02   0:00 [zvol]
root     1446850  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:2-mlx5e]
root     1446858  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:3-events_unbound]
root     1446861  0.0  0.0   2484  1572 ?        Ss   13:02   0:00 /bin/sh /etc/init.d/scst stop
root     1446870  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:4-flush-253:4]
root     1446871  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/10:2-events]
root     1446873  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:5-events_unbound]
root     1446876  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:6-mlx5_cmd_0000:0b:00.0]
root     1446879  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:7-events_unbound]
root     1446884  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:8]
root     1446885  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/u24:9-events_unbound]
root     1447075  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/0:0-events]
root     1449021  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449054  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449055  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449056  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449057  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449058  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449059  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1449305  0.0  0.0      0     0 ?        S<   13:02   0:00 [metaslab_group_]
root     1449766  0.0  0.0   8604  3908 ?        D    13:02   0:00 modprobe -r scst_vdisk
root     1483787  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/2:3]
root     1484517  0.0  0.0      0     0 ?        I    13:02   0:00 [kworker/4:1-events]
root     1511522  0.0  0.0      0     0 ?        S<   13:02   0:00 [metaslab_group_]
root     1608295  0.0  0.0      0     0 ?        S<   13:02   0:00 [z_wr_iss]
root     1608382  0.0  0.0      0     0 ?        S<   13:02   0:00 [metaslab_group_]
root     1621321  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1621323  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1621325  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1621328  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1621339  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1621340  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1621354  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_int_1]
root     1660049  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1660052  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1660091  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1660092  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1660093  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
root     1660095  0.0  0.0      0     0 ?        S<   13:03   0:00 [z_wr_iss]
user   1673212  0.0  0.0   8652  2984 pts/0    R+   13:03   0:00 ps auxw
root     2655872  0.0  0.0      0     0 ?        I    12:48   0:00 [kworker/0:1-events]
root     2730406  0.0  0.0      0     0 ?        I    12:48   0:00 [kworker/11:0-events]
root     2889129  0.0  0.0      0     0 ?        I    12:49   0:00 [kworker/2:2-events]
root     2979228  0.0  0.0      0     0 ?        I    12:50   0:00 [kworker/10:0-events]
root     3156833  0.0  0.0      0     0 ?        S<   12:50   0:00 [z_rd_iss]
root     3314277  0.1  0.0      0     0 ?        I    12:51   0:00 [kworker/1:2-events]
root     3482883  0.1  0.0      0     0 ?        I    12:52   0:00 [kworker/3:0-events]
root     3559988  0.0  0.0      0     0 ?        I    12:52   0:00 [kworker/5:0-ata_sff]
root     4032085  0.1  0.0      0     0 ?        I    12:54   0:00 [kworker/0:2-events]
root     4034034  0.0  0.0      0     0 ?        I    12:54   0:00 [kworker/2:0-events]
root     4142363  0.0  0.0      0     0 ?        I    12:55   0:00 [kworker/u24:1-mlx5_cmd_0000:0b:00.0]
user@nas:/$ Connection to nas closed by remote host.
Connection to nas closed.

WN1X · Apr 20, 2023

Try another web browser. Which browser were you using? What URL are you using to access your TrueNAS server?

-cj- · Apr 20, 2023

WN1X said:
Try another web browser. Which browser were you using? What URL are you using to access your TrueNAS server?

Using Safari on the latest MacOS. The URL it's accessing is the hostname and/or hostname/ui/sessions/signin

Both times when it shut down it dropped me into the Dashboard. But I'll keep an eye for any lingering arguments in the URL, but I don't think that's it.

-cj- · Apr 21, 2023

Update: It did it again. This time I was logged in to the UI for about a minute or two, sitting on the "Storage" screen. The WebUI browser was in the background when it shut down. REALLLLLLLY ANNOYING. I wonder if there's a way to break the shut down sequence temporarily?

Edit: Or better yet, any way to increase the logging to find out what's happening? I've combed through the /var/log logs and nothing indicates that there's an issue or what's triggering the shutdown.

Samuel Tai · Apr 21, 2023

I've seen hard-to-debug shutdown issues caused by flaky keyboards randomly spewing the ACPI shutdown code. How is your VM configured, USB-wise?

-cj- · Apr 21, 2023

Samuel Tai said:
I've seen hard-to-debug shutdown issues caused by flaky keyboards randomly spewing the ACPI shutdown code. How is your VM configured, USB-wise?

Thanks for the reply. Honestly, nothing special - Same general VM setup as my other Linux VMs. I've attached screen shots below, but nothing stands out. In fact this last time I wasn't even doing anything on TrueNAS after logging in and going to the "Storage" tab. I switched to another window and a minute or so later I noticed it was shutting down. I also made sure I didn't have any weird arguments in the URI.

The only thing that's "weird" about this VM is that my fd0 device gets accessed. I'm not even passing it to the VM, but I've been meaning to disable it. There aren't any fd0 errors around the time TrueNAS shuts down, so I dont think it's related.

Edit: Just to disclose everything I'm doing, I ssh into TrueNAS as my admin account, I issue "su -", and I monitor my zpool status since I'm currently resilvering a drive that's been giving me an issue on a Z2 pool. I open a browser window to TrueNAS and immediately try to get out of the Dashboard screen since that's where shutdowns usually occur. This last time I clicked on "Storage" and moved away to another browser window. When it shuts down I either notice because the UI says that it's waiting for the controller to come up or I notice my SSH session being terminated in my terminal window.

So far, I haven't had one of these shut downs occur when I'm using a private browser window, so I'm going to see if that continues to be the case.. But I'm scared to use the UI at this point!

(Sorry for all the details)

WN1X · Apr 22, 2023

Try a different web browser to help isolate the issue.

-cj- · Apr 22, 2023

WN1X said:
Try a different web browser to help isolate the issue.

Good idea. I'm using Edge for now and so far it isn't triggering a shut down. Safari has been consistently shutting down TrueNAS for whatever reason. Will update.

Important Announcement for the TrueNAS Community.

Random Shutdown - Not crashing?

-cj-

Cadet

WN1X

Explorer

-cj-

Cadet

-cj-

Cadet

Samuel Tai

Never underestimate your own stupidity

-cj-

Cadet

Attachments

WN1X

Explorer

-cj-

Cadet

Similar threads

Important Announcement for the TrueNAS Community.

Random Shutdown - Not crashing?

Cadet

Explorer

Cadet

Cadet

Never underestimate your own stupidity

Cadet

Attachments

Explorer

Cadet

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Random Shutdown - Not crashing?"

Similar threads