SOLVED Shell/SMB/Console hangs when accessing a directory

[mcmuller]

Hi folks,
I have a strange problem: While pool is fine and zfs-tools not finding any fault, I cannot access one particular directory in this pool. All neighbour directories are fine, but when I try to "ls" or "ls -l" shell hangs, console hangs, smb-share hangs infinite. Tried to move/copy/delete this directory or files inside, but without luck and the sam symtom. Log shows nothing most of the time, but once I could see output like this:

Code:

May 16 18:25:05 freenas kernel: task:smbd            state:D stack:    0 pid:3349571 ppid:  7057 flags:0x00000000
May 16 18:25:05 freenas kernel: Call Trace:
May 16 18:25:05 freenas kernel:  __schedule+0x282/0x870
May 16 18:25:05 freenas kernel:  schedule+0x46/0xb0
May 16 18:25:06 freenas kernel:  vcmn_err.cold+0x7e/0x80 [spl]
May 16 18:25:06 freenas kernel:  ? kmem_cache_alloc+0xed/0x1f0
May 16 18:25:06 freenas kernel:  ? sa_cache_constructor+0x23/0x40 [zfs]
May 16 18:25:06 freenas kernel:  ? spl_kmem_cache_alloc+0xaf/0x7c0 [spl]
May 16 18:25:06 freenas kernel:  ? spl_kmem_cache_alloc+0xaf/0x7c0 [spl]
May 16 18:25:06 freenas kernel:  ? _cond_resched+0x16/0x40
May 16 18:25:06 freenas kernel:  ? mutex_lock+0xe/0x30
May 16 18:25:06 freenas kernel:  ? dmu_buf_set_user_ie+0x55/0x80 [zfs]
May 16 18:25:06 freenas kernel:  zfs_panic_recover+0x6d/0x90 [zfs]
May 16 18:25:06 freenas kernel:  zfs_znode_alloc+0x6df/0x740 [zfs]
May 16 18:25:06 freenas kernel:  zfs_zget+0x1ae/0x260 [zfs]
May 16 18:25:06 freenas kernel:  zfs_dirent_lock+0x365/0x6c0 [zfs]
May 16 18:25:06 freenas kernel:  zfs_dirlook+0x88/0x2c0 [zfs]
May 16 18:25:06 freenas kernel:  ? zfs_zaccess+0x22f/0x430 [zfs]
May 16 18:25:06 freenas kernel:  zfs_lookup+0x246/0x3f0 [zfs]
May 16 18:25:06 freenas kernel:  zpl_lookup+0xc5/0x1e0 [zfs]
May 16 18:25:06 freenas kernel:  __lookup_slow+0x84/0x140
May 16 18:25:06 freenas kernel:  walk_component+0x154/0x1d0
May 16 18:25:06 freenas kernel:  ? _cond_resched+0x16/0x40
May 16 18:25:06 freenas kernel:  ? path_init+0x1e2/0x3e0
May 16 18:25:06 freenas kernel:  path_lookupat+0x73/0x1c0
May 16 18:25:06 freenas kernel:  filename_lookup+0xaa/0x1b0
May 16 18:25:06 freenas kernel:  ? strncpy_from_user+0x4e/0x140
May 16 18:25:06 freenas kernel:  ? getname_flags.part.0+0x45/0x1a0
May 16 18:25:06 freenas kernel:  vfs_statx+0x74/0x130
May 16 18:25:06 freenas kernel:  ? __fput+0x100/0x240
May 16 18:25:06 freenas kernel:  __do_sys_newfstatat+0x31/0x70
May 16 18:25:06 freenas kernel:  ? exit_to_user_mode_prepare+0x32/0x120
May 16 18:25:06 freenas kernel:  do_syscall_64+0x33/0x80
May 16 18:25:06 freenas kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
May 16 18:25:06 freenas kernel: RIP: 0033:0x7f13b6b8887b
May 16 18:25:06 freenas kernel: RSP: 002b:00007fff291cf9c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000106
May 16 18:25:06 freenas kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f13b6b8887b
May 16 18:25:06 freenas kernel: RDX: 00007fff291cf9d0 RSI: 0000555fcd81773b RDI: 0000000000000048
May 16 18:25:06 freenas kernel: RBP: 00007fff291cfac0 R08: 0000000000000100 R09: 0000000000000001
May 16 18:25:06 freenas kernel: R10: 0000000000000100 R11: 0000000000000246 R12: 0000000000000000
May 16 18:25:06 freenas kernel: R13: 00007fff291cf9d0 R14: 00007fff291cfc00 R15: 0000555fcd817728

I'm on TrueNAS-SCALE-22.02.1, but problem was there before.

Ideas or steps to take anyone??

Greetings from Portugal,
Gregor

 

[Bl00dWolf]

Same here for me.
Only restarting SMB services helps for sometime.

And if you accsess SMB shera via IP, not vai maped network drive, all seems to be ok.

 

[anodos]

Can you send me a debug (system->advanced->save debug). I need to see permissions and share configuration.

 

[anodos]

mapped network drive vs IP access indicates lack of call to dfree for path in question (statfs / statvfs). This means I need to really see full debug.

 

[mcmuller]

Can you send me a debug (system->advanced->save debug). I need to see permissions and share configuration.

Thanks for taking care! Just out of curiosity, I disabled smb, restarted the system, ssh'd into the shell, did a 'ls' on the directory in question, which went fine and then did a 'ls -l' wich ended with a console-hang and the following console-message:

Code:

2022 May 17 16:56:26 freenas PANIC: inode 99224 has invalid mode: 0xfffe

debug is on the way....

 

[anodos]

Recap:
mcmuller was affected by following openzfs issue:

PANIC: inode has invalid mode: 0x0 · Issue #11474 · openzfs/zfs

System information Type Version/Name Distribution Name Archlinux Distribution Version rolling Linux Kernel 5.10.6-arch1-1 Architecture x86_64 ZFS Version 2.0.1-1 SPL Version 2.0.1-1 Describe the pr...

github.com

There was a period of time when TrueNAS was potentially impacted a while back. See timeline for commit history / fix in openzfs issue.
He rebooted so that openzfs module was loaded cleanly, then ran following command:

echo 1 > /sys/module/zfs/parameters/zfs_recover

This disabled panicing on reading mode of affected file. Once he did this he was able to identify exact file by running command:
ls -li <dir where file located> | grep <inode number>
Once file identified it may be removed.

 

[mcmuller]

Thanks for resolving this one! The file needs to be removed, chmod does not cure.