Can't delete files after upgrading to 13.0-U3.1

[bazzacad]

Hi,
I just upgraded from v12 to v13.0-U3.1.
After doing so, I can no longer delete files from one of my SMB shares, the other shares seem to be fine.
When I delete the file in Windows Explorer, it just shows up again after doing a refresh.
Here are the permisions:

If I view the log.smbd file after trying to delete a file, I get these errors:

[2023/01/10 10:24:21.625234, 0] ../../lib/util/fault.c:176(smb_panic_log)
INTERNAL ERROR: Mountpath is incorrect in pid 83835 (4.15.11)
[2023/01/10 10:24:21.625244, 0] ../../lib/util/fault.c:181(smb_panic_log)
If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.sam
ba.org/index.php/Bug_Reporting
[2023/01/10 10:24:21.625285, 0] ../../lib/util/fault.c:182(smb_panic_log)
===============================================================
[2023/01/10 10:24:21.625294, 0] ../../lib/util/fault.c:184(smb_panic_log)
PANIC (pid 83835): Mountpath is incorrect in 4.15.11
[2023/01/10 10:24:21.626584, 0] ../../lib/util/fault.c:288(log_stack_trace)
BACKTRACE: 25 stack frames:
#0 0x8014ab8c7 <log_stack_trace+0x37> at /usr/local/lib/samba4/libsamba-util.so.0
#1 0x8014ab9a1 <smb_panic+0x11> at /usr/local/lib/samba4/libsamba-util.so.0
#2 0x8162c85db <samba_init_module+0x29cb> at /usr/local/lib/shared-modules/vfs/recycle.so
#3 0x8162c625d <samba_init_module+0x64d> at /usr/local/lib/shared-modules/vfs/recycle.so
#4 0x801aeb2b6 <close_file+0x1fa6> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#5 0x801ae9449 <close_file+0x139> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#6 0x801b23dc8 <smbd_smb2_request_process_close+0x828> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#7 0x801b237ea <smbd_smb2_request_process_close+0x24a> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#8 0x801b15edc <smbd_smb2_request_dispatch+0x17ec> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#9 0x801b1a622 <smbd_smb2_process_negprot+0x24f2> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#10 0x80277c5cd <tevent_common_invoke_fd_handler+0x9d> at /usr/local/lib/samba4/private/libtevent.so.0
#11 0x8027829ad <tevent_ctx_get_iocb+0x73d> at /usr/local/lib/samba4/private/libtevent.so.0
#12 0x802781220 <tevent_signal_get_tag+0x230> at /usr/local/lib/samba4/private/libtevent.so.0
#13 0x80277b5e1 <_tevent_loop_once+0xe1> at /usr/local/lib/samba4/private/libtevent.so.0
#14 0x80277b842 <tevent_common_loop_wait+0x32> at /usr/local/lib/samba4/private/libtevent.so.0
#15 0x8027812a0 <tevent_signal_get_tag+0x2b0> at /usr/local/lib/samba4/private/libtevent.so.0
#16 0x801b038e3 <smbd_process+0x7b3> at /usr/local/lib/samba4/private/libsmbd-base-samba4.so
#17 0x1032373 <main+0x44b3> at /usr/local/sbin/smbd
#18 0x80277c5cd <tevent_common_invoke_fd_handler+0x9d> at /usr/local/lib/samba4/private/libtevent.so.0
#19 0x8027829ad <tevent_ctx_get_iocb+0x73d> at /usr/local/lib/samba4/private/libtevent.so.0
#20 0x802781220 <tevent_signal_get_tag+0x230> at /usr/local/lib/samba4/private/libtevent.so.0
#21 0x80277b5e1 <_tevent_loop_once+0xe1> at /usr/local/lib/samba4/private/libtevent.so.0
#22 0x80277b842 <tevent_common_loop_wait+0x32> at /usr/local/lib/samba4/private/libtevent.so.0
#23 0x8027812a0 <tevent_signal_get_tag+0x2b0> at /usr/local/lib/samba4/private/libtevent.so.0
#24 0x10308df <main+0x2a1f> at /usr/local/sbin/smbd
[2023/01/10 10:24:21.626767, 0] ../../source3/lib/dumpcore.c:315(dump_core)
dumping core in /var/db/system/cores
[2023/01/10 10:24:21.639459, 1] ../../source3/librpc/crypto/gse_krb5.c:418(fill_mem_keytab_from_system_keytab)
../../source3/librpc/crypto/gse_krb5.c:417: krb5_kt_start_seq_get failed (No such file or directory)
[2023/01/10 10:24:21.641107, 1] ../../source3/librpc/crypto/gse_krb5.c:418(fill_mem_keytab_from_system_keytab)
../../source3/librpc/crypto/gse_krb5.c:417: krb5_kt_start_seq_get failed (No such file or directory)
[2023/01/10 10:24:22.644998, 1] ../../source3/librpc/crypto/gse_krb5.c:418(fill_mem_keytab_from_system_keytab)

I'm also getting an alert about the smb.core file being created. If I delete it, it just comes back again.
Here is the output of testparm -s:

root@gecko[/var/log/samba4]# testparm -s
Load smb config files from /usr/local/etc/smb4.conf
Loaded services file OK.
Weak crypto is allowed

Server role: ROLE_DOMAIN_MEMBER

# Global parameters
[global]
aio max threads = 2
bind interfaces only = Yes
client ldap sasl wrapping = seal
deadtime = 3
disable spoolss = Yes
dns proxy = No
domain master = No
enable web service discovery = Yes
kerberos method = secrets and keytab
kernel change notify = No
load printers = No
local master = No
logging = file
max log size = 5120
min domain uid = 500
nsupdate command = /usr/local/bin/samba-nsupdate -g
obey pam restrictions = Yes
preferred master = No
realm = TIPPING.LAN
registry shares = Yes
restrict anonymous = 2
security = ADS
server multi channel support = No
server role = member server
server string = FreeNAS Server Gecko
template homedir = /mnt/share/homes/%D/%U
template shell = /bin/sh
unix extensions = No
winbind cache time = 7200
winbind enum groups = Yes
winbind enum users = Yes
winbind max domain connections = 10
winbind nss info = rfc2307
workgroup = STA
recycle:minsize = 20
recycle:maxsize = 50000000
recycle:exclude = *.tmp, .~lock*, *.autosave, *.pdf~
recycle:subdir_mode = 0770
recycle:directory_mode = 0770
idmap config *: range = 70001-100000
idmap config sta: schema_mode = rfc2307
idmap config sta: range = 500-70000
idmap config sta: backend = ad
rpc_server:mdssvc = disabled
rpc_daemon:mdssd = disabled
idmap config * : backend = tdb
create mask = 0770
directory mask = 0770
directory name cache size = 0
dos filemode = Yes
wide links = Yes


[assoc]
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/assoc
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr recycle
recycle:subdir_mode = 0700
recycle:directory_mode = 0777
recycle:touch = yes
recycle:versions = yes
recycle:keeptree = yes
recycle:repository = .recycle/%D/%U
nfs4:chown = true


[drawings]
comment = AutoCAD Drawing Storage
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/Drawings
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr recycle
recycle:exclude_dir = *_backup
recycle:exclude = *.tmp, *~lock*, *.dwl, *.dwl2, *.bak, *.pdf~, acad.dce, *Thumbs.db, *qlm.lck
recycle:subdir_mode = 0700
recycle:directory_mode = 0777
recycle:touch = yes
recycle:versions = yes
recycle:keeptree = yes
recycle:repository = .recycle/%D/%U
nfs4:chown = true


[admin]
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/admin
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr recycle
recycle:subdir_mode = 0700
recycle:directory_mode = 0777
recycle:touch = yes
recycle:versions = yes
recycle:keeptree = yes
recycle:repository = .recycle/%D/%U
nfs4:chown = true

[tipping]
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/tipping
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr recycle
recycle:subdir_mode = 0700
recycle:directory_mode = 0777
recycle:touch = yes
recycle:versions = yes
recycle:keeptree = yes
recycle:repository = .recycle/%D/%U
nfs4:chown = true

[images]
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/images
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr
nfs4:chown = true


[projDocs]
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/projDocs
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr
nfs4:chown = true

[Office]
comment = Office Shared Files
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/Office
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr recycle
recycle:exclude = *.tmp, *Thumbs.db, *~lock*, *.pdf~
recycle:subdir_mode = 0700
recycle:directory_mode = 0777
recycle:touch = yes
recycle:versions = yes
recycle:keeptree = yes
recycle:repository = .recycle/%D/%U
nfs4:chown = true


[engindata]
comment = Engineering Project files
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/engindata
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr recycle
recycle:exclude = *.tmp, *~lock*, *.autosave, *.pdf~, acad.dce, *.dwl, *.dwl2, *.bak, *.bak2, *Thumbs.db
recycle:subdir_mode = 0700
recycle:directory_mode = 0777
recycle:touch = yes
recycle:versions = yes
recycle:keeptree = yes
recycle:repository = .recycle/%D/%U
nfs4:chown = true

[TmpShare]
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/TmpShare
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr
nfs4:chown = true

[homes]
admin users = "@STA\domain admins"
ea support = No
hide dot files = No
level2 oplocks = No
map archive = No
oplocks = No
path = /mnt/share/homes/%D/%U
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr
nfs4:chown = true


[profiles]
admin users = "@STA\domain admins"
ea support = No
hide dot files = No
map archive = No
path = /mnt/share/profiles
read only = No
smbd max xattr size = 2097152
store dos attributes = No
vfs objects = zfs_space zfsacl streams_xattr
nfs4:chown = true

 

[anodos]

There's a recycle bin bug in 13.0-U3.1 (an SMB_ASSERT() can be triggered when user lacks permissions to recycle). Looks like you're possibly hitting that. You can try disabling the recycle bin. It also looks like a large number of auxiliary parameters are in place. If you continue to have issues, please remove / comment them out from your config and see if the issues go away.

 

[bazzacad]

Thanks for the reply. The recycle bin wasn't enabled on this share, so I removed the auxiliary parameters but it didn't help.
Do I need to restart Samba to apply the changes?

 

[anodos]

Thanks for the reply. The recycle bin wasn't enabled on this share, so I removed the auxiliary parameters but it didn't help.
Do I need to restart Samba to apply the changes?

View attachment 62283

If client already has an SMB session, then you will need to restart SMB service.

 

[anodos]

Ah. I haven't seen that particular SMB assertion for recycle in yet. If you still have the corefile, can you zip it up and send it to me via PM?

 

[bazzacad]

OK, I restarted SMB and I'm still unable to delete the file. I'll see if I can send you the core file now.

 

[anodos]

update: share's path was a symbolic link, which was triggering the SMB_ASSERT() in the recycle module. Due to issues with atomicity of recycle operations (rename) in 13 / SCALE we generate per-dataset recyclebins. (with the .recycle being at root of dataset in share). There is a sanity check via fstatfs() that the dataset mountpoint is actually within the share's path. This means odd custom configurations with symlinks (this sort of thing can only be created via shell) will not be compatible with the recycle bin.

 

[thomisus]

@anodos Hi, I'm not shure if it's the same bug , but since upgrading to Truenas CORE 13 i'm getting lots of samba core dumps. I thought the problem was related to some auxiliary parameters, but after switching to default config, i'm still getting core dumps. May I send you my logs in order to understand what's happening?
Best regards