sbmd Core files for executables have been found

[anubis_truenas]

Hi, being a relative noob, I'm struggling to figure out what to do with the warning I started getting this morning:
Core files for executables have been found in /var/db/system/cores.

It seems to be related to the Samba share. As I can see the TRUENAS server in windows explorer, but no longer am able to access the files on it. The corresponding core files also all seem to relate to the smdb service.

After receiving this error I ran the latest upgrade to TrueNAS-SCALE-23.10.1.3. But that didn't have any effect.
How now best to troubleshoot/fix this?

 

[anodos]

I fixed a bug for 23.10.2 where smbd would assert / crash on accessing shadow copies in a ZFS snapdir where the user lacked read access to its mountpoint. O_PATH opens are insufficient to maintain auto-mounted ZFS snapshots in the ZFS ctldir, which means we need to maintain a refcounted non-O_PATH open as samba accesses data there (otherwise fs operatios will inexplicably fail with EBADF IIRC). Refcounting is necessary so that we need know to pull out the door-wedge to allow ZFS to dynamically unmount the snapshot.

There's a separate upstream bug that will be fixed in 23.10.2 where certain operations on alternate data streams while doing a shutdown close of a file could cause a crash.

 

[anodos]

Another problem that we've seen is that users are getting in situations where `/var/run` is 100% filled. We were in some situations git cloning charts to a subdirectory there in some situations where users didn't have an apps pool created. This worked okay for our own charts, but TrueCharts can consume over 1 GiB IIRC. When smbd is unable to initialize a messaging context (in this case due to having no free space available) it will assert.

 

[anubis_truenas]

@anados, the '/var/run' was indeed 100% filled.
I have been playing with both Truenas and TrueCharts apps, but found them too buggy and switched to running the functionality from VM's instead. I therefore deleted all apps and removed the apps pool (as I wasn't using it anyway). Recreating the apps pool has moved the catalogs. The '/var/run/' is no longer at 100% and the samba shares are accessible again.

 

[anubis_truenas]

BTW. Previously I had (for whatever reason) not been able to remove the TrueCharts catalog. I therefore had removed the apps pool all together. Didn't realize that in the background the catalog sync would try to continue regardless. Now, after recreating the apps pool, I was able to remove the TrueCharts catalog.

 

[NetRanger]

Hello,
I'm experiencing precisely the same issue with TrueNAS Scale 23.10.1.3.
This problem arose suddenly while transferring files from a Windows 2022 Server to an SMB share on my TrueNAS Scale server.
SMB permissions are correct and no quotas are defined.
What's particularly puzzling is that I had successfully copied ±26 TB of data without any problems, and the issue only manifested once my pool reached around 65% capacity.
Clicking on "Try again" restarts the file copy operation, but the process halts once more after a short while.
Rebooting the TrueNAS server did not resolve the issue, as the problem persists.
Any troubleshooting hints for addressing this very annoying behaviour would be mostly appreciated.

With my kindest personal regards.

 

[chuck32]

The latter of the message reads "Please create a ticket at https://jira.ixsystems.com/ and attach the relevant core files along with a system debug. Once the core files have been archived and attached to the ticket, they may be removed by running the following command in shell: 'rm /var/db/system/cores/*'."

I suggest you do that. From the information given I guess it's hard to give you any pointers. When you created the ticket you should add your exact hardware information if you want to receive any help.

I assume you checked for the suggestion mentioned in this thread?

Another problem that we've seen is that users are getting in situations where `/var/run` is 100% filled.

Just a personal note: imo a 12 wide vdev is pretty wide for raidz1. OTOH I also assume the scale server is part of your backup routine so you may be correct in taking the risk. Just wanted to point out that one could argue for raidz2 or even raidz3 in such wide setups.