SOLVED All services reset during SMB transfer

[tyler.montney]

Edit

I somehow mistook a PCI slot for PCI-E. That was the root cause and there are no issues with TrueNAS.

Original Post

After troubleshooting, all services go down during the transfer (can't even ping). I've found a few results of people complaining about this, but it's usually a system crash. For me, the services come back up. SMB log simply states that it receives an exit signal. It shuts down then starts again. Messages doesn't contain anything useful. I can copy a 1 GB file directly to the share and it completes. I'll try again and it hangs. I had no issue replicating close to 100 GB, over SSH.

• TrueNAS-12.0-U2.1
• HP Z420 Workstation (8 GB)
• Pool in a raidz2 (10 TB)
• Veeam B&R 11.0.0.837

 

[Redcoat]

• TrueNAS-12.0-U2.1
• HP Z420 Workstation
• Pool in a raidz2
• Veeam B&R 11.0.0.837

Backing up to a simple SMB share. Runs fine for a minute or two then I get this message (Windows and Linux agents).

I have to ask the most basic question - you're not, by chance, using the TrueNAS root account credentials for the SMB share access are you?

I've got 3 w10 boxes backing up everyday and I have noticed that error messages sometimes aren't "seemingly helpfully accurate", also that VA works "internal processing" at the start of a backup for "some time" before failing and providing a message on its progress panel that the SMB target is not available. I cannot tell you how long, but it may be "a minute or two".

 

[tyler.montney]

Nope, created an account for Veeam on the TrueNAS box. I know I had to use the root account to setup replication between my TrueNAS boxes (non-root would keep saying that the password was incorrect; could SSH from the terminal between the boxes).

 

[tyler.montney]

I'll add that Veeam itself can no longer scan the share. It may have something to do with me using multiple NICs?

The SMB service was running, and when I changed the hostname it failed to start.

Code:

smbd -i -d 3

output

Code:

smbd version 4.12.9 started.

Copyright Andrew Tridgell and the Samba Team 1992-2020

uid=0 gid=0 euid=0 egid=0

lp_load_ex: refreshing parameters

Initialising global parameters

Processing section "[global]"

directory_create_or_exist: mkdir failed on directory /var/db/system/samba4/private/msg.sock: No such file or directory

/var/db/system is indeed empty...

 

[Redcoat]

using multiple NICs?

On the same subnet?

 

[tyler.montney]

On the same subnet?

Nope, separate. I've tried to isolate this box as best I can. Only a few hosts have access to this subnet, including Veeam. Originally, Veeam was able to find and use the share (so it did work at some point). Only other thing I changed was the syslog setting for System Dataset (I couldn't find the logs). It made the SMB service restart.

 

[Redcoat]

Does a reboot bring you up with Samba running?

 

[tyler.montney]

Does a reboot bring you up with Samba running?

Hah, I tried that just now per https://www.truenas.com/community/t...r-like-var-db-samba4-20190519-the-date.76566/. No luck.

 

[tyler.montney]

I think I found the problem:

Code:

root@HOSTNAME[/var/db/system]# mkdir test

mkdir: test: Read-only file system

 

[Redcoat]

I just quickly read that.

Do you have a saved config from when Samba was running that you can reverie back to, or maybe you can roll back with your snapshots, for test?

 

[tyler.montney]

I just quickly read that.

Do you have a saved config from when Samba was running that you can reverie back to, or maybe you can roll back with your snapshots, for test?

Nope. Set this system up in the last couple days. Hadn't gotten to backing up the config. There isn't much to reconfigure, I've read that I can blow away the config but keep the datasets?

 

[Redcoat]

mkdir: test: Read-only file system

That's on the Veeam target?

 

[tyler.montney]

That's on the Veeam target?

No, the TrueNAS box where the samba4 folder should be.

 

[Redcoat]

Hadn't gotten to backing up the config.

That and snapshots are good tools when testing and trying.

 

[tyler.montney]

That and snapshots are good tools when testing and trying.

True.

Permissions look fine, root has write and is owner.

Code:

ls -lo

lists "uarch", so it doesn't appear to be immutable. Can make a folder in /var/db so the entire FS is not read-only. Can't move the system folder, get "operation not supported". Removing it gives me "device busy".

 

[tyler.montney]

And

Code:

umount /var/db/system

did the trick. As soon as I did that, the samba4 directory appeared. SMB service started successfully.

Successfully reproduced issue:

1. Uncheck syslog under System Dataset
2. Confirm SMB needs to restart
3. Go to Services and stop SMB, then start it
4. Error will occur
5. Unmount /var/db/system
6. SMB will successfully start

 

[Redcoat]

Good sleuthing, Suggest you make a bug report - link on masthead.

 

[Redcoat]

What happens if this box is checked?

 

[tyler.montney]

Doesn't break anything and I see logs under /var/log/samba4. Unfortunately, none of them seem to log connections. Perhaps increasing "Log Level" will help?

Issue still remains for Veeam stopping mid-backup. I did notice TrueNAS get wonky when Veeam was backing up. It suddenly signed me out. When I logged in and went to shell, the shell became unresponsive and a button "reconnect" appeared. As soon as the backup failed, TrueNAS became responsive again.

Saw a suggestion from another thread to reduce concurrent connections to 1 and set a rate limit of 20 MB/s, in Veeam. No effect.

 

[tyler.montney]

Decided to test by copying files directly to the share (creating files with fsutil).

• 1 MB: OK
• 10 MB: OK
• 100 MB: OK
• 1 GB: OK
• 5 GB: Copies but all further connections fail until it resets.

All services go down, including SSH and HTTP. This seems like it's a bigger problem than SMB? Network card is an Intel 82579LM.