Reset SMB rather than fix it

[sophware]

After yesterday's update, SMB won't start. I think it was having issues ever since the upgrade from Core, but now things show up in Task Manager:

sharing.smb.sync_registry​

Error: [ENXIO] Registry calls not permitted when ctdb unhealthy.

AND

smb.configure15.00%

Waiting for ctdb to become healthy.

Instead of troubleshooting this, can I just clear out any SMB shares or configuration and start clean?

 

[morganL]

Instead of troubleshooting this, can I just clear out any SMB shares or configuration and start clean?

Given its Christmas, probably best to do this and avoid a long wait.

 

[sophware]

Thank you. Makes sense. I wasn't able to start the SMB service to delete anything from it.

The only thing I can think of is to go back to Core or, maybe better, wait.

 

[morganL]

Can you rollback to RC-1.x Make sure it's all healthy then try the update again if you need to?

 

[sophware]

Thanks for the good advice. Going back to 1.x (don't remember which) did enable me to start SMB and delete shares. Now, maybe-related (maybe the real issue, via unhealthy CTDB), I have a partially-deleted Gluster FS cluster volume. It won't delete from TrueCommand. Do you happen to know if I can manually get rid of all traces of it?

 

[anodos]

Make sure glusterd is disabled:

Code:

root@truenas[/tmp]# midclt call service.query '[["service", "=", "glusterd"]]'                         
[{"id": 26, "service": "glusterd", "enable": false, "state": "STOPPED", "pids": []}]
root@truenas[/tmp]# midclt call service.update 26 '{"enable": false}'                                  
26

Then reboot server to make sure state is complete clean via reboot.

 

[sophware]

OK. Is the idea to disable the service so SMB can start? It would be cool to just have a healthy CTDB and be able to create a new Gluster FS cluster volume.

 

[anodos]

OK. Is the idea to disable the service so SMB can start? It would be cool to just have a healthy CTDB and be able to create a new Gluster FS cluster volume.

It depends on what the goal is. If you want to use glusterfs, then clustered SMB is the only available option. No mixing / matching gluster / non-gluster shares. Clustering features will have to be configured through TrueCommand. Once cluster is configured properly through truecommand, the ctdb state will be healthy and allow creation of clustered SMB shares.

If you aren't planning to use glusterfs, then you can take the step I mentioned to disable it and configure non-clustered SMB shares.

 

[sophware]

But if I do want to use it, then I have to disable it? I do want to use it. It was working for a while.

 

[anodos]

But if I do want to use it, then I have to disable it? I do want to use it. It was working for a while.

Maybe try to figure out why ctdb is unhealthy. /var/log/middlewared.log should give some indication of this. `ctdb status` on each node should also help. There's a ctdb shared volume that should also be mounted on each cluster node, so check your mount output and verify that it's mounted correctly.

 

[morganL]

It should be noted that SMB on gluster volumes is still early in it dev cycle. The TrueCommand UI is not complete and there is active testing going on in iX. I don't recommend using this unless its very much a test environment with no other production services.

 

[sophware]

It should be noted that SMB on gluster volumes is still early in it dev cycle. The TrueCommand UI is not complete and there is active testing going on in iX. I don't recommend using this unless its very much a test environment with no other production services.

Thanks for the reminder. I had thought I was not involving SMB. All I did was create a dispersed volume.

 

[sophware]

Maybe try to figure out why ctdb is unhealthy. /var/log/middlewared.log should give some indication of this. `ctdb status` on each node should also help. There's a ctdb shared volume that should also be mounted on each cluster node, so check your mount output and verify that it's mounted correctly.

Code:

root@freenasa[/etc/ctdb]# service ctdb status
● ctdb.service - CTDB
     Loaded: loaded (/lib/systemd/system/ctdb.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Sun 2021-12-26 12:15:36 EST; 1 day 3h ago
       Docs: man:ctdbd(1)
             man:ctdb(7)

Dec 26 12:15:36 freenasa.binti.chezmalloy.com systemd[1]: Condition check resulted in CTDB being skipped.
root@freenasa[/etc/ctdb]#

root@freenasa[/etc/ctdb]# mount | grep -i ctdb
TANK-A/.system/ctdb_shared_vol on /var/db/system/ctdb_shared_vol type zfs (rw,relatime,xattr,nfs4acl)
root@freenasa[/etc/ctdb]# ctdb status
connect() failed, errno=2
Failed to connect to CTDB daemon (/var/run/ctdb/ctdbd.socket)
Failed to read nodes file "/etc/ctdb/nodes"
Is this node part of CTDB cluster?
root@freenasa[/etc/ctdb]#

the middleware log is full of errors, like those shown below. is that less important, knowing that the daemon isn't even running?

Code:

[2021/12/27 13:01:56] (ERROR) GlusterFuseService.mount():122 - Failed to mount /cluster/ctdb_shared_vol with error: Mount failed. Check the log file  for more details.
[2021/12/27 13:06:55] (ERROR) GlusterFuseService.mount():122 - Failed to mount /cluster/ctdb_shared_vol with error: Mount failed. Check the log file  for more details.
[2021/12/27 13:11:56] (ERROR) GlusterFuseService.mount():122 - Failed to mount /cluster/ctdb_shared_vol with error: Mount failed. Check the log file  for more details.
[2021/12/27 13:15:47] (ERROR) middlewared.job.run():421 - Job <bound method ClusterJob.process_queue of <middlewared.plugins.cluster_linux.ctdb_job.ClusterJob object at 0x7fc812f0dd60>> failed
  File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_job.py", line 64, in process_queue
    node = (await self.middleware.call('ctdb.general.status', {'all_nodes': False}))[0]
  File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 64, in status
    result = await self.middleware.call('ctdb.general.wrapper', command)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/cluster_linux/ctdb_general.py", line 35, in wrapper
middlewared.service_exception.CallError: [EFAULT] ctdb command failed with error connect() failed, errno=2
Failed to connect to CTDB daemon (/var/run/ctdb/ctdbd.socket)
Failed to read nodes file "/etc/ctdb/nodes"
Is this node part of CTDB cluster?
[2021/12/27 13:16:55] (ERROR) GlusterFuseService.mount():122 - Failed to mount /cluster/ctdb_shared_vol with error: Mount failed. Check the log file  for more details.
[2021/12/27 13:21:55] (ERROR) GlusterFuseService.mount():122 - Failed to mount /cluster/ctdb_shared_vol with error: Mount failed. Check the log file  for more details.
[2021/12/27 13:26:56] (ERROR) GlusterFuseService.mount():122 - Failed to mount /cluster/ctdb_shared_vol with error: Mount failed. Check the log file  for more details.

 

[anodos]

Failure to mount the ctdb_shared_vol is probably your root cause of the issue (ctdb is used as clustered backend for our directory services and SMB configuration when we are clustered and so it is a requirement, and should be set up properly by TrueCommand). Did you originally set this up through TrueCommand?

 

[sophware]

I did, yes.

 

[morganL]

Thanks for the reminder. I had thought I was not involving SMB. All I did was create a dispersed volume.

If you set up cluster volumes 1st and then SMB second, the system may have assumed it was operating in clustered SMB and gluster mode. This part of the system is still green and needs more UI work in TrueCommand.

 

[sophware]

Fair enough. I don't recall, but guess I might have tried to fire up an ISO share.

Any way to undo what happened and start over, other than reinstalling three servers?

 

[morganL]

Fair enough. I don't recall, but guess I might have tried to fire up an ISO share.

Any way to undo what happened and start over, other than reinstalling three servers?

Those types of recovery tools are the ones still being developed/tested..... so, not sure in your specific case.

 

[sophware]

Thanks for the replies!