Creating cluster issues

[tenknas]

PLAN
Im trying to create a storage cluster using the release version of scale and truecommand. Im setting this all up with Hetzner. Currently I have 3 storage servers that I need to migrate to scale. So the plan was to
1. buy 1 x new storage server + 2 x temporary small servers to just achieve the minimum of 3 servers for the initial cluster (as stated clustering needs 3 servers)
2. move files from older servers and add it to the cluster after
3. remove the 2 temporary servers (I know truecommand doesnt allow removing servers yet, but probably soon?)

SETUP
Specs of the server are
New storage server (old servers have the same specs as this)
Intel W-2145, 128gb ram, 2 x NVMe, 10 x 16TB HDD, 1 x 10gbps nic

Temporary Servers
Intel i7-4770. 32gb ram, 2 x 240gb SSD
As said, plan was to remove them after all servers has been joined to the cluster.

Network info is as follows
Main interface is using ipv4 address included in server.
vlan is setup using hetzner's vswitch setup and hooked up to main interface. Each server has 1 x private network IP with something like 10.0.1.XXX/24

Truecommand Node is on a hetzner cloud server with 2 x AMD vCPU, 2gb ram, 40gb nvme and connected to the vswitch private network same as the dedicated servers. Using truecommand v.2.1

ISSUES:
I followed the guide posted here

TrueNAS SCALE - Clustered SMB HOWTO

TrueNAS SCALE Testers, As many of you know, one of the major new features we’ve been hard at work on with SCALE is the ability to create scale-out clustered SMB shares. Much of this functionality has already landed in 22.02, and the upcoming TrueCommand 2.2 version will provide a more...

www.truenas.com

I got to the part that I was able to add all the 3 servers and started to create the cluster. I used the below configuration for the cluster,

After clicking create was it said "Cluster volume xxxxxxx created successfully." but after that it was just blank on the Cluster Volume page of truecommand. As if nothing happened.

Hope someone can point me to the right direction on getting this to work.

 

[Kris Moore]

Try this on one of the nodes that was part of the cluster:

# gluster volume list

Do you see ctdb_shared_vol and your 'pool-temp1' volume listed?

If so, then the creation worked, but there is a bug in TC where it is not displaying the cluster properly. I'm seeing the same issue here, and have already file tickets and the team is investigating.

 

[tenknas]

Try this on one of the nodes that was part of the cluster:

# gluster volume list

Do you see ctdb_shared_vol and your 'pool-temp1' volume listed?

If so, then the creation worked, but there is a bug in TC where it is not displaying the cluster properly. I'm seeing the same issue here, and have already file tickets and the team is investigating.

Yeah I see them. Do you think its possible to just manually do the glusterfs and mounting from here and just "relink" cluster to truecommand in the future when its stable?

 

[Kris Moore]

You can go through the rest of the walkthrough at this point to setup SMB shares, TC did its part. We're working on a TC update now which will fix the issue with it not showing up in the UI, you won't need to "relink" or do any changes on the TrueNAS side when that fix lands.

 

[tenknas]

You can go through the rest of the walkthrough at this point to setup SMB shares, TC did its part. We're working on a TC update now which will fix the issue with it not showing up in the UI, you won't need to "relink" or do any changes on the TrueNAS side when that fix lands.

Thanks I got it working using your guide and the truecommand documentation here https://www.truenas.com/docs/truecommand/clustering/.

A few questions tho about the setup.
1. I was able to connect clients using the private network which I setup, so anything within 10.0.0.0/16 can connect. But anything outside of this is unable to connect. I need the smb share to be able to be accessible from the public IP of the servers. Is that possible?
2. Security wise, connecting the client to the glusterfs using glusterfs-client didnt seem to ask for any sort of login. Was it because it was using the private network? (Sorry this is my first time to setup a gluster cluster system, so pardon the newbie question).
3. I would like to have 2 types of shares, 1 share should be read and write capable and the other should be read-only, how should I do that?

Finally, is there documentation about the midclt commands? I might be to solve my questions if there was any sort of documentation on it. So far all the commands I know were just based on your post. I couldnt find anything about this in the truenas documentation.

 

[Kris Moore]

1. Yes, if you look at my walkthrough where you setup the SMB VIPS, you should be using public IPs on the same interface / subnet as your clients.

# midclt call ctdb.public.ips.create '{"pnn":0, "ip": "<virtual ip for this host>", "netmask": 24, "interface": "<network interface>"}'

2. Yes, the private network for gluster traffic needs to be that way for that reason exactly, as well as the fact that it passes credentials around in the clear, so you want that away from your users.

3. When you create the various SMB shares, you can set either options to make them RO at the share-level, or do your usual 'chmod <perms> /cluster/<volume>/directory' to make something R/O that way.

No real documentation on the midclt commands, however if you run the 'cli' command, it acts as a front-end to the API/midclt command, and might be more user-friendly to explore the various API options you can set.

 

[tenknas]

1. Yes, if you look at my walkthrough where you setup the SMB VIPS, you should be using public IPs on the same interface / subnet as your clients.



2. Yes, the private network for gluster traffic needs to be that way for that reason exactly, as well as the fact that it passes credentials around in the clear, so you want that away from your users.

3. When you create the various SMB shares, you can set either options to make them RO at the share-level, or do your usual 'chmod <perms> /cluster/<volume>/directory' to make something R/O that way.

No real documentation on the midclt commands, however if you run the 'cli' command, it acts as a front-end to the API/midclt command, and might be more user-friendly to explore the various API options you can set.

About the commands, Im assuming midclt was using this https://www.truenas.com/docs/api/websocket.html and what youre referring to as API is this https://www.truenas.com/docs/api/rest.html.

I think I should be able to work my way around knowing these. Thank you for leading me there and for your time and answering my queries.

Would still want to ask a few questions since I cant wrap my head around these yet,

1. So this was the command I used on each server that was part of the cluster

Code:

midclt call ctdb.public.ips.create '{"pnn":0, "ip": "123.123.123.123", "netmask": 26, "interface": "enp23s0"}

Changing the PNN, IP to the assigned public IP from hetzner and changing the proper interface name the ip is attached to.

However, Ive tried connecting a client that is outside of the 10.0.0.0/16 (private vlan), ie connecting through public network, and still cant connect. Im not sure if Im missing something here.

2. Fair enough, I did create a vlan specifically for the clustering and dont plan to add more servers outside of that scope.

3. Im assuming this was the command to create the share

Code:

midclt call sharing.smb.create '{"path": "/clustersmb", "name": "MYSHARE", "purpose": "NO_PRESET", "shadowcopy": false, "cluster_volname": "myvolume"}'

I created a few more cluster volumes without issueing another sharing.smb.create command and those new ones was instantly accessible through mounting it on clients. Was that supposed to happen?

EDIT:

Ive also tried creating a new gluster volume through TC using the public IPs attached to each node, I got an error with "[EFAULT] volume create: pulbic5: failed: Host 123.123.123.123 is not in 'Peer in Cluster' state". But works no problem when using the private vlan ips within 10.0.1.XXX.

Also getting this from glusterfs log on client when mounting it using the command mount -t glusterfs publicipofnode1:/cluster testmount

Code:

[2022-03-03 10:33:18.036350] I [fuse-bridge.c:5162:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31
[2022-03-03 10:33:18.036405] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2022-03-03 10:33:18.036789] E [fuse-bridge.c:5234:fuse_first_lookup] 0-fuse: first lookup on root failed (Transport endpoint is not connected)
[2022-03-03 10:33:18.037019] W [fuse-bridge.c:1271:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)
[2022-03-03 10:33:18.037169] W [fuse-bridge.c:1271:fuse_attr_cbk] 0-glusterfs-fuse: 3: LOOKUP() / => -1 (Transport endpoint is not connected)
[2022-03-03 10:33:18.040504] W [fuse-bridge.c:1271:fuse_attr_cbk] 0-glusterfs-fuse: 4: LOOKUP() / => -1 (Transport endpoint is not connected)
[2022-03-03 10:33:18.041436] I [fuse-bridge.c:6082:fuse_thread_proc] 0-fuse: initiating unmount of /root/testmount
[2022-03-03 10:33:18.041902] W [glusterfsd.c:1595:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f26aa433609] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xcd) [0x55d9af92886d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x58) [0x55d9af9286e8] ) 0-: received signum (15), shutting down
[2022-03-03 10:33:18.041945] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/root/testmount'.
[2022-03-03 10:33:18.041961] I [fuse-bridge.c:6875:fini] 0-fuse: Closing fuse connection to '/root/testmount'.

 

[Kris Moore]

Changing the PNN, IP to the assigned public IP from hetzner and changing the proper interface name the ip is attached to.

However, Ive tried connecting a client that is outside of the 10.0.0.0/16 (private vlan), ie connecting through public network, and still cant connect. Im not sure if Im missing something here.

Check 'ip a' and look for the public IP's assigned to your local interfaces. Remember, these should be "new" IPs which will float between cluster nodes as they go offline. If they are indeed on the interface, perhaps try running:

# systemctl restart smbd

See if that brings the Samba service back online and servicing them.

Also would be helpful to see output of "systemctl status ctdb" or "systemctl status smbd"

I created a few more cluster volumes without issueing another sharing.smb.create command and those new ones was instantly accessible through mounting it on clients. Was that supposed to happen?

You need to specify, you mean gluster clients or SMB clients? After making a volume, the gluster volume will always be immediately available to clines on the private gluster network / IPs.

Ive also tried creating a new gluster volume through TC using the public IPs attached to each node, I got an error with "[EFAULT] volume create: pulbic5: failed: Host 123.123.123.123 is not in 'Peer in Cluster' state". But works no problem when using the private vlan ips within 10.0.1.XXX.

Also getting this from glusterfs log on client when mounting it using the command mount -t glusterfs publicipofnode1:/cluster testmount

Gluster nodes cannot be re-configured to use new IPs or joined to "multiple" cluster topologies at once this way. You can make new volumes only if you pick the same private network configuration used before.


A bit of background would probably be helpful for you here.

You shouldn't be planning to mix-n-match gluster and SMB clients.

If you are wanting to use SMB mode, you have gluster on a private subnet, and the only systems that can connect to it will be other node members on that private subnet. This is for security, since some secrets and credentials will be passed between nodes "in the clear". In this use-case you'd only use SMB clients to connect to your cluster (from the public VIPs)

If you want to run in "gluster native mode" only, without SMB. Then you can setup a volume / cluster using the TC UI, and picking the public IPs of your TrueNAS boxes. This will create gluster volumes, which clients can mount at will on the public side.

The next update to TC will have most of this logic wrapped into the wizards to try and guide users through picking which mode they intend to use for their cluster.