Anybody get up and running with a Gluster cluster? Experience so far?

[GoingOffRoading]

I've been experimenting with Gluster on some small scale hardware, and will be building TrueNAS machines to move my homelab data set. It has not been a flawless experience, but I'm ready to move forward with new storage nodes, TrueNAS Scale, TrueCommand, and more.

Specifically:

• 1x Node: HP Microserver Gen 10 Plus nodes, 64Gb memory, 4x 18Tb Seagate EXOS setup in RaidZ2
• Then setup this node as a Distributed in TrueCommand, and mount my containers and shares using the Gluster client.
• When more storage is needed, build more of the above nodes, and add them to the cluster.
• Compute is done on Intel NUCs running Kubernetes, so TrueCommand will be hosted there, with persistence on a Replicated Gluster cluster on the NUCs.

I know there are some challenges, beta, etc... But color me very excited anyway.

-------------------------

I have not seen a lot of threads here or elsewhere talking about getting TrueNAS Scale clusters up and running. Anybody care to share what they have done so far?

 

[aervin]

Thanks @GoingOffRoading - there are some things I'd like to add:

• You'll obviously want all SCALE nodes on Angelfish, and TrueCommand 2.1 is the most stable build for clustering.
• TrueCommand 2.1 is still in the experimental phase for the clustering feature.
• Regardless of gluster volume type, adding bricks (nodes) to volumes is not yet exposed. For that matter, removing bricks is not implemented in the TrueCommand interface.

 

[GoingOffRoading]

Thanks @GoingOffRoading - there are some things I'd like to add:

• You'll obviously want all SCALE nodes on Angelfish, and TrueCommand 2.1 is the most stable build for clustering.
• TrueCommand 2.1 is still in the experimental phase for the clustering feature.
• Regardless of gluster volume type, adding bricks (nodes) to volumes is not yet exposed. For that matter, removing bricks is not implemented in the TrueCommand interface.

Thanks!

Thankfully I've made some hardware design changes since writing the original where I will not need to add/remove bricks need any of the related functionality for some period.

I'm on the fence of setting up my new TrueNAS Scale NAS as:

• A stand alone NAS and use NFS to mount the volumes in my baremetal kubernetes cluster.
• As a Gluster node, and mount volumes using the Gluster FUSE API

Setting up to be scalable from day 1 would be dank, and not having the ability to scale Gluster nodes would be moot in my business case for a year or two.

 

[aervin]

@GoingOffRoading , yeah brick limitations are just that for now. Clustered SMB is next on the roadmap, more volume management functionality after that.