Guide to setting up gluster cluster for truenas scale?

Status
Not open for further replies.

tenknas

Dabbler
Joined
Mar 6, 2021
Messages
21
Hello truenas community,

[Intro, you can skip this part]
I'm quite a newbie at mostly everything, I just really like doing storage stuff. So I got 3 servers with a few hard drives on them, I just recently discovered truenas/freenas and now truenas scale. I read about it having gluster which is a software that I wanted to play around with for so long but been overwhelmed by the documentation starting with zfs, so I didnt know where to start. I'm hoping this community could help me out in setting something like this up.

[Requirements]
I would like to know if anyone has a guide on setting up zfs + gluster and clustering 3 servers with multiple hard drives with L2ARC. Would like to know how I can make a gluster cluster with easy scaling in case I wanted to add more servers to the cluster. Would be cool if its really step by step. I wouldnt mind if you point me to a guide thats already available.

[What I currently have setup]
I have already setup the servers with truenas scale and the zfs pool with L2ARC using the really n00b friendly GUI of truenas, however I dont have any clue on how to use gluster since that one doesnt have a GUI.


Thanks!
 

stavros-k

Patron
Joined
Dec 26, 2020
Messages
231
First, let me remind you that SCALE not yet production ready, especially clusters. So in case you use your servers for important files. I wouldn't recommend doing clusters. Current state: (https://www.truenas.com/community/threads/truenas-scale-21-02-is-delivered.91127/).
Clusters tho are available from CLI, but not UI yet. Also i don't think there is any documentation available for now. Even if someone posts a step-by-step guide. At the release stage it might be very out of date!

But if someone want to post, i'd like to see how it goes! (I don't have any nodes at home for testing!)
 

tenknas

Dabbler
Joined
Mar 6, 2021
Messages
21
First, let me remind you that SCALE not yet production ready, especially clusters. So in case you use your servers for important files. I wouldn't recommend doing clusters. Current state: (https://www.truenas.com/community/threads/truenas-scale-21-02-is-delivered.91127/).
Clusters tho are available from CLI, but not UI yet. Also i don't think there is any documentation available for now. Even if someone posts a step-by-step guide. At the release stage it might be very out of date!

But if someone want to post, i'd like to see how it goes! (I don't have any nodes at home for testing!)

Yeah Ive read it isnt for production usage, but I think it zfs and gluster isnt something truenas has developed and both softwares are mature enough for production. And yeah I know the risk of the bugs possible with the current setup, however I think most of the things I would be using are probably ok to use in a non-mission critical usecase. I dont need other functionality other than the storage back end and clustering, all the other features of truenas scale are just cherries on top for what I want to use it for.

In general, I just want to make one big NAS by clustering them using gluster. I just dunno if the existing guides would be compatible with truenas scale, but I have a feeling it would since truenas scale uses debian. Would still like to have guidance on the matter.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
but I think it zfs and gluster isnt something truenas has developed and both softwares are mature enough for production.
Maybe if you're new to a product, don't start assuming all sorts of things?
TrueNAS uses A LOT of middleware and even TrueCommand to manage Gluster. Everything you do will with some certainty be not safe for your data at all and will possibly be depricated/broken by the time RELEASE hits.

And yeah I know the risk of the bugs possible with the current setup, however I think most of the things I would be using are probably ok to use in a non-mission critical usecase. I dont need other functionality other than the storage back end and clustering,
I think you don't fully grasp it:
Let me be clear: Clustering is not at all finished yet. It will break come next update. It might even break in before next-update hits. Beyond testing it's not use-ready for any usecase.

all the other features of truenas scale are just cherries on top for what I want to use it for.
Those "cherries" are the only parts of SCALE that are actually somewhat finished.

In general, I just want to make one big NAS by clustering them using gluster. I just dunno if the existing guides would be compatible with truenas scale, but I have a feeling it would since truenas scale uses debian.
No.
SCALE is an appliance, it doesn't support manually setting up gluster, big chance it won't even survive a reboot. SCALE handles configuration of Gluster using middleware.

Would still like to have guidance on the matter.
I estimate there are about 4 people on this forum that know how to set it up currently using CLI.
(Although it might also be available on TrueCommand 2.0, someone needs to get back to me on that one)

However, as you just explained you didn't even get the basis of what TrueNAS SCALE is and is not or the basics of how configuration on TrueNAS works. So I doubt you are going to get what you ask for, not because you're a bad person or something (I highly doubt it), but simply because it would take ages to explain the basics to you AND give you guidance on what you ask.
 

tenknas

Dabbler
Joined
Mar 6, 2021
Messages
21
Maybe if you're new to a product, don't start assuming all sorts of things?
TrueNAS uses A LOT of middleware and even TrueCommand to manage Gluster. Everything you do will with some certainty be not safe for your data at all and will possibly be depricated/broken by the time RELEASE hits.


I think you don't fully grasp it:
Let me be clear: Clustering is not at all finished yet. It will break come next update. It might even break in before next-update hits. Beyond testing it's not use-ready for any usecase.


Those "cherries" are the only parts of SCALE that are actually somewhat finished.


No.
SCALE is an appliance, it doesn't support manually setting up gluster, big chance it won't even survive a reboot. SCALE handles configuration of Gluster using middleware.


I estimate there are about 4 people on this forum that know how to set it up currently using CLI.
(Although it might also be available on TrueCommand 2.0, someone needs to get back to me on that one)

However, as you just explained you didn't even get the basis of what TrueNAS SCALE is and is not or the basics of how configuration on TrueNAS works. So I doubt you are going to get what you ask for, not because you're a bad person or something (I highly doubt it), but simply because it would take ages to explain the basics to you AND give you guidance on what you ask.

First off thank you for the time you spent on writing your reply and dissecting my comment. I mean it in a good way.

Yes, Ive only discovered truenas after I watched an LTT video on how to make your own Jellyfish (another storage solution which is rather expensive). They were using truenas core tho.

Indeed have not grasped what truenas is yet, but like you said the cherries are somewhat finish which is something I am interested in using as part of my setup. And again this isnt something mission critical, so anything breaking is ok. Im here to learn what it is and what it can be and what it can provide. I am no sysadmin, Im just a hobbyist. Any of what Im doing will not even be used by anyone but me. Its just my first dip into the hobbist level of data hoarding. Possibly if I learn more, I can deploy some sort of solution for friends and family like a data storage for photos and videos in the far future when truenas scale becomes production ready.

For now, why Im asking is I wanted to make a cluster work even if its not yet offered by truenas, because there isnt really a solution yet. Its more of I want to know if I can pull off a gluster cluster thru some guidance here. Anyway, I'll stop here and just come back after a few months when Truenas scale has reached production ready levels. I'll just switch to another linux distro and use zfs and gluster in its raw form for my small project.

Thanks.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I think with either 21.04 and 21.06 + truecommand 2.0, you'll be more than able to have this running in a breeze, no worries there :)

"I am no sysadmin, Im just a hobbyist."
As someone who is actively developing for TrueNAS SCALE (Apps in my case), I didn't find gluster "done enough" to start working with yet... So to give you some contrast: I would really not touch it as a hobbyist yet ;-)
 

tenknas

Dabbler
Joined
Mar 6, 2021
Messages
21
I think with either 21.04 and 21.06 + truecommand 2.0, you'll be more than able to have this running in a breeze, no worries there :)

"I am no sysadmin, Im just a hobbyist."
As someone who is actively developing for TrueNAS SCALE (Apps in my case), I didn't find gluster "done enough" to start working with yet... So to give you some contrast: I would really not touch it as a hobbyist yet ;-)
Ive not touched truecommand yest tbh, still finding time to read the documentation.

Ive not enabled gluster under truenas on my tests, Ive installed it via CLI, since from what Ive read from the release/dev notes and what you guys wrote here, its not yet ready under truenas. Since scale is debian, a lot of documentation for gluster is available for debian anyway, but since I dont know how truenas scale works in the background, I am not sure how it will affect installing and running gluster manually, hence why I wanted to get opinions.

Anyway, I would probably go with it and document my experience if I do make it work, if not well at least I learned something. :)

As always, thanks for actively responding. Kudos on the work on truenas.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Ive not touched truecommand yest tbh, still finding time to read the documentation.

Ive not enabled gluster under truenas on my tests, Ive installed it via CLI, since from what Ive read from the release/dev notes and what you guys wrote here, its not yet ready under truenas. Since scale is debian, a lot of documentation for gluster is available for debian anyway, but since I dont know how truenas scale works in the background, I am not sure how it will affect installing and running gluster manually, hence why I wanted to get opinions.

Anyway, I would probably go with it and document my experience if I do make it work, if not well at least I learned something. :)

As always, thanks for actively responding. Kudos on the work on truenas.
TrueCommand is (going to be) used to setup gluster on SCALE in the first place....

However: If you already installed it manually using the commond line (which is not the same as CLI), you probably already nuked your install.
I think I already explained you will fuck up the whole system if you do this, which you did. So i'm out.
 

tenknas

Dabbler
Joined
Mar 6, 2021
Messages
21
TrueCommand is (going to be) used to setup gluster on SCALE in the first place....

However: If you already installed it manually using the commond line (which is not the same as CLI), you probably already nuked your install.
I think I already explained you will **** up the whole system if you do this, which you did. So i'm out.
I'll just ditch the system if it gets you all so worked up about it. :wink:

Will comeback after a year probably.

If the mods can close this thread that would be great.
 

sophware

Dabbler
Joined
Oct 16, 2020
Messages
37
Right on tenknas. Sorry you had to deal with this BS. You saved me the experience!

It hasn't been a year, but are you thinking of coming back (if the turkeys here can be avoided)? I have three servers with drives, too.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
if the turkeys here can be avoided

Well, ornias may have been a bit harsh, but not incorrect. This has nothing to do with "turkeys here;" when someone is trying to tell you that the thing you want isn't ready for prime time, you should probably at least try to listen to that, because someone is trying to save you from a significant misadventure.

As someone who's been doing storage on FreeBSD since its earliest days, I've built lots of storage, including designing stuff that's been scaled into the petabytes. Gluster is a really cool technology, but it also comes with some substantial risks, in that there are lots of moving bits that can break. iXsystems has spent a decade perfecting FreeNAS, and now at least a year on SCALE. Even so, SCALE itself is still in relative infancy. Gluster upstream is generally considered to be stable, but the process of putting a usable user interface on it is complex, and until there are thousands of these successfully deployed on SCALE, and all the sharp edges and pointy bits are known and documented, my opinion is that strong caution is warranted.

The OP wanted to know if there was a step by step guide to this. There isn't, and not only isn't there one, but what IS there really isn't even "point and click" grade, and may not be for some time. I would say that it is actually pretty far into the "under development" side, very distant from having easy Linux howto guides on how to do it, sorry.

If the mods can close this thread that would be great.

That's a reasonable suggestion.
 
Status
Not open for further replies.
Top