Is there any guides or info on how to set up a multi node system?

Status
Not open for further replies.

Poppa

Explorer
Joined
Jun 3, 2017
Messages
87
I have 3 file servers in operation connected through a 40GBE switch. One is a Norco 4224 with 24 drives running Nappit, a Supermicro with 36 drives running FreeNAS v11 and a Chenbro with 45 drives running FreeNAS v11.3. I also have a another server with Open Media Vault that isn't being used at the moment.

I am interested in the idea of setting up a cluster using TrueNAS scale when the stable version is released but there doesn't seem to be much info on TrueNAS scale and how to setup nodes and clusters or if you will be able to use servers with existing FreeNAS installs and data on them.

I understand that Scale is still in an alpha state but any info on the expected setup process for Nodes and Clusters as well as how to set up a cluster using an unmanaged switch would be very helpful.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
This set-up should all work. Deploy SCALE on each node, setup a pool on each, run TrueCommand 2.0 nightly. You will find a cluster management function.

Documentation on the process is coming with 21.04
 

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
I'll be posting some more detail on this process + screenshots in the coming week or so, waiting for a few fixes to land :)
 

Poppa

Explorer
Joined
Jun 3, 2017
Messages
87
This set-up should all work. Deploy SCALE on each node, setup a pool on each, run TrueCommand 2.0 nightly. You will find a cluster management function.

So according to the TrueCommand documentation, in order to setup a local cluster, I have to use your paid cloud service just to manage my own local storage trough one interface rather then individually?

I know you guys want to make money but I am not running a business here. I am just a hobbyist and data hoarder. This is my own personal data and paying a monthly subscription to you based on how many drives I have locally is outrageous especially considering I don't run my servers 24/7 and at that point, considering how much that would cost me every year, I may as well just put everything on AWS.

Can't I just install TrueCommand on an old PC or a Dell poweredge and completely manage everything locally without your middleman cloud service?
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
To be 200% clear:
This is mostly about the Storage clustering...
Kubernetes clustering is also in the works but take a little more design-and-polish.
Though it's definately getting there too :)
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
So according to the TrueCommand documentation, in order to setup a local cluster, I have to use your paid cloud service just to manage my own local storage trough one interface rather then individually?

I know you guys want to make money but I am not running a business here. I am just a hobbyist and data hoarder. This is my own personal data and paying a monthly subscription to you based on how many drives I have locally is outrageous especially considering I don't run my servers 24/7 and at that point, considering how much that would cost me every year, I may as well just put everything on AWS.

Can't I just install TrueCommand on an old PC or a Dell poweredge and completely manage everything locally without your middleman cloud service?
Poppa, that's not quite right. TrueCommand is needed, TrueCommand Cloud is just one of the options... you can also run a local Docker container or VM. TrueCommand is free for the first 50 drives.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
Attached is a preview of the TrueCommand Ui setting up clusters.

eFu0ovGrn40eKIJ5TZx5FeltRshmIs66Qb8P8Egns2LBb8QVf4qWH2Bu9MI4VuZvKM6aNFKPQiW47Xa4Yln4W_-mCVHMOdQnsCdk9UynP_w7CbR4gsVY_rUyauU6VmC3BJwq8sGSlRA
 

Poppa

Explorer
Joined
Jun 3, 2017
Messages
87
TrueCommand is needed, TrueCommand Cloud is just one of the options... you can also run a local Docker container or VM. TrueCommand is free for the first 50 drives.

So two questions.

1. What does TrueCommand cost if you have more than 50 drives in your cluster? A reasonable one time lifetime license?

2. Is TrueCommand part of TrueNAS Scale or is it it's own program? Does it require an OS like Windows or Linux or is it self contained like FreeNAS/TrueNAS? The install page mentions Docker or VM but I have never setup a container or virtual machine before.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
For TrueCommand over 50 drives, budget for $2 per drive per year and includes email support. It is naturally a container application (no OS). Runs on a laptop, or takes only a few GB of RAM on a server. It is even one of the Container applications on the TrueChart application catalog.
 

Poppa

Explorer
Joined
Jun 3, 2017
Messages
87
For TrueCommand over 50 drives, budget for $2 per drive per year

For my 3 storage servers, that's $354 a year that I would be paying to IXSystems just for a little bit of convenience in managing my own property that I have sitting right next to me (well not right next to me because of the noise). THAT IS INSANE! Like I said, I understand you want to make money and want businesses to pay but for a home user, that's ridiculous. You have to know that is going to piss people off, right? You guys should rethink your pricing model because that seems extortionate and it would be cheaper to put all my data on a cloud service.

Any other questions or concerns I have about TrueNAS, Scale or TrueCommand are moot until you guys get it together.
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
For my 3 storage servers, that's $354 a year that I would be paying to IXSystems just for a little bit of convenience in managing my own property that I have sitting right next to me (well not right next to me because of the noise). THAT IS INSANE! Like I said, I understand you want to make money and want businesses to pay but for a home user, that's ridiculous. You have to know that is going to piss people off, right? You guys should rethink your pricing model because that seems extortionate and it would be cheaper to put all my data on a cloud service.

Any other questions or concerns I have about TrueNAS, Scale or TrueCommand are moot until you guys get it together.
There are multiple options depending on your situation and needs. We always try to keep a free option available. However, we also need to pay our staff and don't receive donations like a charity.

The 105 drives and 3 servers probably have a value of > $10,000. The 500+w of power has an effective cost of over $1,000/yr depending on the need for cooling. My guess is usable capacity is well over 200TB. If 200TB was put in AWS S3 (2c per GB per month), the bill would be up to $48,000 per year. Is this math correct? The TrueCommand license and support would be $200 per year... a tiny fraction (< 1%) of the Cloud storage costs.

The 3 servers can operate as 3 separate TrueNAS (CORE or SCALE) nodes and replicate the data between them. This is all for free. I agree that is great value.. it is very hard to beat free.

TrueCommand is there to help automate the processes and save time and reduce mistakes. The APIs and CLI of SCALE are all available for free, so you can manually stitch together the clusters for free. If time has zero value, then TrueCommand may not be worthwhile.

There are also free and trial TrueCommand licenses for contributors to the open source....please contact iXsystems (me) if you need this.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
For my 3 storage servers, that's $354 a year that I would be paying to IXSystems just for a little bit of convenience in managing my own property that I have sitting right next to me (well not right next to me because of the noise). THAT IS INSANE! Like I said, I understand you want to make money and want businesses to pay but for a home user, that's ridiculous. You have to know that is going to piss people off, right? You guys should rethink your pricing model because that seems extortionate and it would be cheaper to put all my data on a cloud service.

Any other questions or concerns I have about TrueNAS, Scale or TrueCommand are moot until you guys get it together.

$354 a year at $2 per drive implies 177 drives.

So you've paid for one hundred seventy seven drives, presumably at a hundred or more dollars apiece, plus the servers to hold them, so we're talking maybe $20,000-$30,000 invested, and you're not willing to pay $354 a year for software? That's like paying for two more drives.

Exactly what do you want the pricing model to be? Would you like the developers to work for free? A measly $354 a year is nothing in the grand scheme of things, most software licensing is substantially more than that. VMware's hypervisor licensing often costs as much as the servers running the hypervisors, just for one obvious example.

And cheaper to put all your data on a cloud service? That's insane. Assuming you have 1TB drives and you have 177 of them, your pool capacity is probably 100TB. Simply *storing* 100TB of data, no access, is likely to run you four figures PER MONTH. Good luck with that, don't let the egress charges hit you in the butt on the way out...

Unlike Nexenta, the iX people have been very generous with handing out FreeNAS and TrueNAS for free. If you find it objectionable that they'd like to sell you some inexpensive software that helps pay the bills, then don't buy the software. However, to call it insane, or to suggest that this is going to piss people off, or that it is extortionate, well, I find that disappointing.

As a professional infrastructure person, I "pay" iXsystems for FreeNAS by contributing time and expertise here in the forums in order to help people adopt their product, which hopefully translates to additional visibility and sales of their commercial products. I also don't "sell" against them, either for hardware or consulting, though heaven knows it would be easy enough to.

I don't care for the mentality that I'm reading here, though,. that somehow iX owes you something for less than what they've decided to sell it for. If you don't want to pay for it, you still have perfectly usable NAS units.
 

Poppa

Explorer
Joined
Jun 3, 2017
Messages
87
I seem to have struck a nerve with some people here so I am sorry for that.

Yes I have a lot of hard drives but I have been buying the cheapest drives I can. All my hardware is old or from surplus. I only run them when I want to transfer something. In short, I haven't spent as much as some may think and yes, I think being charged $354+ a year for something I already own is ridiculous. If IXS want's to charge for their software then they can do that and it would be perfectly reasonable but that is NOT what they are doing. In reality, they would be charging me to "rent" the software and only for an extra feature and some convenience at the price of about 3 new drives a year. This is as asinine as Bidens plan to charge people a tax for every mile they drive. I find it insulting that they would charge people every year based on the amount of something they already own rather then a flat rate based on their use case. Paying that much+ every year to rent software is crazy. I buy all my games and have a little more than 150 games in my library and the total amount of money I paid for all my games is less then the yearly cost IXS wants to charge me for that extra feature and convenience. Again, I am a home user and am not running a business.
 

bodly

Dabbler
Joined
Jan 1, 2021
Messages
16
For my 3 storage servers, that's $354 a year that I would be paying to IXSystems just for a little bit of convenience in managing my own property that I have sitting right next to me

Give me a little bit of convenience or give me death! The key word there being "give".

If you don't think that it adds enough value, then don't use it.
 

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
I seem to have struck a nerve with some people here so I am sorry for that.
It's not about a nerve, I to would prefer a little more lenient limit (like 75 drives).
But at 177 gosh damn drives, you're pretty deep into enterprse drive-counts and that comes with an enterprise price. Thats the case with just about every company making enterprise storage software or solutions.

I only run them when I want to transfer something.
So do you NEED high available clustered storage? Because you don't HAVE to use it.
It's a freemium addon. Which functionallity is basically completely free. The premium part is having a fancy gui to make the cluster.

I find it insulting that they would charge people every year based on the amount of something they already own rather then a flat rate based on their use case.
They do: use case is "Enterprise drivecounts". With a price of "2$ per drive".
We are talking scaleable storage here, thats always based on hardware (just like any enterprise software package tbh).
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I seem to have struck a nerve with some people here so I am sorry for that.

Yes I have a lot of hard drives but I have been buying the cheapest drives I can. All my hardware is old or from surplus. I only run them when I want to transfer something. In short, I haven't spent as much as some may think and yes, I think being charged $354+ a year for something I already own is ridiculous. If IXS want's to charge for their software then they can do that and it would be perfectly reasonable but that is NOT what they are doing. In reality, they would be charging me to "rent" the software and only for an extra feature and some convenience at the price of about 3 new drives a year.

Again, lots of software is sold this way. Many companies have discovered that their business models fail if they cannot guarantee a revenue stream to continue to pay their people, and in fact, the failure to charge people on an ongoing basis has actively led to evil issues.

In particular, for years, IoT devices such as home NAT gateways ("routers"), IP cameras, smart televisions, smartphones, etc., have been sold as abandonware -- long lifetime devices which may see firmware updates for a handful of years, or even just a few months, or maybe not even at all. This is a massive problem in the industry, and it comes from an economic model where the company selling the device is under intense pressure to sell an inexpensive device (probably compromising quality in the process), and then once the sale is made, there is no additional revenue to cover a much more expensive phase of the device's lifecycle: keeping the firmware patched, secure, and distributed to devices already in the field. While the initial software design might have involved a dozen developers over the period of a year, the need for ongoing support really needs you to retain maybe two or three people capable of touching all aspects of that device's firmware, because you never really know where what portions of the firmware a CVE or other security notification will involve. But that's not what happens.

Anyways, lots of things are sold on an annual basis, and are better for it. Many games require a subscription, which keeps the company invested and cranking out updates. Quicken spun off from Intuit a few years ago and moved to a subscription model, which guaranteed them a revenue stream that has allowed them to succed. Hell, even the Cisco phones in the office here are $20/each per year for ongoing support and software licensing.

This is as asinine as Bidens plan to charge people a tax for every mile they drive. I find it insulting that they would charge people every year based on the amount of something they already own rather then a flat rate based on their use case.

Really? Seems sensible from a certain point of view. I've got a truck in the driveway that saw about 200 miles driven in the last year during the pandemic. I would hate to pay a flat rate just to have a vehicle sitting in the driveway, which, ironically, is what the Republicans around here have enacted for hybrid vehicles.

We already pay for roads in a lot of areas with taxes on gasoline, which used to scale pretty well as it translated fairly well to miles driven, but now with electric vehicles, that's a bit of a problem, which is why governments have been looking for ways to fund roads through something other than a gas tax. You can think it "asinine", but the world is changing, and the construction workers who build roads need to be paid.

While I am little-l libertarian, I see huge value to our society in having modern infrastructure systems. Transportation, water, energy, communications, etc. We're seeing what thinking small about these issues gets us -- the tragedies in Texas a month ago because most power vendors chose not to winterize was totally avoidable. Telecom vendors managed to derail the mid-90's plan to bring high speed communications to the entire country, and have been dragging their feet for a quarter of a century on the issue. Finding ways to pay for better infrastructure is going to be challenging, but history shows that it pays off. I don't expect that it has to be done the way my politics would prefer, and I'm even adult enough to understand the fallacies of the "libertarian model" for these things in practice; all that needs to happen is that we need to come to agreement on some viable way forward.

Charging people a tax for every mile they drive would seem to me to be exactly in line with so-called "user fees" that Republicans used to be so fond of.

Paying that much+ every year to rent software is crazy. I buy all my games and have a little more than 150 games in my library and the total amount of money I paid for all my games is less then the yearly cost IXS wants to charge me for that extra feature and convenience. Again, I am a home user and am not running a business.

You paid two and a half bucks per game? For what games?
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504

amiskell

Patron
Joined
Jun 25, 2015
Messages
266
iX should just give away everything they create for free? They already provide a enterprise grade NAS software at zero cost. TrueCommand is separate software product from their NAS offerings and not part of SCALE itself.

Considering that you're saying you only turn things on when you want to transfer something, it really sounds like TrueCommand isn't for you particularly.
 
Last edited by a moderator:

Poppa

Explorer
Joined
Jun 3, 2017
Messages
87
But at 177 gosh damn drives, you're pretty deep into enterprse drive-counts

"Enterprise" drive counts would easily be in the thousands. I understand that most people don't have servers in their homes or more than a few drives in their PC but any business that requires lots of reliable data storage is going to have way more than what I have. There are other people besides me on r/DataHoarder or r/homelab who have about as many as or more drives than me. IXS needs to make the free limit at least 250 if they want to be reasonable.
 
Status
Not open for further replies.
Top