Is there any guides or info on how to set up a multi node system?

[HoneyBadger]

You all should be thankful for Poppa to pop up and still trying to make his stance without any support against the whole forum. Including mods and administrators (I feel bad for that kind of publicity you get from a professional perspective for that...)

He brings up valid points of critique. Instead of taking it and trying to solve the problems, you just nuke the hell out of him. you even attended some sheep following the leaders in this unfair battle... group dynamics are hard to handle, he :P??

I haven't seen anything unprofessional from the administrative/engineering team, in fact one of them not only highlighted the availability of the APIs and CLI in a free fashion but even offered an avenue to obtaining a TrueCommand license gratis by contributing to the development.

TrueCommand is there to help automate the processes and save time and reduce mistakes. The APIs and CLI of SCALE are all available for free, so you can manually stitch together the clusters for free. If time has zero value, then TrueCommand may not be worthwhile.

There are also free and trial TrueCommand licenses for contributors to the open source....please contact iXsystems (me) if you need this.

There is a certain sentiment that I believe hits at the core of the argument here, and that is simply "Time has value."

Whether that is the time saved by using the paid TrueCommand software to create the cluster, or the time invested by the development team to create the tools, or the time spent by those of us bringing our experience and knowledge to the forum - it all has value.

And certainly in the current climate of "work from home" and employers attempting to blur the lines of the work/life balance, it's extremely relevant that some people have different values on their time, and it may not be a flat scale.

Many people have "sold" the first 35 hours of a given week of their time to an employer. If that employer asked for another 35 hours in the same week; well, I imagine the price would rise significantly.

Food for thought.

 

[amiskell]

the pricing policies from lifetime licenses to software-as-a-service models is bloodsucking to me. as a business, yes, as long as yor cashflow works well, its okay, but for a private person its not a perfect good model. what would be a productive, better or more fair way? could it be more granular? or even dynamic?

Just as a point, a lot of software companies have been moving to this model because to constantly maintain and update software costs money, you have to pay programmers, QA people, pay for infrastructure, etc.

As a side effect, the subscription model allowed for customers to have license to the latest version of software at all times.

While the pay once and use the software forever (including updates) is nice, it's just not the reality anymore. As I said, it costs money to constantly maintain and update software, to add new features and the like. Some companies still have the model of pay once for a specific version and get updates/new versions at an additional cost, but that's just not where the industry is heading in general, especially with most software including some kind of cloud component to it which requires the software vendor to constantly pay for infrastructure.

Am I a huge fan of it, not really, but being in this industry and seeing how it works from a company's bottom line perspective and real impacts it has on people (maintaining staffing based on revenue, etc) I understand why this model is becoming more and more popular.

the code base once was totally free, well, as it seems to me, he is just plain right on that point. I can understand that people take it with a grain of salt, that free software gets "used to lure people in", or at least it might feel so to some people... its nice that you turned it into a well running business and that a bunch of people can live of it, but this burden of history is yours, so just take it, as you already live it :P its not bad, especially not in the US to do such opportunistic things. and to be fair, you still offer a lot for free and I dont see him complaining a lot about this, but its understandable, valid and also already gone with the wind of time...

The clustering feature is free within SCALE, you just have to do it via the CLI. TrueCommand is a wholly separate product not derived from FreeNAS/TrueNAS itself. It's a separate product you can use to control multiple systems (clustered or not) via a single GUI interface.

charging by drive count is somehow a strange criteria. a drive can have from a few (hundred) gb up to 18?tb so that doesnt tell anything about the machine one is operating. I remember charging per CPU, than per CPU CORE, but all that doesnt reflect on the value of a system. what would be a better way? (benchmarking the hardware? hardware price list?) lets just have some visions of a better system there

UnRAID, arguably one of FreeNAS/TrueNAS's "competitors", uses exactly this model. There are different tiered licenses based on number of drives in the system. Granted, they do use the lifetime license model currently, but have recently stated that the lifetime license model may not be sustainable as time goes on and they may need to move to subscription based services/add-on services.

 

[ornias]

You all should be thankful for Poppa to pop up and still trying to make his stance without any support against the whole forum. Including mods and administrators (I feel bad for that kind of publicity you get from a professional perspective for that...)

No. I'm not gratefull for a bucket of rethorical mistakes. At ALL.

He brings up valid points of critique. Instead of taking it and trying to solve the problems, you just nuke the hell out of him.

I already tried to move this discussion somewhere which is discussable/solvable: Asking for a little more room from iX.
Like I said: there is a place and a way to discuss things like civil beings. But this is just plain choosing beggar behavior.

The complaint isn't "I've this reasonable setup and I want a little more room". The complaint is "I've an extreme setup and I feel entitled to have this cheaper.

you even attended some sheep following the leaders in this unfair battle... group dynamics are hard to handle, he :P??

It might also be wise to not start calling people out on behavior you barely seem to understand.

And actually, I try to stay nicer despite of group dynamics. If it was up to me, I would've utterly trashed him with pages full mopping up his attitude, rethorical mistakes and entitlement. Before I even start with explaining what is wrong about his opinion.

1. the pricing policies from lifetime licenses to software-as-a-service models is bloodsucking to me. as a business, yes, as long as yor cashflow works well, its okay, but for a private person its not a perfect good model. what would be a productive, better or more fair way? could it be more granular? or even dynamic?

TrueCommand is NOT made for homeuse. Period.
Though I'm annoyed by these arbitrary limits being enforced in-code.

However: I agree the limits are a bit too strict. At least give us about 75 drives to build an actually decently sized demo cluster (3x4U 24 Bay for example).

1. the code base once was totally free, well, as it seems to me, he is just plain right on that point. I can understand that people take it with a grain of salt, that free software gets "used to lure people in", or at least it might feel so to some people... its nice that you turned it into a well running business and that a bunch of people can live of it, but this burden of history is yours, so just take it, as you already live it :P its not bad, especially not in the US to do such opportunistic things. and to be fair, you still offer a lot for free and I dont see him complaining a lot about this, but its understandable, valid and also already gone with the wind of time...

I think free isn't the problem, open-source is. It's a shame they moved away from opensource and only because they used some undescribed upstream tooling. If it's free or not, would not be something I care much about, thats theirs to decide. The legacy wasn't build on Free, It was (mostly) build on opensource.

1. charging by drive count is somehow a strange criteria. a drive can have from a few (hundred) gb up to 18?tb so that doesnt tell anything about the machine one is operating. I remember charging per CPU, than per CPU CORE, but all that doesnt reflect on the value of a system. what would be a better way? (benchmarking the hardware? hardware price list?) lets just have some visions of a better system there
   

1. 
   It's not a strange metric at all, because ZFS performance is mostly depending on the drive count. As well as clustering also depending on the drive count. Using drivecount automatically forced users to make choices between:
   A. Faster storage but less space
   B. Slower storage with more space
   
   Within reason ofc.
   
   

   [*]he gives you a lot of respect and credits for your work in many different ways and I dont see any official representative here treading him equally respectful

   
   He does not give respect at all. He is an utter entitled douche. If this is what you call respect...
   As a self-proclaimed arrogant bastard, I can ensure you: he is not respectfull AT ALL.
   
   Also you make it sound like your big friend here, has had feedback from a lot of representatives, which he has not. Remember: Moderators here are NOT iX representatives or employees.
   
   
   

   [*]its a business opportunity here, to make points and show more openness, open your heart for the weak and talk about it, and with "dragon dreaming" in mind, the naysayers are your biggest asset if you know how to use their information correctly. if you take them into your design process and work till they are satisfied, you will have a way better and more competitive product. (thats why I wrote my first sentence :) )

You make it sound like we are a bunch of iX employees defending the product. While we in fact are opensource enthousiast users, not employees, who just disagree with the way he presents a self-proclaimed problem that's just some GUI not being free.

I hope for all parties to learn a lot from this conversation and that everyone benefits for the greater good we all rely on :)

You can sugar coat it all you want, but this is also grossly arrogant as well.
Get of your comfortable moral highground and learn about people disagreeing.

 

[ornias]

Partly agree, but only partly. That has been the dominant model in the B2B world for quite some time and it's just how it is. We fork out 30k per year for Atlassian products. OTOH the productivity of our teams has soared since we replaced a zoo or Redmine, Mediawiki and tons of other not-quite-integrated stuff with JIRA and Confluence. Privately I still refuse to invest in any mobile "app" that is not buy once but a subscription model.

Just the fact that it's normal doesn't make it right though. I think it's wrong, at least the prices are.
But: the problem here is trying to defend an entitled statement by shouting "how strange is this", is indeed wrong. It's normal for B2B, just not "right", but thats a LOT bigger discussion than just 1 trialware iX UI.

The code base is still totally free. 100%.

Well, lets say 95%, this clustering UI should've been put into SCALE. There is no reason IMHO this should've been paywalled into truecommand.
Actually, I would've prefered the proxmos solution when it comes to clustering UI. versus a seperate container app, regardless of pay.

It seems the only reason TrueCommand exists, is the fact they can close-source it despite the rest of TrueNAS SCALE being opensource. Thats not behavior I support, Which I also made clear to iX both privately and publicly. But after making my stance clear, I left it at that.

To be clear: TrueCommand exists because their venture capitalist owner wants higher margins.
It's either TrueCommand (Cloud) or them being forced to close-source the whole project by the venture capitalist firm (whose name is quite easily accessable because they own all the board seats)

TrueCommand is an additional Enterprise Management product. And there even is a free tier! For 50 disks! Who in his sane mind thinks 50 disk drives is a common hobbyist setup?

*hand up*
Well not true hobbiest, but I consider up to 24*3 bays acceptable for homelab standards.

Of all people I know who use Free/TrueNAS for their private network I am an absolute power user! I run three Supermicro servers that including memory and disks easily cost me 4k€ or some such. And still I count just 20 drives.
We even run the free tier at work, because our two large TrueNAS based hypervisor systems that are only used for VMs have 6 NVME drives each. With a maximum capacity of 10 drives per system.

There are many different ways of being a power user.
But when you actually start clustering on top of ZFS, the amount of disks goes up considerably when you take added redundancy requirements into account. ;-)

I run TrueCommand because I am curious how the product develops. If we get anywhere near a "vSphere" management capability, I'd happily pay a couple of hundred per year for that should I ever deploy more systems and cross the 50 disk limit. See our Atlassian bill above. Lookup VMware vSphere pricing. And no, there is absolutely *no* hobbyist free tier for vSphere apart from the isolated hypervisor (ESXi). If you want the management console, VM migration and stuff, the "developer network" membership is 200 per year (!) if I remember correctly. Definityle that order of magnitude.

To be fair: even if you want to stay within the Free-Tier, there is a lot you can do.
For example:
5 2U servers with 12 3.5" bays. Excluding system disks, is precisely 50 drives. ;-)
Thats quite a decently sized cluster which can contain insane amounts of power!

 

[HoneyBadger]

Lookup VMware vSphere pricing. And no, there is absolutely *no* hobbyist free tier for vSphere apart from the isolated hypervisor (ESXi). If you want the management console, VM migration and stuff, the "developer network" membership is 200 per year (!) if I remember correctly. Definityle that order of magnitude.

I assume you're referring to VMUG EVALExperience, which is not only priced at USD$200/yr but is also expressly forbidden from being used in any commercial capacity. Get caught using VMUG licenses for your business and you're in for a world of pain. This is a restriction that doesn't exist in TrueCommand.

Well not true hobbiest, but I consider up to 24*3 bays acceptable for homelab standards.

I like the suggestion of a 75-disk limitation as the math works well both for 3x 4U 24-bay 3.5" servers, 3x 2U 2.5" server, or 6x 12-bay 2U 3.5" server. (Although the 50-disk works well now as you explained for 5x 2U 12-bay, less system disks, or 3x 3U 16-bay, if system disks are separate bays or internal devices.)

 

[ornias]

I haven't seen anything unprofessional from the administrative/engineering team, in fact one of them not only highlighted the availability of the APIs and CLI in a free fashion but even offered an avenue to obtaining a TrueCommand license gratis by contributing to the development.

I want to highlight this:
There is a LOT of room with iX to get around such arbitrary disk limits/licenses and other issues. They aren't assholes and completely understand that an average datahoarder may walk into issues or community projects also need resources.

However: They are not some sort of secret santa.
Being a choosing beggar never got anyone very far.

There is a certain sentiment that I believe hits at the core of the argument here, and that is simply "Time has value."

Precisely!


I want to end this with a warning:

Clustering is in ALPHA!
- I think we should wait a bit longer how iX is going to work out the CLI version of clustering.
- For all we know this solution will be super simple to use and fully supported using the new iX CLI solution.

However: If it's not supported, requires midclt hacking or otherwise frustrates the user into TrueCommand, then we SHOULD pick up our digital pitchforks. And I would be the first on the baricade.

 

[ornias]

I like the suggestion of a 75-disk limitation as the math works well both for 3x 4U 24-bay 3.5" servers, 3x 2U 2.5" server, or 6x 12-bay 2U 3.5" server. (Although the 50-disk works well now as you explained for 5x 2U 12-bay, less system disks, or 3x 3U 16-bay, if system disks are separate bays or internal devices.)

It's indeed a discussion that should be had in due time. :)
btw: Any links to some relatively cheap (second hands?) 3U 16 bay systems? :D

 

[Patrick M. Hausen]

I assume you're referring to VMUG EVALExperience, which is not only priced at USD$200/yr but is also expressly forbidden from being used in any commercial capacity.

Yes. I researched if I could get the full vSphere features for my home lab, but deemed that too expensive for just toying around. All the private-yet-productive workloads are easily covered by CORE.

However: If it's not supported, requires midclt hacking or otherwise frustrates the user into TrueCommand, then we SHOULD pick up our digital pitchforks. And I would be the first on the baricade.

If there's an API and enough people are sufficiently interested, there will be a UI.

 

[ornias]

If there's an API and enough people are sufficiently interested, there will be a UI.

If you consider TrueCharts and Jailman, I think you've an idea what my "Digital Pitchfork" looks like Patrick ;-)

 

[Patrick M. Hausen]

To be clear: TrueCommand exists because their venture capitalist owner wants higher margins.
It's either TrueCommand (Cloud) or them being forced to close-source the whole project by the venture capitalist firm (whose name is quite easily accessable because they own all the board seats)

In that case they should work hard on features that make it worth buying TrueCommand. At the moment I can get better reports from Grafana, for example.

 

[Patrick M. Hausen]

If you consider TrueCharts and Jailman, I think you've an idea what my "Digital Pitchfork" looks like Patrick ;-)

You have my sword.

 

[ornias]

In that case they should work hard on features that make it worth buying TrueCommand. At the moment I can get better reports from Grafana, for example.

Well, I can assure you TrueCommand looks very nice in the eyes of a (non-IT trained) venture capitalist during the board room presentation ;-)
So for all it's worth, it worked.

But jokes, pitchforks and swords aside:
I looked at TrueCommand before and found it... "bare".

 

[Kris Moore]

To be clear: TrueCommand exists because their venture capitalist owner wants higher margins.
It's either TrueCommand (Cloud) or them being forced to close-source the whole project by the venture capitalist firm (whose name is quite easily accessable because they own all the board seats)

Sorry, but if that isn't the biggest colossal nugget of FUD I've read this decade, I don't know what is... iX is not VC funded (we're proud of that) and TC doesn't exist because anybody forced it on us. It was developed in response to asks by our enterprise customers explicitly and was aimed at that segment. Clustering made a lot of sense for us to develop in that product due to where it sits in the topology, and lets us do neat things with regard to talking to a group (cluster) of systems at once from the outside, handle failures of individual nodes, etc etc.

 

[ornias]

Sorry, but if that isn't the biggest colossal nugget of FUD I've read this decade, I don't know what is...

I'm okey in being proven wrong, but it isn't needed to call me out on FUD of the decade.
I don't think it's fair to throw that at someone who did research to come up to said conclusion.

That being said: I should've been more clear it was an estimate of what could be the cause, not a statement of fact.
My appologies!

iX is not VC funded (we're proud of that)

So you're whole board consists of folk from "Olympus Parters" according to bloomberg, "Olympus Parters" a firm that only does venture capital specialising in "value added investment" and does not have "board for hire" services as far as I can find. I think it's a fair expectation from my side to think this firm (which only does venture capital) has done a capital injection.

Then again: I COULD be wrong here, it's an analysis based on data form bloomberg. Which we all know isn't the best of sources!

and TC doesn't exist because anybody forced it on us. It was developed in response to asks by our enterprise customers explicitly and was aimed at that segment.

I might be wrong ofcoarse, to be clear to everyone: Above was my ANALYSIS not a statement of fact.
I don't know for a fact that this was forced on anyway!

Clustering made a lot of sense for us to develop in that product due to where it sits in the topology, and lets us do neat things with regard to talking to a group (cluster) of systems at once from the outside, handle failures of individual nodes, etc etc.

Sorry, prefer the solution from Proxmox and others, with a true single pane being part of the product. But thats preference.

 

[Kris Moore]

So you're whole board consists of folk from "Olympus Parters" according to bloomberg, a firm that only does venture capital specialising in "value added investment" and does not have "board for hire" services as far as I can find. I think it's a fair expectation from my side to think this firm (which only does venture capital) has done a capital injection.

Then again: I COULD be wrong here, it's an analysis based on data form bloomberg. Which we all know isn't the best of sources!

I'd love to see that source / link, are you sure you found the right 'iXsystems'? First I've ever heard of Olympus Partners and I've been at iX for 15+ years... What board members are you referring to specifically?

 

[danb35]

I COULD be wrong here, it's an analysis based on data form bloomberg. Which we all know isn't the best of sources!

Yeah, we know they nailed the SuperMicro hack...

 

[ornias]

I'd love to see that source / link, are you sure you found the right 'iXsystems'? First I've ever heard of Olympus Partners and I've been at iX for 15+ years... What board members are you referring to specifically?

Well, that would explain my major fuckup. >.<

iXsystems Inc - Company Profile and News

Company profile page for iXsystems Inc including stock price, company news, executives, board members, and contact information

www.bloomberg.com

Yeah, we know they nailed the SuperMicro hack...

Yeah, if that's truely wrong... I'm never going to base anything on bloomberg, not even bussiness info!
Now i'm truely annoyed :P

 

[Poppa]

if they think it's a good deal

Which it's not. That's my point.

The ONLY thing you are missing is a fancy GUI for the clustering.

Which IXS want's to charge me $350+ a year for WHICH IS INSANE! That's been my point.

Yeah, thats why I would personally prefer a limit of 75 drives for the free tier instead of 24. That just adds up to a redunant cluster of 3 basic industry standard 24 bay 4U cases filled with drives.

Those 24 drive SAN shelves have kind of been going out of style I think. Aren't most enterprises using the 45-60 drive top loading units that BackBlaze uses? A 42U rack can fit 10 of those plus a 2U switch so 2 racks can hold 1200 drives. I may be using old hardware myself but I don't think the free limit should be based on rack density.

-----------------------------------------------------------------------------------------------------------------------------

Like I said, I understand IXS's position but most of the people here don't seem to understand mine and are being overly agreesive. IXS considers their software to be enterprise in nature, fine. But NAS's are becoming more and more common for a variety of reasons including amateur photographers, streamers/youtubers, home security, Plex and general data backups. When it comes to a NAS, FreeNAS is kind of the gold standard for these devices because of it's features and maturity which the FreeNAS community as well as myself is grateful for. But I feel like IXS is being shady and trying to take advantage of FreeNAS's status and legacy by charging some users too high a price for a single useful but not critical feature based soly on how much hardware they own which IXS arbitrarily defines as either consumer or enterprise based on what they think is typical for a consumer or enterprise user. The value just isn't there. A GUI is NOT worth what is being asked for it but for someone like me is necessary. The whole reason why I never set up a cluster such as Ceph in the first place is because the clustered file systems I looked into all required the use of a CLI and me not being an advanced enough user who doesn't understand the fundamentals of Linux/Unix doesn't trust CLI's and that something won't go wrong if a command is entered incorrectly, furthermore, I don't have the patience or will to memorize and keep track of all the commands I would need to know to properly setup and manage a storage cluster. I have a lot of drives because I am a data hoarder and I use raid-Z3 for maximum redundancy, that is why I have so many drives. That redundancy has already cost me and now a 3rd party wants me to pay even more because of it. That is really shitty.

 

[Kris Moore]

Lol, speaking of "Free" this is what I got when I hit that Bloomberg link :) I'll take your word for it though, sounds like wrong company ;)

 

[ornias]

Lol, speaking of "Free" this is what I got when I hit that Bloomberg link :) I'll take your word for it though, sounds like wrong company ;)

Okey.. the irony...
Anyway, I did check the address... and well, some guy named "Morgan" showed up on it that sounded somewhat familiar :P

How about some "Fair Use" ;-)

To be fair:
I've NO IDEA how to check US companies thoroughly... Not my specialty :P