RE-Evaluating TrueNAS from the Historical Perspective..

[Kris Moore]

Not all of us. I like my jails and I'm still researching what I'm going to do with them when TrueNAS does finally drop FreeBSD as it's base. I detest Docker.

I can understand this. I was in that camp at one point. But I embraced the change and honestly trying to run Jails now is just so painful. There is a reason why FreeBSD got left behind when it comes to anything approaching mainstream containerization. But I can still understand how some folks still like that DIY approach that Jails brings. But for me personally, once I took a week and figured out some of the ins-n-outs of Dockerfile, and runtime options, it was more than sufficient for all my needs, hands down. Saved me countless hours of time in the long run.

 

[Samuel Tai]

I like jails, but the wider industry basically abandoned them as fancy chroots, and moved on to virtualization and containers for much greater flexibility, granularity, and scalability. When 11 offered the Docker host bhyve VM, I could already see this was going to be the path away from plugins/jails.

 

[winnielinnie]

I share the same sentiments as @Jailer, @danb35, and @Samuel Tai.



A friend of mine showed me their SCALE setup, using a handful of Apps, and the entire thing looked like a confusing mishmash of a mess. Pages, menus, and lists that concern "Apps" seemed rushed and non-intuitive. Nothing was straightforward nor clearly explained in the GUI. All the text fields and drop-downs seemed to be slapped on for the sake of time, to the point that manually using the shell seemed less daunting than the official GUI method. Not to mention the excess slough of old "images" and "deployments".

There's "a lot going on" and it felt too crowded.

Hilariously, the GUI would hang on certain drop-down menus if there was only one "option" to choose from. (For example, you click on the drop-down to see the available selections, and if there was only one choice, the GUI would hang for about 10 seconds. This did not occur on drop-downs with multiple selections available.)



On the other hand, in regards to Core, to treat each jail as its own "mini server", contained within itself, and without the overhead of virtualization is truly under-appreciated.

 

[nabsltd]

If you are hanging your hat on the cluster feature only, then sure, it's a technical preview.

Of course that's the only thing I care about.

I've got dozens of scale-up (i.e., single node) storage options, from completely free to various levels of cost. The hard part is choosing, not finding.

I've also got a reasonable handful of clustered hypervisors (again, from free to "how much can you afford") to run code in various ways, including VMs, different types of containers, etc. Again, the hard part is choosing.

But, find me a clustered storage system that supports block, NFS, and SMB, and has a relatively easy to use GUI interface. OK, now find me one that doesn't cost an arm and a leg and doesn't have insane vendor lock-in for hardware. Listen to the crickets.

Ten years ago, I built a 6-node, 320TB gluster system as a proof of concept for storage that was relatively safe, relatively fast, but didn't have the EMC price tag. It used the same kind of hardware recommended for TrueNAS...Supermicro chassis and motherboard (my thought was that if they were good enough for EMC, they should work fine), ECC RAM, "NAS" hard drives, etc., The only difference was everything was brand new and I was using hardware RAID instead of ZFS. The hard part was setting up gluster and then trying to get a system to talk to it that couldn't use the gluster native client. I have dreamed of an easy-to-use GUI setup that allows Windows users to get access to that kind of cheap, scalable storage since then. TrueNAS SCALE sounds like it will be what I want, but I couldn't give a rat's ass about any of the other features (containers, etc.), since those are covered dozens of times over. I know other people don't share my sentiment, so I have no issue with those features being included.

Again, though, all the features in the current SCALE release are things we already have. It's nothing new, and nothing amazing. Easy to use scale-out storage without a high software license cost or vendor lock-in...that's something amazing.

 

[NickF]

Of course that's the only thing I care about.

I've got dozens of scale-up (i.e., single node) storage options, from completely free to various levels of cost. The hard part is choosing, not finding.

I've also got a reasonable handful of clustered hypervisors (again, from free to "how much can you afford") to run code in various ways, including VMs, different types of containers, etc. Again, the hard part is choosing.

But, find me a clustered storage system that supports block, NFS, and SMB, and has a relatively easy to use GUI interface. OK, now find me one that doesn't cost an arm and a leg and doesn't have insane vendor lock-in for hardware. Listen to the crickets.

Ten years ago, I built a 6-node, 320TB gluster system as a proof of concept for storage that was relatively safe, relatively fast, but didn't have the EMC price tag. It used the same kind of hardware recommended for TrueNAS...Supermicro chassis and motherboard (my thought was that if they were good enough for EMC, they should work fine), ECC RAM, "NAS" hard drives, etc., The only difference was everything was brand new and I was using hardware RAID instead of ZFS. The hard part was setting up gluster and then trying to get a system to talk to it that couldn't use the gluster native client. I have dreamed of an easy-to-use GUI setup that allows Windows users to get access to that kind of cheap, scalable storage since then. TrueNAS SCALE sounds like it will be what I want, but I couldn't give a rat's ass about any of the other features (containers, etc.), since those are covered dozens of times over. I know other people don't share my sentiment, so I have no issue with those features being included.

Again, though, all the features in the current SCALE release are things we already have. It's nothing new, and nothing amazing. Easy to use scale-out storage without a high software license cost or vendor lock-in...that's something amazing.

I totally agree with these observations and they 100 percent align with my own professional experience. SCALE for me was always about the promise of the end goal of clustered storage as you outline.

The steps along the way are basically cool to play with, but as you’ve said there are so many options to do all of those things that it never really mattered for me. It takes literally like 5 minutes to deploy an Ubuntu VM and snap install docker. Portainer makes
Managing docker (and kubernetes clusters!) simple. ESXI is free for home users and VMUG gives you all the bells and whistles for cheap. XCPNG and Proxmox cover the same bases in their own ways.

But clustered storage? Crickets indeed

 

[indivision]

I still think this just boils down to "why doesn't Scale fully support clustering like Proxmox?!"

But, that is ignoring the fact that many (most?) NAS users don't have any need or interest in clustering and Proxmox is missing many features that those users want in Scale. The fact that Scale doesn't (yet) support clustering does not render it a shitty version of a super soaker by any reasonably objective basis. That is purely a "me me me" opinion.

I also don't think it's fair to beat up on Corral over the disk replacement issue. It's not like that was a secret. If users didn't want to deal with that, they could have simply remained on pre-Corral versions until that feature was implemented as planned.

As far as jails vs. docker goes. The only thing I really miss from jails is the ability to work with a blank one. If there was some easy way to work on a custom/blank docker image that would be nice. I know VM is basically that. But, resource usage and network setup is more involved than the blank jail.

That said, the bottom line speaks for itself. In months, there are exponentially more apps available via Kubernetes than were available via years of the jail system being in use.

Apps in Scale really are not that complicated to use. You can do it!

 

[Ericloewe]

On the other hand, in regards to Core, to treat each jail as its own "mini server", contained within itself, and without the overhead of virtualization is truly under-appreciated.

Containers on Linux are not fundamentally different. The main difference is that on FreeBSD there are typically two userland tarballs to choose from, whereas on Linux there are many, many weird and wonderful binary blobs and a glorified shell script replacement to build upon them.

Jails vs. Docker is, to a large extent, about subtle, but relevant differences, and mixes up the layers: Docker is actually comparable to a Jails manager like iocage, an there's no reason why you couldn't run the Docker applications on FreeBSD, with adaptations to use jails - but you'd miss the killer app that is Dockerhub, since everything there is compiled for Linux (yeah, yeah, Linux compatibility on FreeBSD is a thing, but let's keep this simple for now).
Docker likes to pretend that containers are black boxes, but that's nonsense. It's all running on my machine, I can go peek into whatever I want, but Docker makes such things needlessly complicated for very little gain - the standard procedure for accessing the filesystem used by one or more containers is, bizarrely, to spin up a container! Instead of, you know, allowing them to be mounted by the host...

 

[garm]

Not all of us. I like my jails and I'm still researching what I'm going to do with them when TrueNAS does finally drop FreeBSD as it's base. I detest Docker.

This, if TrueNAS ditches BSD i guess Im back to building my own systems

 

[DaSnipe]

SCALE is light years better in Apps and Hardware support, and works great personally, dunno the hate. Anyways CORE is going to be a niche thing, whereas SCALE has the potential to be a true solution to do multiple things

 

[danb35]

But for me personally, once I took a week and figured out some of the ins-n-outs of Dockerfile, and runtime options, it was more than sufficient for all my needs, hands down.

Apparently "run arbitrary software with arbitrary persistent configuration and data" is not among your needs--I haven't found*, and nobody's been able to explain to me, a way to do this with Docker--and especially not a way that's anywhere near as simple as "spin up a jail and install your software there." It's trivial with LXC containers, but SCALE doesn't support those.

I understand that Docker is very popular--if I hadn't been aware at the time, the reaction here to your putting FN10 out of its misery amply demonstrated that. And the Apps implementation in SCALE, even at this point, seems like what plugins always should have been. But Docker doesn't replace jails.

*See also https://www.truenas.com/community/threads/separate-linux-container.102153/ and https://www.truenas.com/community/threads/use-a-custom-jail-in-scale-possible-alternatives.100109,

 

[danb35]

when TrueNAS does finally drop FreeBSD as it's base.

Now when I say things like that, I get yelled at. But yes, Kris' and Morgan's denials notwithstanding, I expect this is coming, and sooner rather than later.

 

[danb35]

SCALE is light years better in Apps and Hardware support, and works great personally, dunno the hate.

The "hate" seems to result from the fact that the one thing it was advertised as doing that would have been unique, it doesn't yet do.

 

[NickF]

The "hate" seems to result from the fact that the one thing it was advertised as doing that would have been unique, it doesn't yet do.

It’s not hate that it doesn’t yet do it. It’s hate that they don’t consider it a beta or a technical preview even though it doesn’t live up to the name.

 

[danb35]

I also don't think it's fair to beat up on Corral over the disk replacement issue.

It's absolutely fair. Disk replacement is core functionality in any storage server, and to knowingly ship a "release" product ("not for production use" was a subsequent retcon; at release time it was claimed to be production-ready) without it is inexcusable. But it's hardly as though that were the only show-stopping bug in FN10.

Apps in Scale really are not that complicated to use.

Agreed; such experience as I have with them indicates they're far superior to plugins.

 

[morganL]

Hi Morgan,

Sorry for my insomnia riddled response. As always, I appreciate your candor and all of the help you provide here on these forums.

Onto the discussion.

I understand that there has to be a version 1.0. That goes for any software product, whether it be open source or not. I get that benchmarks are needed so that you know where to drive development. You need to size your teams appropriately based on those numbers and any corresponding projected income from business users that may be attributed to the ongoing success of the free version.


You identified that the target audience of the release should be nerds like us here on the forums or on Reddit. Folks who are intelligent enough to help test and report bugs and folks who just want to run a single node with some Docker containers and have some storage that's safe and reliable.

What percentage of that target audience, which you define as over 20,000 systems, were expected to use MinIO and what percentage is actively using it? Without clustered Kubernetes support, what is the use case? Same question for Gluster native, without having SMB clustering, what's the point? How many users actually had a use case that couldn't have just been resolved with ZFS replication?

With the release of TrueCommand yesterday, we got SMB clustering, several months after launch. But even that, in it's current form has serious limitations. If I have a dataset I want clustered, but I have another dataset I don't have enough space to cluster on my other systems, I can't leverage the feature. Having to choose between cluster and breaking all of my other existing shares is a really tough pill to swallow. What's worse is that in the video it a warning that flies in from no where and is quickly glossed over.

Same goes with iSCSI, NFS, etc. Why bother having clustered storage if you cannot multipath your I/O between more than one system? You mention KVM, even if external systems can't access the clustered pools, surely we should have been able to create highly available VMs. This functionality has been in Proxmox for years, shouldn't that have been in the release? In the absence of all of these things, what good is a scale-out system if none of the underlying technologies in the platform can utilize it.

It’s like you guys built a really cool car. But you forgot to put the passenger seat in, the backseats are missing, and you don’t have any carpet. You have a great sound system, nice wheels and tires, and even a fairly strong engine. But you can’t enjoy it with anyone else, and you can’t bring your kids to school. You don’t just need to tune the engine and adjust the feel of the suspension. You need to finish the car.

As a follow up coming from a different direction, I'm confused about the goal here. By supporting the single-node model well, you have indeed created a larger user testing base. But what percentage of the testing base you've defined will ever SCALE OUT? Your userbase of dudes like me who have racks in their homes is a fraction of what the larger overall userbase is. The other users will come, because there is a serious desire from alot of folks to be using Docker and Kubernetes as a professional learning platform or a hobby for fun things in their house.

But the actual users you need to impress are not those users, it's the business customers who will buy actual hardware and support contracts from you. It's those users who you are hurting by releasing a half-baked product, and it's those users who pay for the salaries of all of the folks at iXSystems. I am also one of those users, I've priced out buying IX Hardware for some video surveillance projects. The problem is that big name competitors are putting more and more downward pressure on the market. When I go to my director and say I can buy a Lenovo DE system for the same money or less than an IX system, it's a hard sell. Now, with SCALE there are more differentiating features on the horizon, that sell is going to get easier. But I can't in good conscious tell him that I want to make a serious capital investment on a product that is fundamentally incomplete at this point....


I'm not asking you to, really. I'm just asking you not to call incomplete product "release".


That is totally understood, but not really my point. I understand the trouble with competing priorities. I think all of us in IT do, and I am also willing to wait for the pie to finish baking in the oven.


I've already staged my home environment for testing this winter or next spring, whenever you guys get around to releasing Bluefin. I've migrated all of my VMs save 1 off of my production VMWare box and into SCALE. I have a third box sitting in the rack waiting for a couple more pieces of storage. I'm actually very excited to see where this bus brings me.

My final words will be that I still think it’s a marketing problem. If you guys said from the onset that you were porting TRUENAS CORE from BSD to a Debian base, with the goal being solid KVM snd Docker support, we would all be singing the praises. You could have silently worked on the SCALE OUT pieces and announced them after the initial release as being part of version 2.0. But you called it SCALE, and talked about how awesome SCALE OUT is going to be. Then you released a product that doesn’t have any useful scale out features…. The timing and content of the product announcement and the fact that Angelfish 22.02.2 is advertised as a complete end product that is ready for business use is just not okay…

​

​

Thanks again!​

I think most of the conflict comes from the expectation that the day 1 use-case was business users with scale-out requirements. That might be your business requirement and our aspiration, but we always understood that production users would be reluctant to leap from Isilon or Nutanix to an unproven SCALE infrastructure. We are planning for a long game, not a short game.

We deliberately chose a strategy to enable home labs and single node systems to be useful to enable a platform of software quality. You can argue with calling it "TrueNAS SCALE" but in the long term, it seems to be a good name and changing names is difficult. Without that early-adopter market, we'd be very slow to deliver enterprise-quality.

The blog for 22.02.2 was actually very clear about the status of TrueNAS SCALE. Some may have missed this, but we try very hard to be clear on where we are at:

Who Should Use TrueNAS SCALE?​

At this U2 stage of its Software Development Lifecycle, TrueNAS SCALE is primarily for simpler deployments and tech labs. It is particularly well suited for users with Linux Apps and Virtualization requirements, in addition to standard storage needs. Users with scale-out storage requirements can start verifying for their specific use-cases or interests.

and later in the blog:

Production users with standard NAS (NFS, SMB, iSCSI, S3) requirements are still advised to use TrueNAS CORE and Enterprise, which have a hundred times more data under management and over ten years of operation and stability. TrueNAS SCALE has inherited some of that maturity and the automated testing but has not yet completed its enterprise software quality lifecycle.

Since we wrote those lines... the "data under management" for SCALE is now over 10% of that under management for CORE. So we probably overstated the discrepancy in maturity.

We certainly understand that vendors like Lenovo can undercut iXsystems pricing. They have Taiwan cost and volume advantages that we don't. Our focus is on software, support and service while keeping system/appliance prices very reasonable. Lenovo can assist you with ZFS and SMB/NFS issues, but iX can.

This year, the majority of our business is dual controller (HA) systems and a limited % will be scale-out clusters. Next year we expect the scale-out clusters confidence and acceptance to grow significantly. Many production users cannot wait... we understand and accept that, but we do work with users with who want to collaborate for the longer term benefits.

Rome wasn't built in a day.... but perhaps on the 1st day it provided shelter for a few good warriors.

 

[indivision]

It’s not hate that it doesn’t yet do it. It’s hate that they don’t consider it a beta or a technical preview even though it doesn’t live up to the name.

Doesn't this just reduce the point to semantics? "I find the label of the project confusing."

Is that really a genuine obstacle? There are people researching which software to use for clustering and they haven't been able to figure out precisely what Scale does and doesn't do because of the release labels? I don't find this plausible.

It also doesn't really sync with the sheer volume of writing about this (super soaker stories, etc).

It's absolutely fair. Disk replacement is core functionality in any storage server, and to knowingly ship a "release" product ("not for production use" was a subsequent retcon; at release time it was claimed to be production-ready) without it is inexcusable. But it's hardly as though that were the only show-stopping bug in FN10.

It wasn't technically without that capability. I replaced a disk with a few lines in the CLI with Corral.

I don't think your point is unreasonable about how it should be labeled. But, imo, semantic issues with the release label do not warrant characterizing an entire effort/project as a disaster, what-not-to-do, etc. I'm not saying that you have made that characterization necessarily. But, that appears to be the way it has been brought up in this thread.

 

[danb35]

But, imo, semantic issues with the release label do not warrant characterizing an entire effort/project as a disaster, what-not-to-do, etc. I'm not saying that you have made that characterization necessarily. But, that appears to be the way it has been brought up in this thread.

That was iX' characterization (and why they killed it), and there's a reason we treat it like Voldemort around here--FN10 was an unmitigated disaster. SCALE is not FN10 (and is, IMO, looking much more promising), but there's still a lot of promise that has yet to be fulfilled. And what you're trying to downplay as "semantic issues with the release label" are actually major issues--in most of the world, "release" means an assertion by the developer that the software is feature-complete, tested, and stable. iX apparently reserves that assertion for -U2 or later with CORE, and for "something that hasn't happened yet and has no ETA" for SCALE.

 

[Patrick M. Hausen]

On the other hand, in regards to Core, to treat each jail as its own "mini server", contained within itself, and without the overhead of virtualization is truly under-appreciated.

High five, bro!

 

[Patrick M. Hausen]

Doesn't this just reduce the point to semantics? "I find the label of the project confusing."

Naming things and expiring caches - there is a reason these are frequently named the hardest problems in software development.

You create a particular image with some message. And the originator and messenger are the ones whose duty is considering the impact on the target audience and phrasing the message accordingly.

I'll be giving a talk with much TrueNAS content and endorsement tomorrow and I am still thinking about where to put SCALE in the global context.

If you are interested:

Building infrastructure with Open Source - a TrueNAS and OPNsense walkthrough

t3dd22.typo3.com

Kind regards,
Patrick

 

[indivision]

That was iX' characterization (and why they killed it), and there's a reason we treat it like Voldemort around here--FN10 was an unmitigated disaster.

I think it was more complicated than that.

I recall the main reason given by iX was that there was too much of a gulf between the two code-bases, creating a lot of double-work. And Corral was built with a front-end framework that was abandoned after Corral had started and used it for a while.

If Corral was just wildly unpopular there would be no reason to treat it like Voldemort. I think the real reason for trying not to mention it is that a lot of users were disappointed in the move and it stirred up a lot of drama.

Anyway, I'm not wanting to dig up ancient history here. I'm pretty sure that it's water under the bridge by now for everyone (including myself). I don't think it was helpful to bring up in this thread in the first place.

SCALE is not FN10 (and is, IMO, looking much more promising), but there's still a lot of promise that has yet to be fulfilled. And what you're trying to downplay as "semantic issues with the release label" are actually major issues--in most of the world, "release" means an assertion by the developer that the software is feature-complete, tested, and stable. iX apparently reserves that assertion for -U2 or later with CORE, and for "something that hasn't happened yet and has no ETA" for SCALE.

Alpha, Beta, Release, Release Candidate, xyz-it. These are labels. Semantics. What one company or developer defines as a beta (or other stage) varies from others a lot.

It is also very common for there to be releases for public use that are not feature complete. For many projects, how can you even define what "complete" means? Plex, for example, has added many new features over the years. Does that mean that it wasn't complete enough for release 2 years ago?

Bugs can be found in software releases even after testing for stability. Companies that have incredible amounts of resources spent on testing still release products that bugs are found in. "release" simply does not mean "bug free".

For those reasons, yes, I AM downplaying complaints about the labels. iX hasn't poorly labeled or planned Scale (OR Corral) based on the labels and feature-sets they released with.