J
jkh
Guest
Hi folks,
So, last week (on Wednesday morning, March 15th to be precise) the release of FreeNAS Corral happened, a release which has truly been a long time in the making and a greatly anticipated event, by FreeNAS' users and its developers alike. Like all greatly anticipated events in life, however, it either exceeded, met, or fell short of some early adopter's expectations, depending on the complexity of their usage scenario(s) and what they were, in fact, actually expecting from the first release.
For a small minority, FreeNAS Corral's release was also a non-event because those users were already running FreeNAS 10, had been for many months, and "release day" was just another day for them, albeit one with a new logo and brand in their browser windows. Among the more conservative FreeNAS users, release day was also a non-event simply because those users are still hanging back and watching to see how others fare with a "dot-zero release" of this entirely new and obviously ambitious technology offering.
Whichever of these camps you fell into, you're probably now all wondering the same thing: "How have things gone so far, now that we're 5 days into this post-RELEASE process?"
Let's just say it's been an interesting week, and one in which a surplus of both discussion and tickets have been available. Some folks love it, some folks hate it, and many folks are simply on the fence. Some folks haven't even bothered to look at it but already have strong opinions to express, the flood of discussion being so great that it even crashed the FreeNAS Forums for awhile on Thursday. Given such a wide range of opinions, it's also been really hard for early adopters to know the "mild hysteria" from the genuine, unbiased reporting about Corral, so I'd like to just take this opportunity to wade in and attempt to at least frame some of those discussions in a larger context.
I do have a lot of things to cover so I make no promises about how short this will be. You may wish to make yourself a cup of tea!
I will also break this into 6 sections, just to make it easier to jump to the section you're specifically interested if it's all TL;DR for you and you'd like to skip ahead, though I also hope you won't because I'm going to hide lots of random tidbits of information here and there, just to make it worth reading all the way through. You're welcome! ;)
Sections:
1. Why this week was totally normal.
2. The release schedule and overall release quality.
3. Software updates and FreeNAS-Corral-STABLE.
4. The problem with PC hardware (unless you buy it from iXsystems :D).
5. Documentation, where is all of it?
6. The Open source / Community development process going forward.
1. The first, and probably the most important thing, I want to say is that all of this reaction (both positive and negative) is all Totally Normal. Yes, even given the deluge of commentary and occasional bouts of freaking out about how much has changed or how it now behaves, this is a normal and expected state of affairs. To perhaps point out the obvious again: FreeNAS Corral is a major new software release. It's a rewrite with the accompanying rare opportunity to do a bunch of things in more modern ways, use some fundamentally new technologies and, where practical, even jettison some of the old approaches to solving certain problems where newer and better ("better" being admittedly in the eye of the beholder) alternatives exist. That latter category is perhaps the most controversial but also a necessary activity if a maintainable code base is any sort of priority, which of course it must be (and even then, it's often negotiable - please keep reading).
This is the natural progression of human technology, both software and otherwise, and if you look around you, you'll see plenty of examples of things that changed in ways that made you first go "WAAAAAAHH! I LIKED THAT HOW IT WAS!" or even "I'LL NEVER BE ABLE TO USE THAT!" but then maybe 6-12 months later, you had a hard time even remembering how things used to be, so yeah, right now FreeNAS Corral looks new and scary and different and not "comfortable" to some of you and that's OK. Corral is only 5 days old, and almost everyone's definition of FreeNAS is still "9.10.2" (or even 9.3) given that it's what everyone currently knows and loves. Major changes like this, and the subsequent maturation of such changes, takes time and that brings me to my second bullet:
2. Corral Release schedule and overall quality. Some folks have been commenting that FreeNAS Corral looks somewhat "unpolished" to them (perhaps a politically correct way of putting it), it also being occasionally asked in the forums whether or not FreeNAS Corral was truly "ready" or if was it "rushed" to make some arbitrary deadline. "Why didn't everyone involved wait for another X <weeks> / <months> / <years> before releasing it?" some have even asked.
Let's talk a little bit about that because, believe it or not, there actually was a really long release process involved here and let me start by calling your attention to this link. That was the announcement for the first-ever ALPHA release of FreeNAS 10, the predecessor to FreeNAS Corral, and it happened in October of 2015. Needless to say, the development team also didn't just start working on that ALPHA a week before the announcement, so I think it's accurate to say that the Corral/10 project has been going for well over 2 years at the time of this writing.
Two years is a long time in the software business, and along that 2 year timeline, the dev team has also done an ALPHA, an ALPHA2, a BETA, a BETA2 and a Release Candidate. That's a lot of "pre-releases" for any piece of software, even with the little GUI set-back it had between ALPHA and ALPHA2, and a factor to be very wary of is something called "testing fatigue" setting in, something which can turn the whole release engineering exercise into a series of rapidly diminishing returns after the release engineering "funnel" narrows past a certain point. It also bears noting that software engineers are a bit like artists or musicians; their work needs an audience or their work becomes increasingly abstract and without focus, yet another reason why just constantly pushing out a release schedule isn't necessarily going to make it "better", though it will for sure make it "later" as everyone involved feels a decreasing sense of urgency and purpose.
Those were and have been real risks, so instead we kept up the heat, and in the run up to RELEASE, the engineering team fixed literally over a thousand bugs (in just in that milestone, I'm not even counting previous ones) and focused their very best efforts on ensure that anything "blocking" the release was fixed before it went out the door. For what it's worth, the update server was also tracking several thousand unique FreeNAS 10 machines successfully installed and checking in before the release, so while the number of pre-release testers and developers wasn't in the millions (and never has been) it wasn't exactly zero, either.
Now, of course, even given all of the above one could still argue for "more time" in any release schedule. "How much time?" asks the project manager, somewhat plaintively. "Who knows! Whenever it's ready, then ship it!" say the users and the engineers both, since nobody likes being rushed into anything. "When it's ready" also sounds like a good slogan, something suitable for a bumper sticker or a project tee-shirt, even though without hard data in the form of schedules and trouble ticket counts, the term "ready" is also essentially so subjective as to be meaningless.
How subjective? How meaningless? Well, let's talk about a little video game series called "Duke Nuke 'em" for a moment. The first game shipped in 1991, and I remember it well because I and many other folks enjoyed playing it and its sequel (somewhat un-creatively called "Duke Nuke 'em II") in 1993, then we had a full 3 year wait for the next sequel ("Duke Nuke 'em 3D") in 1996, at which point the developers promised to top all their previous efforts with another sequel called "Duke Nuke 'em Forever", a game which was promised to be awesome but also had a ship date of "when it's ready." Well, I'm sure the developers were not solely to blame for what happened, or rather what did not happen, next, but suffice it to say that the next game in the series didn't ship until fifteen years later, by which point most of the fans of the original game were either deceased or just didn't care anymore, having long since hung the snarky title of "Duke Nuke 'Em Whenever" on the promised game and forgotten about it.
The development team really didn't want to ship a FreeNAS Corral "whenever" edition, we wanted to ship it while we still had some momentum in our development effort and FreeNAS Corral still had the opportunity to be relevant in the storage / application hosting community. We also wanted and needed the time and space to create the requisite software updates which would turn Corral into the same kind of highly polished end-result of evolution that previous major FreeNAS releases have been, all of which were far from perfect when they were first released. It really cannot be stressed enough that for each and every major version in FreeNAS' history, whether it was 8.0 or 9.3.0 or 9.10.0, there was always a "first release" followed by a series of updates, often very rapid updates, all of which were created in direct reaction to the kind of bug reports that only arrived as the result of a RELEASE and subsequent frantic, focused polishing efforts. That's just how the human tendency to procrastinate and software engineering tend to intersect, and it can be a little messy until things naturally sort themselves out.
Some folks also have short memories and tendency to forget those early releases when they talk about FreeNAS Corral today, just as they have forgotten FreeBSD releases like "2.0" or "the very first iPhone", but I was there working on all of those products and I certainly remember them very well, even if I would have a hard time looking at either one of them today. If you set FreeBSD 2.1.5 or even FreeBSD 2.0.5 against FreeBSD 2.0, or the very first iPhone against an iPhone 7, you would simply have a really hard time believing they were products of the same essential organizations. What saved them all was this simple equation: Time to market + Urgency = Success.
We are, in any case, now in a post-RELEASE world and there's just no point in playing "what if" with the release date, or speculating on what would have happened (or not) if we had waited another year or two or fifteen, just like Duke Nuke 'Em Forever. We're there now, we've branched for FreeNAS-Corral-STABLE (and have kicked off the FreeNAS-Corral-Nightlies again) for the sake of rapid feature and stability convergence, and it's now time to focus on reporting and fixing the most significant issues in a series of updates. Which brings us to my 3rd bullet:
3. Updates and Improvements. Yes, of course there will be updates to FreeNAS Corral; having them was obviously always part of the plan. By the time you read this message, in fact, there will already be an update available for FreeNAS Corral labelled "10.0.1", one which fixes over 30 of the most significant problems users have reported in the last 5 days, both with FreeNAS Corral itself and with the upgrade process from 9.10 to Corral.
We went out of our way, and put considerable effort, into making sure that FreeNAS Corral was itself is an update which could be applied to FreeNAS 9.10, rolled back if the user didn't like it, installed again if they thought there was a better update available later, moved back and forth amongst its own update version, etc.
A lot of commercial products don't even offer capabilities like that (upgrades are a purely one-way street), so it was a worthwhile goal nonetheless, even if life would have been much simpler for us in engineering if we had simply demanded that all 9.x users did from-scratch installs of FreeNAS Corral instead of upgrading to it. The total dissimilarity between the database formats and the way that 9 and Corral do so many things differently made this really, genuinely hard, but the more test cases we've gotten, the better this process has also become.
Users are also certainly free to do fresh installs if they don't trust our migrator. The FreeNAS installer has, for a long time, offered an option to "install into a new boot environment" rather than simply formatting the boot drive, so we really do offer a lot of options here even though our initial shot at the auto-migrator was somewhat fragile and didn't like either encrypted volumes or LAGGs as much as we hoped it would (yes, we successfully upgraded all of our 9.10 boxes to 10 before we released, including the ones with encrypted volumes and LAGGs, but clearly we missed some things).
We also knew the influx of new users would show us anything and everything we missed, and we were not wrong. The sheer diversity of hardware out there in the FreeNAS community is simply staggering and we, needless to say, do not have access to more than a fraction of it in our lab. What works on our machines, and we always test on them before any release (be it BETA, RC or RELEASE), does not always work for everyone else.
While we're talking about updates, let's also talk about "best practices" for doing updates, just in case some folks are still scared of or by the process:
Many people now take the precaution of backing up their 9 databases before attempting an upgrade, something which has always been a good idea and a policy we have always recommended going all the way back, even during "minor release upgrades" because, hey, who knows about these things and it's cheap insurance to have what is essentially a backup to your backup (Boot Environment).
That way, should even if the worst should happen and your USB boot drive gives up the ghost during the upgrade itself because it didn't like the sudden surge of write requests, fine, just grab another USB boot device and do a fresh install on it, then restore your old configuration database. Both FreeNAS 9 and FreeNAS Corral store all of their relevant configuration information in a single database file which can be easily downloaded and uploaded to a new install, restoring the old configuration, though that said they are not database compatible (you cannot load a 9 database into Corral, or vice-versa; the 9-to-Corral migrator is a more complex process). In no case does the upgrader or installer touch the data pool, so even "erase installs" are perfectly safe for the data itself. It's just not involved until everything is fully back up again and the user wants to explicitly import it.
4. Hardware. Since we're talking about USB drives now, let's talk about something else which has come up frequently this week, and that's Hardware! The wonderful wonderful (not) world of PC hardware and why it always seems to pick upgrade time as the Best Possible Time To Commit Seppuku (or otherwise prove itself unworthy of whatever new software you want to install on it, be it FreeNAS Corral or the latest version of Windows).
I've seen a few postings in this forum about how "FreeNAS Corral killed my thumb drive!" or how someone's PC used to work marvelously with 9.10 but then they installed Corral on it and the power supply started making a loud buzzing sound, the keyboard would only type Cyrillic characters even when it wasn't actually plugged into anything, and the software claimed that Virtual Machines and Docker Containers were not supported even though the guy behind the counter at Joe's discount PC and Tata Indica repair shop swore that this model of CPU supported both VT-x and EPT instructions and to ignore the silk screen clearly showing "AMD Turion with 3DNow!" on the chip. The user, of course, blames Corral because everything was JUST FINE before the upgrade!
Let's be honest here, folks. Not only is the fact that many "off the shelf" (or occasionally "from the back of the closet") PCs even work at all a miracle of modern technology, the process involved with selecting various motherboards, CPU chipsets, RAM, disk controllers and storage, and then sticking them into the same box, is a little bit like playing russian roulette and not just once, but every single day. Will the system keep working with just one power supply? Did some of that RAM I got a great deal on just go bad? Things are acting a little oddly - what version of firmware should my LSI disk controller be running for best results, given the driver in the specific OS version underlying FreeNAS?
This is exactly why Enterprise grade hardware costs more, not just because the hardware components are almost always of better individual quality but because it's all also carefully hand-selected by folks who do it all day long as a profession. The professionals have a very strict list of hardware and higher operating parameters driving their selection process: This controller, that firmware revision, those drives, that pair of redundant power supplies for when one fails, these SATA DOMs to mirror and not some random USB sticks for a boot drive, etc etc. Those are some of the key details one must know about, which is a big part of what allows iXsystems to sell higher-end hardware and, ultimately, fund a great deal of the FreeNAS development I have been describing in this message.
For everyone else on random non-enterprise PC hardware, well, your best resources are always going to be one another in these Forums and a healthy respect for the sheer number of things that can go wrong with your stuff. Don't always blame the software! Sometimes it's just bad timing, bad luck, or changing too many variables at once.
Speaking of software, Corral is also, at the end of the day, just another software appliance based on FreeBSD 11, and a lot of the things FreeBSD "believes" about your hardware or feels capable of doing with it is a direct function of what is in the FreeBSD kernel, a bit of very low-level code which we literally have no choice but to take the word of on many things. We've already seen at least one ticket filed with the upstream FreeBSD project, for example, where the user, and the Intel ARK for that matter, swore up and down that their processor supported EPT but FreeBSD's
Other best-practice things to check: Is your boot pool 80% full before you even start the upgrade? The migrator code which tries to write out new database entries in a new boot environment probably won't be happy about that. Make some space first by cleaning up some old BEs before you even start - thinks will go happier.
Are you sure that memory is good? When did you last run a memtest overnight? We had one "failed install" where the guy was getting "websocket connection interrupted" and a lot of very suspicious looking core dumps. One of our senior engineers got a feeling after looking at the oddness of the crashes and asked if user had run a memory test recently. This system reportedly ran quite well with 9.10, but after testing it was indeed found to have bad memory! Once the bad memory was replaced, the Corral install was perfectly happy. We've had other installs fail, or at least appear to fail, because the user had 20,000 snapshots on their ZFS pool before they started the upgrade and, of course, the upgrader had to manually import all 20,000 snapshots into the new ZFS cache and this took such a really long time that the user thought the migrator had crashed, but it hadn't. Oh yeah, that machine was mine, and I do wish I'd cleaned up some old snapshots before doing the upgrade, but oh well, at least it was a good test!
5. Documentation
Speaking of esoteric topics like ZFS caches, you've also probably been wondering and/or complaining about the relatively minimal state of the FreeNAS Corral Documentation this week. What happened to the user guide? Isn't there a FAQ?? ("Yes"). This new Corral thing is complicated and it really needs a frickin' User Manual, where is it? ("click that link, it's a work in progress").
The short answer is that it's coming along, but how fast it comes along is going to depend a lot more on user contributions this time around. There are simply far more new user scenarios to cover for Corral, and our Docs team is very small. The Docs team also had the comparative luxury, back when it worked (and still works) on the 9 documentation, of having a product that was relatively feature-frozen for some time and just working on bug fixes and stability. The behavior of everything in 9 was well understood, and there was an existing FreeNAS wiki of guides and whatnot to borrow liberally from, so they did and were able to write the user guide without killing themselves in the process.
In Corral, we have decided to take more of a "howto" approach simply because there's too much to cover in a comprehensive guide; such a guide would take at least a year, during which we'd still be without a guide and have very little else besides. Instead, we have created the Documentation wiki at wiki.freenas.org and have a liberal policy of granting accounts to pretty much anyone who demonstrates any skill at writing documentation and requests an account. Desired account name, email address to send password to, and boom - you're a docs editor!
What I also hope, and have used this technique a lot myself, is that the increasing number of question on the Forum will also provide "highly directed fodder" for documentation, e.g. you see someone ask a fairly general but important question, and you may not know the whole answer to that question but you do know who to ask, so you do. Then you take their answer and try it out to see if it actually works, perhaps improve on it a little in the process, then add it to the wiki. Done! You have just learned something and documented something as a "cookbook howto" at the same time. It's a good trade, and if enough people do it, soon the users will be pretty well covered on almost all basic operations.
Having more "pop out" help in the GUI itself, in a variety of different languages, has also always been a goal but we were also forced to pick our priorities: Should we make the existing GUI perform all its functions and have at least some number of "seat belts" to prevent users from going off a cliff with it (and there are still dozens more of those to be added) even though it would be all in english and without help text? Or should er try to provide help text and maybe a major translation but wind up with some sections of the UI that are not even implemented, because we ran out of time and GUI resources in trying to do both? We decided for the former trade-off for RELEASE time, and now it's time to emphasize the latter, that's all.
There is also CLI docs hidden around here and there, for those who like CLI docs. There's "help" and "?" in the CLI itself, of course, there's the getting started guide on github (also due to be merged into the online docs but this hasn't happened yet) and there's the built-in http://yourip/cli doc browser, which definitely needs some volunteers to help flesh it out. There are also numerous examples here and also installed on every FreeNAS Corral box (hint:
For myself, I will also continue to record howto videos on YouTube and post what other occasionally helpful (I hope) bits of information I can here, but so far I have to say that the very best help I've seen has come from you, the users. You've taken the fact that something was not entirely complete and instead of getting discouraged by that, you've let it motivate you into filling in some of the missing pieces. That's awesome!
Which brings me to my final point, and thanks for reading so far since I know this has been long.
6. Community & "Open Source platform"
One of the biggest personal goals I've had for FreeNAS Corral as its project leader was to finally make Corral a FreeNAS platform that others could easily contribute to. Not just where hacking on the code for the middleware or the GUI is concerned, though it's always totally great if someone dives in that far but that's not very easy. I mean easy for slightly less daunting tasks like contributing curated Docker containers, making custom VM templates, contributing to the "live docs" and howto guides, and otherwise leveraging the fact that we use github and dockerhub so much more aggressively as part of our infrastructure now that anyone can now sign up for an account on these various services and start essentially creating their own customizations to FreeNAS.
Creating "Plugins" for 9 was and is always distressingly hard, by contrast, and very few people have ever acquired the knack for doing so, but creating a Docker container for 10 is very easy by comparison and many have already created and contributed some. Probably at least a third of the entries in the FreeNAS Corral curated collection were contributed, and the infrastructures provided by github and dockerhub made it much easier to document the process as well.
Since I'm speaking on the topic of what users want and might contribute, I'd like to just say a few last words about jails. Many people do not know that in the very earliest days of FreeNAS 10, the plan absolutely was to have jail support. The problem was, we also wanted a "jail manager" to manage the jail metadata since almost no one directly manipulates the jails anymore, they use some front-end tool like warden, ezjail or iocage, and our first impulse of course was to use the warden, since it was written by one of our very own iX employees and was already being used in FreeNAS 9, but at that point the warden was also pretty old and almost entirely unmaintained, so the author asked us to wait instead for a new thing that would be taking over for the warden and be much better in every way. Well, we waited, but then the new thing went into a rewrite process of its own and we said "wow, how ironic, the rewrite is now waiting for the rewrite" so, since we figured this was still going to have to be ultimately compatible with 9 anyway, we'd just wait to see how the rewrite all panned out in 9 and meanwhile we could focus exclusively on Docker as our "plugin replacement strategy" since 9 plugins were never an option, jails or no jails, given the previous comments about how hard to create they were and the fact that 100,000+ Docker containers beat 30 plugins any day.
So, now Corral is out and we still have people asking about jails since they like them simply for the purpose of compiling their own software in them and don't want to run a VM. While we have come to resist the idea of jails somewhat simply because it sure seems like Docker containers are providing all of the same functionality with far more software availability (everything from Oracle databases to Swift language development environments) and far less overhead that some people seem to think, we're still also open to the idea. The day that the new replacement for the warden eventually makes it into FreeNAS 9.10.3 (we think), we can simply copy over the relevant bits, slap a coat of paint on them in the form of middleware and CLI support, and boom, there's the old "very low overhead" thing back but compatible with FreeNAS 9's future direction (9 users have already probably noticed a lot of their jails related tickets being pushed forward with references to "a jails rewrite" - that's this) and everyone is happy again. That's what I meant when I said earlier that everything was open to negotiation, even though many things have changed.
More change is coming, and I'm very proud of what we've all accomplished here so far, bugs and all. Now alI we ask for is the community's help in getting Corral polished to a high gloss, so everyone can run it! Please keep those tickets coming, and please keep installing and testing it so that you can file tickets. Every ticket you find and we fix (or some other volunteer does, why not?) will make the end-result just that much better!
- Jordan
for the FreeNAS Dev Team
So, last week (on Wednesday morning, March 15th to be precise) the release of FreeNAS Corral happened, a release which has truly been a long time in the making and a greatly anticipated event, by FreeNAS' users and its developers alike. Like all greatly anticipated events in life, however, it either exceeded, met, or fell short of some early adopter's expectations, depending on the complexity of their usage scenario(s) and what they were, in fact, actually expecting from the first release.
For a small minority, FreeNAS Corral's release was also a non-event because those users were already running FreeNAS 10, had been for many months, and "release day" was just another day for them, albeit one with a new logo and brand in their browser windows. Among the more conservative FreeNAS users, release day was also a non-event simply because those users are still hanging back and watching to see how others fare with a "dot-zero release" of this entirely new and obviously ambitious technology offering.
Whichever of these camps you fell into, you're probably now all wondering the same thing: "How have things gone so far, now that we're 5 days into this post-RELEASE process?"
Let's just say it's been an interesting week, and one in which a surplus of both discussion and tickets have been available. Some folks love it, some folks hate it, and many folks are simply on the fence. Some folks haven't even bothered to look at it but already have strong opinions to express, the flood of discussion being so great that it even crashed the FreeNAS Forums for awhile on Thursday. Given such a wide range of opinions, it's also been really hard for early adopters to know the "mild hysteria" from the genuine, unbiased reporting about Corral, so I'd like to just take this opportunity to wade in and attempt to at least frame some of those discussions in a larger context.
I do have a lot of things to cover so I make no promises about how short this will be. You may wish to make yourself a cup of tea!
I will also break this into 6 sections, just to make it easier to jump to the section you're specifically interested if it's all TL;DR for you and you'd like to skip ahead, though I also hope you won't because I'm going to hide lots of random tidbits of information here and there, just to make it worth reading all the way through. You're welcome! ;)
Sections:
1. Why this week was totally normal.
2. The release schedule and overall release quality.
3. Software updates and FreeNAS-Corral-STABLE.
4. The problem with PC hardware (unless you buy it from iXsystems :D).
5. Documentation, where is all of it?
6. The Open source / Community development process going forward.
1. The first, and probably the most important thing, I want to say is that all of this reaction (both positive and negative) is all Totally Normal. Yes, even given the deluge of commentary and occasional bouts of freaking out about how much has changed or how it now behaves, this is a normal and expected state of affairs. To perhaps point out the obvious again: FreeNAS Corral is a major new software release. It's a rewrite with the accompanying rare opportunity to do a bunch of things in more modern ways, use some fundamentally new technologies and, where practical, even jettison some of the old approaches to solving certain problems where newer and better ("better" being admittedly in the eye of the beholder) alternatives exist. That latter category is perhaps the most controversial but also a necessary activity if a maintainable code base is any sort of priority, which of course it must be (and even then, it's often negotiable - please keep reading).
This is the natural progression of human technology, both software and otherwise, and if you look around you, you'll see plenty of examples of things that changed in ways that made you first go "WAAAAAAHH! I LIKED THAT HOW IT WAS!" or even "I'LL NEVER BE ABLE TO USE THAT!" but then maybe 6-12 months later, you had a hard time even remembering how things used to be, so yeah, right now FreeNAS Corral looks new and scary and different and not "comfortable" to some of you and that's OK. Corral is only 5 days old, and almost everyone's definition of FreeNAS is still "9.10.2" (or even 9.3) given that it's what everyone currently knows and loves. Major changes like this, and the subsequent maturation of such changes, takes time and that brings me to my second bullet:
2. Corral Release schedule and overall quality. Some folks have been commenting that FreeNAS Corral looks somewhat "unpolished" to them (perhaps a politically correct way of putting it), it also being occasionally asked in the forums whether or not FreeNAS Corral was truly "ready" or if was it "rushed" to make some arbitrary deadline. "Why didn't everyone involved wait for another X <weeks> / <months> / <years> before releasing it?" some have even asked.
Let's talk a little bit about that because, believe it or not, there actually was a really long release process involved here and let me start by calling your attention to this link. That was the announcement for the first-ever ALPHA release of FreeNAS 10, the predecessor to FreeNAS Corral, and it happened in October of 2015. Needless to say, the development team also didn't just start working on that ALPHA a week before the announcement, so I think it's accurate to say that the Corral/10 project has been going for well over 2 years at the time of this writing.
Two years is a long time in the software business, and along that 2 year timeline, the dev team has also done an ALPHA, an ALPHA2, a BETA, a BETA2 and a Release Candidate. That's a lot of "pre-releases" for any piece of software, even with the little GUI set-back it had between ALPHA and ALPHA2, and a factor to be very wary of is something called "testing fatigue" setting in, something which can turn the whole release engineering exercise into a series of rapidly diminishing returns after the release engineering "funnel" narrows past a certain point. It also bears noting that software engineers are a bit like artists or musicians; their work needs an audience or their work becomes increasingly abstract and without focus, yet another reason why just constantly pushing out a release schedule isn't necessarily going to make it "better", though it will for sure make it "later" as everyone involved feels a decreasing sense of urgency and purpose.
Those were and have been real risks, so instead we kept up the heat, and in the run up to RELEASE, the engineering team fixed literally over a thousand bugs (in just in that milestone, I'm not even counting previous ones) and focused their very best efforts on ensure that anything "blocking" the release was fixed before it went out the door. For what it's worth, the update server was also tracking several thousand unique FreeNAS 10 machines successfully installed and checking in before the release, so while the number of pre-release testers and developers wasn't in the millions (and never has been) it wasn't exactly zero, either.
Now, of course, even given all of the above one could still argue for "more time" in any release schedule. "How much time?" asks the project manager, somewhat plaintively. "Who knows! Whenever it's ready, then ship it!" say the users and the engineers both, since nobody likes being rushed into anything. "When it's ready" also sounds like a good slogan, something suitable for a bumper sticker or a project tee-shirt, even though without hard data in the form of schedules and trouble ticket counts, the term "ready" is also essentially so subjective as to be meaningless.
How subjective? How meaningless? Well, let's talk about a little video game series called "Duke Nuke 'em" for a moment. The first game shipped in 1991, and I remember it well because I and many other folks enjoyed playing it and its sequel (somewhat un-creatively called "Duke Nuke 'em II") in 1993, then we had a full 3 year wait for the next sequel ("Duke Nuke 'em 3D") in 1996, at which point the developers promised to top all their previous efforts with another sequel called "Duke Nuke 'em Forever", a game which was promised to be awesome but also had a ship date of "when it's ready." Well, I'm sure the developers were not solely to blame for what happened, or rather what did not happen, next, but suffice it to say that the next game in the series didn't ship until fifteen years later, by which point most of the fans of the original game were either deceased or just didn't care anymore, having long since hung the snarky title of "Duke Nuke 'Em Whenever" on the promised game and forgotten about it.
The development team really didn't want to ship a FreeNAS Corral "whenever" edition, we wanted to ship it while we still had some momentum in our development effort and FreeNAS Corral still had the opportunity to be relevant in the storage / application hosting community. We also wanted and needed the time and space to create the requisite software updates which would turn Corral into the same kind of highly polished end-result of evolution that previous major FreeNAS releases have been, all of which were far from perfect when they were first released. It really cannot be stressed enough that for each and every major version in FreeNAS' history, whether it was 8.0 or 9.3.0 or 9.10.0, there was always a "first release" followed by a series of updates, often very rapid updates, all of which were created in direct reaction to the kind of bug reports that only arrived as the result of a RELEASE and subsequent frantic, focused polishing efforts. That's just how the human tendency to procrastinate and software engineering tend to intersect, and it can be a little messy until things naturally sort themselves out.
Some folks also have short memories and tendency to forget those early releases when they talk about FreeNAS Corral today, just as they have forgotten FreeBSD releases like "2.0" or "the very first iPhone", but I was there working on all of those products and I certainly remember them very well, even if I would have a hard time looking at either one of them today. If you set FreeBSD 2.1.5 or even FreeBSD 2.0.5 against FreeBSD 2.0, or the very first iPhone against an iPhone 7, you would simply have a really hard time believing they were products of the same essential organizations. What saved them all was this simple equation: Time to market + Urgency = Success.
We are, in any case, now in a post-RELEASE world and there's just no point in playing "what if" with the release date, or speculating on what would have happened (or not) if we had waited another year or two or fifteen, just like Duke Nuke 'Em Forever. We're there now, we've branched for FreeNAS-Corral-STABLE (and have kicked off the FreeNAS-Corral-Nightlies again) for the sake of rapid feature and stability convergence, and it's now time to focus on reporting and fixing the most significant issues in a series of updates. Which brings us to my 3rd bullet:
3. Updates and Improvements. Yes, of course there will be updates to FreeNAS Corral; having them was obviously always part of the plan. By the time you read this message, in fact, there will already be an update available for FreeNAS Corral labelled "10.0.1", one which fixes over 30 of the most significant problems users have reported in the last 5 days, both with FreeNAS Corral itself and with the upgrade process from 9.10 to Corral.
We went out of our way, and put considerable effort, into making sure that FreeNAS Corral was itself is an update which could be applied to FreeNAS 9.10, rolled back if the user didn't like it, installed again if they thought there was a better update available later, moved back and forth amongst its own update version, etc.
A lot of commercial products don't even offer capabilities like that (upgrades are a purely one-way street), so it was a worthwhile goal nonetheless, even if life would have been much simpler for us in engineering if we had simply demanded that all 9.x users did from-scratch installs of FreeNAS Corral instead of upgrading to it. The total dissimilarity between the database formats and the way that 9 and Corral do so many things differently made this really, genuinely hard, but the more test cases we've gotten, the better this process has also become.
Users are also certainly free to do fresh installs if they don't trust our migrator. The FreeNAS installer has, for a long time, offered an option to "install into a new boot environment" rather than simply formatting the boot drive, so we really do offer a lot of options here even though our initial shot at the auto-migrator was somewhat fragile and didn't like either encrypted volumes or LAGGs as much as we hoped it would (yes, we successfully upgraded all of our 9.10 boxes to 10 before we released, including the ones with encrypted volumes and LAGGs, but clearly we missed some things).
We also knew the influx of new users would show us anything and everything we missed, and we were not wrong. The sheer diversity of hardware out there in the FreeNAS community is simply staggering and we, needless to say, do not have access to more than a fraction of it in our lab. What works on our machines, and we always test on them before any release (be it BETA, RC or RELEASE), does not always work for everyone else.
While we're talking about updates, let's also talk about "best practices" for doing updates, just in case some folks are still scared of or by the process:
Many people now take the precaution of backing up their 9 databases before attempting an upgrade, something which has always been a good idea and a policy we have always recommended going all the way back, even during "minor release upgrades" because, hey, who knows about these things and it's cheap insurance to have what is essentially a backup to your backup (Boot Environment).
That way, should even if the worst should happen and your USB boot drive gives up the ghost during the upgrade itself because it didn't like the sudden surge of write requests, fine, just grab another USB boot device and do a fresh install on it, then restore your old configuration database. Both FreeNAS 9 and FreeNAS Corral store all of their relevant configuration information in a single database file which can be easily downloaded and uploaded to a new install, restoring the old configuration, though that said they are not database compatible (you cannot load a 9 database into Corral, or vice-versa; the 9-to-Corral migrator is a more complex process). In no case does the upgrader or installer touch the data pool, so even "erase installs" are perfectly safe for the data itself. It's just not involved until everything is fully back up again and the user wants to explicitly import it.
4. Hardware. Since we're talking about USB drives now, let's talk about something else which has come up frequently this week, and that's Hardware! The wonderful wonderful (not) world of PC hardware and why it always seems to pick upgrade time as the Best Possible Time To Commit Seppuku (or otherwise prove itself unworthy of whatever new software you want to install on it, be it FreeNAS Corral or the latest version of Windows).
I've seen a few postings in this forum about how "FreeNAS Corral killed my thumb drive!" or how someone's PC used to work marvelously with 9.10 but then they installed Corral on it and the power supply started making a loud buzzing sound, the keyboard would only type Cyrillic characters even when it wasn't actually plugged into anything, and the software claimed that Virtual Machines and Docker Containers were not supported even though the guy behind the counter at Joe's discount PC and Tata Indica repair shop swore that this model of CPU supported both VT-x and EPT instructions and to ignore the silk screen clearly showing "AMD Turion with 3DNow!" on the chip. The user, of course, blames Corral because everything was JUST FINE before the upgrade!
Let's be honest here, folks. Not only is the fact that many "off the shelf" (or occasionally "from the back of the closet") PCs even work at all a miracle of modern technology, the process involved with selecting various motherboards, CPU chipsets, RAM, disk controllers and storage, and then sticking them into the same box, is a little bit like playing russian roulette and not just once, but every single day. Will the system keep working with just one power supply? Did some of that RAM I got a great deal on just go bad? Things are acting a little oddly - what version of firmware should my LSI disk controller be running for best results, given the driver in the specific OS version underlying FreeNAS?
This is exactly why Enterprise grade hardware costs more, not just because the hardware components are almost always of better individual quality but because it's all also carefully hand-selected by folks who do it all day long as a profession. The professionals have a very strict list of hardware and higher operating parameters driving their selection process: This controller, that firmware revision, those drives, that pair of redundant power supplies for when one fails, these SATA DOMs to mirror and not some random USB sticks for a boot drive, etc etc. Those are some of the key details one must know about, which is a big part of what allows iXsystems to sell higher-end hardware and, ultimately, fund a great deal of the FreeNAS development I have been describing in this message.
For everyone else on random non-enterprise PC hardware, well, your best resources are always going to be one another in these Forums and a healthy respect for the sheer number of things that can go wrong with your stuff. Don't always blame the software! Sometimes it's just bad timing, bad luck, or changing too many variables at once.
Speaking of software, Corral is also, at the end of the day, just another software appliance based on FreeBSD 11, and a lot of the things FreeBSD "believes" about your hardware or feels capable of doing with it is a direct function of what is in the FreeBSD kernel, a bit of very low-level code which we literally have no choice but to take the word of on many things. We've already seen at least one ticket filed with the upstream FreeBSD project, for example, where the user, and the Intel ARK for that matter, swore up and down that their processor supported EPT but FreeBSD's
hw.vmm.vmx.cap.unrestricted_guest sysctl(2) came back with "0" instead of "1" and in such cases, there is nothing we can do but assume that FreeBSD is telling the truth, because the underlying bhyve(1) command for running VMs is going to look at the very same system property and say "NO! I REFUSE!" when it comes to running a UEFI guest OS. We just detect that flag earlier and get blamed for having the seat-belt that will prevent the far odder and harder to diagnose behavior of bhyve later falling over ungracefully. This is when having some skill with filing tickets against other open source projects, not just FreeNAS, can come in handy because FreeNAS is an amalgamation of many open source technologies; we do not "own" it all ourselves or even pretend to be experts on all of it.Other best-practice things to check: Is your boot pool 80% full before you even start the upgrade? The migrator code which tries to write out new database entries in a new boot environment probably won't be happy about that. Make some space first by cleaning up some old BEs before you even start - thinks will go happier.
Are you sure that memory is good? When did you last run a memtest overnight? We had one "failed install" where the guy was getting "websocket connection interrupted" and a lot of very suspicious looking core dumps. One of our senior engineers got a feeling after looking at the oddness of the crashes and asked if user had run a memory test recently. This system reportedly ran quite well with 9.10, but after testing it was indeed found to have bad memory! Once the bad memory was replaced, the Corral install was perfectly happy. We've had other installs fail, or at least appear to fail, because the user had 20,000 snapshots on their ZFS pool before they started the upgrade and, of course, the upgrader had to manually import all 20,000 snapshots into the new ZFS cache and this took such a really long time that the user thought the migrator had crashed, but it hadn't. Oh yeah, that machine was mine, and I do wish I'd cleaned up some old snapshots before doing the upgrade, but oh well, at least it was a good test!
5. Documentation
Speaking of esoteric topics like ZFS caches, you've also probably been wondering and/or complaining about the relatively minimal state of the FreeNAS Corral Documentation this week. What happened to the user guide? Isn't there a FAQ?? ("Yes"). This new Corral thing is complicated and it really needs a frickin' User Manual, where is it? ("click that link, it's a work in progress").
The short answer is that it's coming along, but how fast it comes along is going to depend a lot more on user contributions this time around. There are simply far more new user scenarios to cover for Corral, and our Docs team is very small. The Docs team also had the comparative luxury, back when it worked (and still works) on the 9 documentation, of having a product that was relatively feature-frozen for some time and just working on bug fixes and stability. The behavior of everything in 9 was well understood, and there was an existing FreeNAS wiki of guides and whatnot to borrow liberally from, so they did and were able to write the user guide without killing themselves in the process.
In Corral, we have decided to take more of a "howto" approach simply because there's too much to cover in a comprehensive guide; such a guide would take at least a year, during which we'd still be without a guide and have very little else besides. Instead, we have created the Documentation wiki at wiki.freenas.org and have a liberal policy of granting accounts to pretty much anyone who demonstrates any skill at writing documentation and requests an account. Desired account name, email address to send password to, and boom - you're a docs editor!
What I also hope, and have used this technique a lot myself, is that the increasing number of question on the Forum will also provide "highly directed fodder" for documentation, e.g. you see someone ask a fairly general but important question, and you may not know the whole answer to that question but you do know who to ask, so you do. Then you take their answer and try it out to see if it actually works, perhaps improve on it a little in the process, then add it to the wiki. Done! You have just learned something and documented something as a "cookbook howto" at the same time. It's a good trade, and if enough people do it, soon the users will be pretty well covered on almost all basic operations.
Having more "pop out" help in the GUI itself, in a variety of different languages, has also always been a goal but we were also forced to pick our priorities: Should we make the existing GUI perform all its functions and have at least some number of "seat belts" to prevent users from going off a cliff with it (and there are still dozens more of those to be added) even though it would be all in english and without help text? Or should er try to provide help text and maybe a major translation but wind up with some sections of the UI that are not even implemented, because we ran out of time and GUI resources in trying to do both? We decided for the former trade-off for RELEASE time, and now it's time to emphasize the latter, that's all.
There is also CLI docs hidden around here and there, for those who like CLI docs. There's "help" and "?" in the CLI itself, of course, there's the getting started guide on github (also due to be merged into the online docs but this hasn't happened yet) and there's the built-in http://yourip/cli doc browser, which definitely needs some volunteers to help flesh it out. There are also numerous examples here and also installed on every FreeNAS Corral box (hint:
print(cli_src_path) in the CLI and then start poking around in there).For myself, I will also continue to record howto videos on YouTube and post what other occasionally helpful (I hope) bits of information I can here, but so far I have to say that the very best help I've seen has come from you, the users. You've taken the fact that something was not entirely complete and instead of getting discouraged by that, you've let it motivate you into filling in some of the missing pieces. That's awesome!
Which brings me to my final point, and thanks for reading so far since I know this has been long.
6. Community & "Open Source platform"
One of the biggest personal goals I've had for FreeNAS Corral as its project leader was to finally make Corral a FreeNAS platform that others could easily contribute to. Not just where hacking on the code for the middleware or the GUI is concerned, though it's always totally great if someone dives in that far but that's not very easy. I mean easy for slightly less daunting tasks like contributing curated Docker containers, making custom VM templates, contributing to the "live docs" and howto guides, and otherwise leveraging the fact that we use github and dockerhub so much more aggressively as part of our infrastructure now that anyone can now sign up for an account on these various services and start essentially creating their own customizations to FreeNAS.
Creating "Plugins" for 9 was and is always distressingly hard, by contrast, and very few people have ever acquired the knack for doing so, but creating a Docker container for 10 is very easy by comparison and many have already created and contributed some. Probably at least a third of the entries in the FreeNAS Corral curated collection were contributed, and the infrastructures provided by github and dockerhub made it much easier to document the process as well.
Since I'm speaking on the topic of what users want and might contribute, I'd like to just say a few last words about jails. Many people do not know that in the very earliest days of FreeNAS 10, the plan absolutely was to have jail support. The problem was, we also wanted a "jail manager" to manage the jail metadata since almost no one directly manipulates the jails anymore, they use some front-end tool like warden, ezjail or iocage, and our first impulse of course was to use the warden, since it was written by one of our very own iX employees and was already being used in FreeNAS 9, but at that point the warden was also pretty old and almost entirely unmaintained, so the author asked us to wait instead for a new thing that would be taking over for the warden and be much better in every way. Well, we waited, but then the new thing went into a rewrite process of its own and we said "wow, how ironic, the rewrite is now waiting for the rewrite" so, since we figured this was still going to have to be ultimately compatible with 9 anyway, we'd just wait to see how the rewrite all panned out in 9 and meanwhile we could focus exclusively on Docker as our "plugin replacement strategy" since 9 plugins were never an option, jails or no jails, given the previous comments about how hard to create they were and the fact that 100,000+ Docker containers beat 30 plugins any day.
So, now Corral is out and we still have people asking about jails since they like them simply for the purpose of compiling their own software in them and don't want to run a VM. While we have come to resist the idea of jails somewhat simply because it sure seems like Docker containers are providing all of the same functionality with far more software availability (everything from Oracle databases to Swift language development environments) and far less overhead that some people seem to think, we're still also open to the idea. The day that the new replacement for the warden eventually makes it into FreeNAS 9.10.3 (we think), we can simply copy over the relevant bits, slap a coat of paint on them in the form of middleware and CLI support, and boom, there's the old "very low overhead" thing back but compatible with FreeNAS 9's future direction (9 users have already probably noticed a lot of their jails related tickets being pushed forward with references to "a jails rewrite" - that's this) and everyone is happy again. That's what I meant when I said earlier that everything was open to negotiation, even though many things have changed.
More change is coming, and I'm very proud of what we've all accomplished here so far, bugs and all. Now alI we ask for is the community's help in getting Corral polished to a high gloss, so everyone can run it! Please keep those tickets coming, and please keep installing and testing it so that you can file tickets. Every ticket you find and we fix (or some other volunteer does, why not?) will make the end-result just that much better!
- Jordan
for the FreeNAS Dev Team
