Important announcement regarding FreeNAS Corral

[Magnus33]

Its all really depends what iX does from this point on as to what the future of FreeNAS is.

It simply can't go on as it is in any long term capacity and hope the future is going to be great.

Changes need to be made not just to FreeNAS but to how things are done.

Oh and our lady genius in this coding area mentions going back to 9.10 won't be as completely painless as they think it is.
Not the end of the world but with more issues then they presently think.

So backup anything important just in case.

 

[James Snell]

....FreeNAS 9.10.3 is currently slated for May, and here is a look at the current roadmap (subject to change as we move farther along, of course):
....

• VM Support: We have added a new “VM” menu which allows you to host your own Virtual Machines on FreeNAS, landing in 9.10.3.

• Docker support: As a Virtual Machine-driven service.

Thanks a lot for the update Kris. I tried Corral recently and after a short adaptation period to the UI, thought it seemed really lovely. However, I had installed it on an older Xeon machine that supports both Vt-x and EPI, however I couldn't use neither Docker nor VMs due to insufficient VM support. This forced me back to 9.10.x because my application strictly requires some kind of safe isolated environment for some customizations and I ended up just using jails (actually for the first time, they're pretty slick). On the one hand, I think requiring a newer CPU is totally reasonable. But on the other, the reality is that some of us are going to use $80 dumpster-found servers that are quite capable with older features.

I guess there's no real point to my story, except:

• I tried Corral
• It looked outstanding to me
• It couldn't actually fit my use case requirements
• I sadly moved back to 9.10.x (only sad because I thought my hardware would end up unsupported soon)
• I'm happy about FreeNAS's change of direction, as it likely benefits my particular case

A final comment, another FreeBSD appliance-type distro I use is pfsense. It runs on a Pentium 4, with 768MB of RAM. While this is hilariously old, it handles tons of traffic extremely well. Computing's getting weirder since we have so much decentish legacy gear in the wild. It seems obsolescence is now a lot less due to hard limits. It's probably best for the FreeNAS community to focus on features and stability, rather than legacy support. But I suppose there's such a thing as a happy medium, since if your only users end up being those with top-end gear, you could limit exposure to folks that just want to 'kick the tires' with old hardware, who then later commit better gear. My first experiences with FreeNAS were with complete garbage gear. It went great, so I bought decent new hardware for it. My latest story has to do with my first satellite FreeNAS server that I recently squirrelled-away in my parents basement. It was an $80 workstation machine I bought off ebay. If it goes well, I'll probably end up committing better resources to it too.

Thanks for the continued efforts and I think it's wonderful to try bold new projects and occasionally make major direction changing choices. Keep up the outstanding leadership and work. Thanks community & iX!

 

[nasrettin]

I am really disappointed with the decision of rebasing. Dockers in Corral really worked for me and I really liked Corral's UI -simple enough to understand what's going on but sophisticated enough to run complicated set-ups. I really don't want to revert back to 9.10 and/or get used to another UI but I guess I will have to unless I move away from freenas completely.

It would really help me feel better about all this if I understood the details behind the decision of rebasing to 9.10. Kris Moore mentions issues with MontageJS and Plan 9 filesystem code but not enough details is provided for me to be able to dig through and find out what exactly were the wrong decisions made in Corral and why IX decided to roll back.

Also, I really admired jkh and his communication in this forum with the community. It is a real loss that he moved away all of a sudden. Perhaps I am reading too much between the lines but in his last post he writes:

"Even though some folks have somewhat stubbornly refused to see FreeNAS as a genuinely open source project and instead one in which iXsystems calls all the shots and its users are just spectators, you collectively have more power and influence than I think you've ever realized. "

I wonder what's the story behind here? If I were to trust IX systems for their future releases (and I really would like to), I would like to understand a little bit of what's going on behind the scenes. It would be especially enlightening if someone could explain in some detail what the reasons are for rebasing to 9.10.

Thanks.

====

William Grzybowski gave some details that was helpful to me but it seems that his post disappeared. Luckily, I had it opened in my browser so here is a copy of it in case it helps others:

"First I would like to make it clear this is my personal opinion. Do not take it in anyway as official reasons.

It was not about anything the user can see, its all about what is under the hood.
The following image illustrates it pretty well:



In my vision thats exactly what happened.
Corral was very beautiful looking, from outside. The UI is quite beautiful, the CLI did what it was supposed to do, it has shiny dockers.

There are several things that did lead to this.

First point is the UI. Apparently writing full blown Rich Internet Applications is hard. As you did read in this first post the core of the UI was MontageJS, it does not matter anymore why it was chosen, but even the developers who helped write this framework did not like it. It was already expected to be replaced to something different but were stopped by someone so the product could actually be released. The whole UI was built without a single automation test and there are probably about 5 people in the world who really understand this framework. So how were we supposed to keep supporting this virtually dead javascript framework (to us at least) or completely swap it with something else and without automatic testing in place? its like getting a Muscle car, put the engine of a Ferrari _while_ its still running and expect everything to keep working perfectly. So yeah, not happening...
Even after this long period of development there were basic things that simply did not work in the GUI. One can't even replace a disk from GUI. It only works well in Chrome. It does not provide historical reporting stats. There is no documentation. Translation is not supported, it was not built with that in mind.

The backend architecture is over engineered. There was no such thing as "do not reinvent the wheel". For instance, did you know the DHCP protocol was rewritten in a python library and is leading to several compatibility issues? The well known, stable and worldwide used system logging (syslog-ng) was rewritten to a python log daemon (from scratch), making diagnosing debugging from a support perspective much harder? The core of the middleware was about to be rewritten from scratch using another language? Cython modules created without thread-safety in mind (e.g. smbconf) which can cause service corruption?

Several changes were made to FreeBSD which could not be upstreamed, making it very difficult to maintain. ZFS was modified to have a special aclmode making it incompatible with every other ZFS implementation (breaking replication for instance).

The whole middleware does not have a single functional test. Thats right, whole middleware was built without a testing harness. (Fix one thing, break another).

The system is bloated. Have you actually tried running a "top" in a stock system and check how many "auxiliary" services there are? Close to a dozen python services, eating up a lot of RAM.

There is no documentation for the new REST API. There is no compatibility whatsoever with the old REST API rendering multiple applications useless (specially for the enterprise side, e.g. Zabbix, vCenter, etc).

Several design decisions were made in the course of development that simply made a seamless migration from old codebase to Corral extremely hard and painful.

The list goes on, but I will stop here. I am probably getting too technical.
All of that (and more) made us realize it was going to be a lot more work to bring it to sanity then incorporate the features on the 9.10 codebase.
Also to give you an idea, the new experimental UI (which you can find in 9.10 nightlies) was written in about a week, works on every major browser (even mobile) and supports translation out of the box. That has shown us it does not need to be complicated. "

 

[Ericloewe]

There are some details in an earlier post:
https://forums.freenas.org/index.ph...rding-freenas-corral.53502/page-7#post-369843

 

[WrlsFanatic]

FWIW, the devs are open to having a migration path from Corral to 9.10.3, and I'll be sure to lobby for that to happen. Docker is the only complication, so let me ask something: Is it better to hang on to Corral until a 9.10.4 that includes Docker support or migrate the Docker VM to 9.10.3, despite the lack of Docker integration with the host?

100% wait for 9.10.4. I can't see myself switching until 9.10.5 or later, frankly. I just have the server with a Plex container right now, so I can probably make it another year before I upgrade. I'd much rather have the full UI overhaul out and full docker support and then a bugfix release out - especially given this news - before I migrate.

 

[WrlsFanatic]

I'm not trying to be condescending, although I know it may come off that way, but you're under 30 years of age, or have less than 5 years in the IT business, correct?

You never, and I mean never put a major release into production within 30 days of its release.

I said in another post that anyone who put this into production at a business made a mistake.

 

[Matt Platte]

I'd put Docker orchestration into a VM and absolve the host of all direct Docker responsibility. Better to have FreeNAS certified (whatever that might mean) VM's for Kubernetes, Swarm, etc. and let the user choose their poison.

 

[JoshDW19]

I said in another post that anyone who put this into production at a business made a mistake.

No bones about it we messed up and I understand the frustration. Just know this was a very difficult decision to make and it took months of research at different development intervals and weeks of deliberation to reach a consensus. Although it was a tough decision, I'm hopeful that everyone will see this was the best option going forward for FreeNAS. We appreciate every one of you as a community member @WrlsFanatic and I just wanted to let you know this decision wasn't made lightly.

 

[duel007]

This is really disappointing. I was excited for 9 to die.

 

[Stanri010]

There are some details in an earlier post:
https://forums.freenas.org/index.ph...rding-freenas-corral.53502/page-7#post-369843

I just don't agree with the rationale and the direction the "new" people in charge now are taking FreeNAS.

See, to me it sounds like when the "new platform" team fell apart when members left or were ousted, the "stability" team took one look and said we don't want to fix what the other team had done. It's understandable that the "stability" team didn't see it as their project and of course never had buy in on the decisions made. The result was them taking the easy way out and picking and choosing some of the features they could implement and throw the rest in the trash.

If they really owned up to the whole debacle, they would pull FreeNAS Corral from stable and put it into another releases candidate or beta and continue working from there. Instead they're abandoning that platform and early adopters and so much of the work that was previously done. The old team that worked on Corral were forward thinking and tried to do things different, but were perhaps pushed by management to push a product that was not ready. The team taking over is much more aligned with the enterprise and don't care for the new features. It's great for enterprise IT users and I'm sure that's where the money's at but not so good for the home user or hobbyist. This does not allow FreeNAS to compete with other feature rich NAS softwares on the market.

 

[Ericloewe]

I just don't agree with the rationale and the direction the "new" people in charge now are taking FreeNAS.

See, to me it sounds like when the "new platform" team fell apart when members left or were ousted, the "stability" team took one look and said we don't want to fix what the other team had done. It's understandable that the "stability" team didn't see it as their project and of course never had buy in on the decisions made. The result was them taking the easy way out and picking and choosing some of the features they could implement and throw the rest in the trash.

If they really owned up to the whole debacle, they would pull FreeNAS Corral from stable and put it into another releases candidate or beta and continue working from there. Instead they're abandoning that platform and early adopters and so much of the work that was previously done. The old team that worked on Corral were forward thinking and tried to do things different, but were perhaps pushed by management to push a product that was not ready. The team taking over is much more aligned with the enterprise and don't care for the new features. It's great for enterprise IT users and I'm sure that's where the money's at but not so good for the home user or hobbyist. This does not allow FreeNAS to compete with other feature rich NAS softwares on the market.

What new features? All the big additions in Corral are getting added to 9.10.

 

[Ericloewe]

I'd put Docker orchestration into a VM and absolve the host of all direct Docker responsibility.

That's planned as a stopgap for 9.10.3.

 

[JoelN]

There are some details in an earlier post:
https://forums.freenas.org/index.ph...rding-freenas-corral.53502/page-7#post-369843

". . . some of the remaining team members began exposing some of their own concerns regarding the architecture that they'd been previously holding back."

This is the part that is most perplexing to me. That team members were "holding back concerns" which were apparently important enough to curtail development would seem to indicate serious communication issues. Unless I am reading this wrong.

 

[Stanri010]

What new features? All the big additions in Corral are getting added to 9.10.

What about features that were integrated into the design rationale of FreeNAS 10 such as usability and quality of life improvements.

Such as permissions being set up automatically and generally working right out of the box? I remember I spent days trying to get permissions to play nice when I first installed FreeNAS 9. The clean install of Corral worked right out of the box.

How about automatically enabling the shares when you create a dataset for that kind of share?

FreeNAS 9 feels like it was made by engineers for engineers while 10 made it a little more accessible. I don't think a simple reskinning 9.10.3 will fix all of the nuances that were designed into 10.

 

[William Grzybowski]

What about features that were integrated into the design rationale of freeNAS 10 such as usability and quality of life improvements.

Such as permissions being set up automatically and generally working right out of the box? I remember I spent days trying to get permissions to play nice when I first installed FreeNAS 9. The clean install of Corral worked right out of the box.

How about automatically enabling the shares when you create a dataset for that kind of share?

FreeNAS 9 feels like it was made by engineers for engineers while 10 made it a little more accessible. I don't think a simple reskinning 9.10.3 will fix all of the nuances that were designed into 10.

Nothing is set on stone.
Can you help us telling what you most liked about Corral which you would like incorporated/changed in 9?
Nothing you described here seem to be impossible to change in 9, quite the contrary.
We would love to hear everything you have to say and transform them in possible features for improvement: http://bugs.freenas.org
I can say we strongly believe in slowly iterating something to make it better.
Corral itself did not simply appear out of the blue, it learned from 9 mistakes but it also did commit its own mistakes.

 

[Stanri010]

Nothing is set on stone.
Can you help us telling what you most liked about Corral which you would like incorporated/changed in 9?
Nothing you described here seem to be impossible to change in 9, quite the contrary.
We would love to hear everything you have to say and transform them in possible features for improvement: http://bugs.freenas.org
I can say we strongly believe in slowly iterating something to make it better.
Corral itself did not simply appear out of the blue, it learned from 9 mistakes but it also did commit its own mistakes.

While it's clear the decision has been made and we can try to align 9.10.3 to be more similar to what 10 was supposed to be. It might help a lot of us end users understand what the rationale was to backpedal and go back to FreeNAS 9.

What specifically was it that was so bad that could not be fixed or rectified that necessitated the rollback to FreeNAS 9? I've thought about it and I can only conclude that something or multiple things about corral could not be easily rectified so they were left on the chopping board and thus are going to be left out of 9.10.3. IF there was a fix for it to be put into 9.10.3 then why not just fix the issue with FreeNAS Corral? It sounds like the team decided some features were too hard to fix or it'd take too long so they weren't going to fix it and just abandon it. By making the decision to abandon 10, that just means something's going to be left out and I'd be curious to know what that was.

 

[Ericloewe]

Such as permissions being set up automatically and generally working right out of the box?

Permissions cannot work out of the box for everyone, that's what the sane defaults are for: Everyone reads, owner writes. This is a boogeyman which was blown out of proportion.

How about automatically enabling the shares when you create a dataset for that kind of share?

I'm sure that could be done. Or at least improvements in the interaction between the shares and datasets UIs.

 

[JoshDW19]

While it's clear the decision has been made and we can try to align 9.10.3 to be more similar to what 10 was supposed to be. It might help a lot of us end users understand what the rationale was to backpedal and go back to FreeNAS 9.

What specifically was it that was so bad that could not be fixed or rectified that necessitated the rollback to FreeNAS 9? I've thought about it and I can only conclude that something or multiple things about corral could not be easily rectified so they were left on the chopping board and thus are going to be left out of 9.10.3. IF there was a fix for it to be put into 9.10.3 then why not just fix the issue with FreeNAS Corral? It sounds like the team decided some features were too hard to fix or it'd take too long so they weren't going to fix it and just abandon it. By making the decision to abandon 10, that just means something's going to be left out and I'd be curious to know what that was.

This post from Brett Davis (our executive vice president) on reddit may explain more. I'm going to re-paste and link it here for you:

*Brett: We had two development teams at iX during the creation of FreeNAS Corral. One worked on the currently shipping product (FreeNAS 9.x, and the enterprise version, TrueNAS), and one on our re-write (FreeNAS Corral). Both teams operated independently from one another, under separate leadership. The former, managed by Kris, were charged with evolving our shipping products, so they continued to update and focus on stability for FreeNAS/TrueNAS 9.x (v9.10 was released for both during this time, they were also evolving the UI, etc) while the latter focused on developing the new platform (FreeNAS Corral). The plan after release of FreeNAS Corral was to eventually merge the teams and technologies, focusing on one platform.

With one team focused on "stability" and one focused on "new", there was a difference in philosophies between the teams, as I'm sure you can imagine. There was certainly a faction at iX that expressed concern about the release, but the team in charge of Corral decided to release anyway. The leadership for that team promptly left, and once that happened, some of the remaining team members began exposing some of their own concerns regarding the architecture that they'd been previously holding back.

This prompted the thorough engineering review that lead to the collective decision that Corral, while having awesome and exciting ideas, was ultimately going to be a much longer path toward stability than instead re-basing the new and exciting features on our already stable 9.x series.

So, we now have a singular development team again after merging, and they are working together to do what we probably should have done long ago and focus our efforts on one product. Lessons learned for all.*

 

[Stanri010]

This post from Brett Davis (our executive vice president) on reddit may explain more. I'm going to re-paste and link it here for you:

*Brett: We had two development teams at iX during the creation of FreeNAS Corral. One worked on the currently shipping product (FreeNAS 9.x, and the enterprise version, TrueNAS), and one on our re-write (FreeNAS Corral). Both teams operated independently from one another, under separate leadership. The former, managed by Kris, were charged with evolving our shipping products, so they continued to update and focus on stability for FreeNAS/TrueNAS 9.x (v9.10 was released for both during this time, they were also evolving the UI, etc) while the latter focused on developing the new platform (FreeNAS Corral). The plan after release of FreeNAS Corral was to eventually merge the teams and technologies, focusing on one platform.

With one team focused on "stability" and one focused on "new", there was a difference in philosophies between the teams, as I'm sure you can imagine. There was certainly a faction at iX that expressed concern about the release, but the team in charge of Corral decided to release anyway. The leadership for that team promptly left, and once that happened, some of the remaining team members began exposing some of their own concerns regarding the architecture that they'd been previously holding back.

This prompted the thorough engineering review that lead to the collective decision that Corral, while having awesome and exciting ideas, was ultimately going to be a much longer path toward stability than instead re-basing the new and exciting features on our already stable 9.x series.

So, we now have a singular development team again after merging, and they are working together to do what we probably should have done long ago and focus our efforts on one product. Lessons learned for all.*

My question was based off of that knowledge that you linked, so let me rephrase the question.

What was it about FreeNAS 10/Corral that made it much more difficult to stabilize:

This prompted the thorough engineering review that lead to the collective decision that Corral, while having awesome and exciting ideas, was ultimately going to be a much longer path toward stability than instead re-basing the new and exciting features on our already stable 9.x series.

I assumed it was due to the way the development team approached specific features and again, I assumed that the solution was to cut those features out to reach stability. That's the only way you guys can push out a 9.10.3 next month.

So, what were those features that aren't being implemented or what makes 9.10.2 more suited to be the starting point. Is it iSCSI? Backwards compatibility? Resource usage? There must have been a smoking gun to have scrapped nearly 2 years worth of work.

 

[Ericloewe]

What was it about FreeNAS 10/Corral that made it much more difficult to stabilize:

My guess is simply "more bugs", since it's a new codebase and many essential features were not working properly yet (even disk replacements were messed up).