FreeNAS 9.2.0-RC2 now available

[jkh]

Hey folks,

Due to the huge amount of community participation in FreeNAS 9.2.0-RC (during which 49 bugs, some of them fairly significant, were reported and fixed) we have decided to take the unusual step of rolling and releasing another Release Candidate image, FreeNAS 9.2.0-RC2, due to the significant number of changes we had to make in fixing those bugs.

We were all extremely gratified by the response to FreeNAS 9.2.0-RC. While it is always unfortunate (and a little embarrassing) to have significant bugs in a Release Candidate image, we are nonetheless a small development team and simply cannot visit all the corners and various usage cases of FreeNAS by ourselves. All of you together, however, can! That is where community participation, and your willingness to use the bug tracker, has been absolutely invaluable in to helping to polish FreeNAS 9.2.0! Since FreeNAS 9.1.1 was released, we have now fixed 245 bugs (and requested feature enhancements) in the bug tracker and countless more during our own testing and "fix a bug to fix a bug" investigations. We have also substantially improved performance, the UI, and the overall level of polish in FreeNAS along the way.

In any case, barring any more significant bugs being found, this will be our last Release Candidate before going to 9.2.0-RELEASE (hopefully to be released by Christmas). Please jump on this one with the same enthusiasm you did the first release candidate, and if you find a bug, please file a bug report as quickly as you can! Should any "show stopper" bugs be found, we will obviously not release 9.2.0-RELEASE until those bugs are fixed, so in a very real sense, all of you are gating the release process!

If you have any questions about this process during the run-up to 9.2.0-RELEASE, you can also look for us here on the forums or in the #freenas channel on irc.freenode.net. The FreeNAS development team is more than happy to engage with the community in real-time!

Please see the ReleaseNotes for a list of changes in this RC2 release (which have changed since RC, so even those who already installed 9.2.0-RC may wish to review them).

Again, the 9.2.0-RC2 bits may be found here: http://cdn.freenas.org/9.2.0/RC2/

Thanks for your help in making 9.2.0 an excellent release!

Sincerely,

The FreeNAS Development Team

 

[TenzoR]

Very exciting, can't wait till release!

 

[DrKK]

Jordan:

While this appears much better than the beta and RC1, and a lot of the bugs that were annoying me seem fixed:

I still had a few problems right out of the gate going with a standard installation trajectory---so nothing special or fancy, just what any user would do. I'm sure there's bugreports on this, but really, I'm not sure how this all escapes notice since it's in the standard install itinerary for an end-user. I'm not going to make a laundry list here, since I think most/all of these are already bug reports...but still, I will name one that really sends me into a WTF spiral: The fact that a user can do a standard install, with standard everything, go to the first GUI access, be advised to change his password, do so, AND THEN BE GREETED with a scary-looking traceback error, that just blows my mind. All three of us in Mumble last night, on totally different hardware, had the same result on a fresh install of the RC2. Does this mean no one over there even tries to install the final compiled RC's on a test server before they hit the file server to make sure there aren't any nubcake mistakes?

 

[Yatti420]

Odd.. GUI upgrade worked fine..

 

[cyberjock]

The issue is for new installs from CD. https://bugs.freenas.org/issues/3691

 

[jkh]

Jordan:

While this appears much better than the beta and RC1, and a lot of the bugs that were annoying me seem fixed: but still, I will name one that really sends me into a WTF spiral

I'll take full ownership of that one - I got tunnel vision on verifying that a bunch of *other* bugs had been fixed and did a series of GUI upgrades to get from build to build as part of that process. Obviously for a full and actual RELEASE build we have a much more comprehensive process that involves a spreadsheet checklist with both i386 / x64 installs (from scratch) on actual hardware, also visiting every corner of the UI we can think of and verifying that things like AD and LDAP work in a more enterprise-y environment (I hardly ever test AD as part of my daily "living on" experience because we don't use AD here, for example), etc etc.

Anyway, it takes over a week, if not sometimes two weeks, to go through all those tests, so I knowingly cut some corners to get another RC out the door and, well, it bit me! We're still looking for where that traceback occurs - how it got in there at all is a mystery.

At the very least, I'll be testing both email notifications AND full, from-scratch installs as part of the "short version" of the checklist, even for BETA or other pre-release candidates. We DO have QC measures in place, despite all evidence to the contrary. We just sometimes choose not to skip them for reasons of expediency, and sometimes we even get away with it! Then someone loses and eye and we remember "Ohh, right, that rule about not running with sharpened pencils is on the wall for a REASON!"

 

[Enlightend]

https://bugs.freenas.org/issues/3656 is noted as bypassed with a patch, yet I still get the exact same behavior.

Did this patch not make it into RC2 or am I experiencing the same behavior from an entirely different bug?

 

[jkh]

https://bugs.freenas.org/issues/3656 is noted as bypassed with a patch, yet I still get the exact same behavior.

Did this patch not make it into RC2 or am I experiencing the same behavior from an entirely different bug?

The patch definitely made it into RC2. If you're still seeing this, PLEASE comment in that bug so we can revisit it. Any interesting data you can paste in would also be appreciated. Thanks!

 

[Enlightend]

Ok, done that.

From the wording of the talk in the bug report, the solution might have been focussed a tad to much on zvols that are parts of zpools, rather then zvols for iscsi extents that are hosted in a pool.

Since I have been reading up on zvol vs file extent lately and saw file based has an edge on 10Gbe links, I've begun switching over to that.
But I'll keep a small zvol based iscsi extent live to do some future testing.

 

[Milkwerm]

I got the trace back error page on reboot of a GUI upgrade today FYI...(from 9.1.1)

 

[Magnus33]

No reason at all to be embarrassed since you guys are working with lord knows how much code there going to be bugs.

Now if there weren't any would be getting worried and then shortly expecting a announcement from a evil island lair about world domination ;)

 

[ThreeDee]

well poo .. I can't log in after a gui upgrade from 9.1.1

..it asked me to reset password and I did .. won't take the new or old password now ...

won't take blank .. or admin plus no password ..

 

[Dusan]

well poo .. I can't log in after a gui upgrade from 9.1.1

..it asked me to reset password and I did .. won't take the new or old password now ...

What user name are using to login into the GUI? It should be root now (instead of admin).

 

[ThreeDee]

woops .. Thanks a bunch .. that was it.

 

[Scareh]

We DO have QC measures in place, despite all evidence to the contrary. We just sometimes choose not to skip them for reasons of expediency, and sometimes we even get away with it! Then someone loses and eye and we remember "Ohh, right, that rule about not running with sharpened pencils is on the wall for a REASON!"

that right there, thats what scares the living shit out of me.
You are playing with real people's data here, there should never be an excuse to not test something before you upload it as beta/rc/whatever. There will be allways an idiot who thinks oh look an RC, its a valid image, lets use it.
And then it crashes because *you* chose to skip a corner in quality testing? Be ready for a sh*tstorm heading your way then.
Delay a RC/beta/whatever for as long as you want for all i care BUT NEVER EVER EVER EVER EVER (you get the point) skip your basic QA-testing. Thats just plain stupid, and not even close of what i expected of you guys.

 

[cyberjock]

Scareh,

To be completely honest, I fully understand and appreciate your concern. I have the exact same concern and I'm always hesitant to do an upgrade. But, if you were around when 9.1.0 was in beta testing we had some users with problems similar to the ones you are concerned about.

A handful of users had non-standard configurations that the server admins had screwed up when setting up their pools and it made pools unmountable in 9.1.0. Of course, some of these people had also upgraded their pool to v5000 so they couldn't downgrade back to 8.3.1. Their data wasn't in jeopardy, but it was completely inaccessible to the owner. So from their perspective the data was lost. One of the FreeNAS developers came up with a patch for those users that was used by a handful of people and it did get their pools back online.

Overall, I share your scepticism with regards to playing with people's data. But, I think that if all hell broke lose and a bunch of users suffered a bug that didn't destroy the pool I think that the FreeNAS developers would be interested in troubleshooting the problem and coming up with a solution.

By far, the biggest "threat" I've seen to people's data is doing things that are incredibly stupid. Usually it centers around not understanding the need for quality hardware with ECC RAM, then extends to them not understanding what they are doing in the WebGUI and making one or more *serious* noob mistakes leading to them killing their pool. In the most recent case I've worked with the user that did this:

1. Had a 4 disk RAIDZ1(remember how much I criticize RAIDZ1s....)
2. Did a disk replacement without a disk being bad. (In fact, all 5 of their disks were good and tested good by me last week)
3. Had non-ECC RAM. RAM tests done last week found no errors, but that doesn't rule out bit-flips etc.
4. After doing the disk replacement never left the server up long enough to ever finish a resilver. (Why did they do this? They didn't know better!) They'd boot up the server and use it for an hour to copy files to or from the server, then promptly shutdown to "save power". Some of their hard drives had more power cycles than power-on hours. This was part of their daily cycle to backup small mounts of data at the end of the workday which typically took 15-20 minutes at the most.
5. About a week or two later they booted up the server to find that the pool was in an unmountable status.

I was unable to get their pool back despite spending more than 15 hours reading and trying various things. The pool just would not mount under any circumstances.

Overall, I'd think that there's 3 situations that would occur if a serious bug were found:

1. Bug is found that prevents access to the data, but the data is safe. This is the scenario that we saw pre-9.1.0.
2. Bug is found that causes some kind of damage to the pool, but is undoable. I'm fairly confident that in this situation a patch of some kind would be created.
3. Bug is found that destroys data with no chance of recovery. This would obviously be very crappy, and it is quite possible that if this situation were to happen it would result in the project never being trusted again.

Edit: Finished my sentences. :)

 

[Enlightend]

Scareh:

1: ZFS is independant of FreeNAS, to screw up your data, you have to do something insanely stupid. Aside that, you should always have several versions of the config file for freenas, so that you can easily do a fresh install, import the zfs filesystems and restore the config.

2: You have nothing to complain about if you are installing RC's and Beta's on a system with non volatile data, it's your error. Not theirs for skipping a QA step, you apply anything, RC, Beta or in any other way untested to a live system and there is only one person to blame when things go bad and that is you.

 

[cyberjock]

Actually, I think that #1 is not entirely correct. I believe FreeNAS' ZFS code is a combination of RELEASE and STABLE code put together by the iXsystems guys. As such, it's basically customized code the FreeNAS guys put together. If I'm correct then it is possible that a bug could be introduced that could cause unrepairable damage. But don't quote me on this. ;)

I know that our performance boosting encryption patch is not even in STABLE yet. It's had only limited testing, but the performance boost of more than 8 fold in my own testing shows that it was a patch that was in dire need of being used. I'm no C programmer, but I'm fairly sure that the patch is safe as some test pools worked without problems on 9.1.1 and 9.2.0. Of course, this in no way means that the code has been "extensively" tested and certainly shouldn't be taken to assume that every "edge case" has been fully explored just based on my limited experiments.

At the end of the day, the best solution in my opinion is to either trust iXsystems to not kill your data and use FreeNAS, or decide they haven't earned your trust and should use a competing product. Look at how much people criticize Microsoft and yet people still use it. I'd bet that Microsoft has been responsible for significant data loss over the years either through their action or inaction.

iXsystems does have a business side of the house (TrueNAS) and it would reflect very poorly on them if people using FreeNAS were to start suffering massive quantities of lost data as a result of buggy code.

So the real question is do you trust a well known name like John Hubbard to make sure he's not releasing a software package that could cost you your data or some other package controlled by someone else?

We're all human and prone to mistakes. So backups are definitely a good idea. ;)

 

[Enlightend]

Cyberjock: with 1# you are already talking at the level of the pools being imported and in use by freenas. What I'm saying is that before you do that, ZFS filesystems are quite safe. It's not like the old SoftRaid and MDRaid bonanza where you could utterly murder a storage pool by just recompiling something.

To do the same with ZFS, you already have to go trough steps of importing the pools and addressing the data.

A good solution to make things even safer is making Freenas ZFS bootable, like Solaris does. If the system boots from ZFS, you are already relatively certain that it doesn't fuck that up.

As for the later part, I have to repeat, if you install RC, Beta or any other non Release system software on your system, you are to blame for data loss. That's what those labels are for.
Using non release software on supposed important data is leaps beyond that and must make one consider what the hell they are doing.

 

[Sir.Robin]

Great stuff :)