Path to success for system upgrades

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
To add my story here: I started with PCs in 1990 on a 286 with a 42 MB hard disk (Seagate ST251-1), which about one year later had issues with faulty sectors. This was a couple of years before (consumer) hard disks started to internally re-map bad sectors. This was the first and last time that I lost data. Ever since, I have been paranoid about backups (and more importantly restores).

I started with simple floppy disks for source code, spreadsheets, etc. and went on for a DAT streamer in 1996. This only lasted 3 years and after that abysmal experience I switched to a QIC streamer, which lasted until about 2008. Well, that's when I put it out of service. It was in perfect working condition but 12 GB capacity per cartridge started to be an issue. Since then I have used hard disks in various ways, since streamers have become a prohibitive upfront investment for me. I would still prefer streamers, but that is a different story.

All the people I know (incl. at work) initially think of my efforts as overkill. Until they loose 10 years of digital pictures, esp. when children are involved. That is when they are willing to invest time and money. The same goes for many companies, unfortunately. A friend told me about a malware attack on his employer about a year ago. All of a sudden there was budget for keeping backups longer than just 30 days, a properly segmented network, and other things their IT department had wanted for more than a decade. Everybody (incl. me - see above) has to learn this the hard way, I guess.

A side note on NAS gear that is typically more in the consumer space. I am currently in the process of switching to a new FreeNAS box. There were long deliberations as to whether I should go for Synology instead. The core reason why I stayed with FreeNAS is that it I have flexibility. From a usability and ease-of-use perspective I got the impression that Synology is (far?) superior. But that comes at the price of simplicity. A mass market product needs to keep support tickets under control and the only way for that is to limit people's options. And I wanted to stay flexible, even if that meant to spend more money (hardware specs are not comparable to the Synology model in question) and time for setting things up.

Finally, I am not going for TrueNAS 12 right now but start with FreeNAS 11.3 U5. Yes, I have seen and read many highly positive comments about v12 and how stable it is. But IMHO nobody can be really sure for a couple of weeks that no hidden errors exist. I work in the enterprise software space and I don't know of any customer that went into production with a x.0 version of our products. You simply don't do that.
 
Last edited:

ornias

Wizard
Joined
Mar 6, 2020
Messages
1,458
Finally, I am not going for TrueNAS 12 right now but start with FreeNAS 11.3 U5. Yes, I have seen and read many highly positive comments about v12 and how stable it is. But IMHO nobody can be really sure for a couple of weeks that no hidden errors exist. I work in the enterprise software space and I don't know of any customer that went into production with a x.0 version of our products. You simply don't do that.
As discussed ealier, the x.0 version (Release) of TrueNAS 12, is basically what previously would've been 12U1. "You simply don't do that. " based on the "feeling" behind a version number is also not very rational imho. You need to verify per product what a version number means.

For example iXsystems decided to use OpenZFS 2.0 RC for TrueNAS Core, because they also changed versioning. Current OpenZFS RC is closer to what would've been ZFS-on-Linux 0.9.1 already and possibly closer in versioning to what previously would've been 0.9.2.
Yet a lot of people would've jumped the bandwagon already on 0.9.1 if it was just called that.

Versioning maters, but not every version means the same when it comes to RC statuses.

A lot of people use it for months by now, a couple of weeks more aren't going to show any new data breaking bugs (it also uses stock OpenZFS 2.0, which has a significantly bigger userbase already than TrueNAS)
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,919
As discussed ealier, the x.0 version (Release) of TrueNAS 12, is basically what previously would've been 12U1. "You simply don't do that. " based on the "feeling" behind a version number is also not very rational imho. You need to verify per product what a version number means.
[..]

I don't think that we fundamentally disagree on anything here. My background is, among other things, mission-critical transaction processing software (think e.g. CICS). So with the old versioning scheme I would not have switched to a new version before U2. I agree with you that a "feeling" is not the best reason for such a decision and can say with confidence that this was not the case here.

Thanks for your thoughts.
 

Berna

Dabbler
Joined
Mar 23, 2018
Messages
11
I think OP is coming from a standpoint of "I don't want to lose my data", where a backup is always, always a good idea anyway.

In my case, I've started with 11.2RC1, went to 11.3 early, and now to 12.0 early, and I haven't had issues that weren't easily resolved. In order to keep things chill for me, I am:

- Not using GELI encryption, that's too brittle / scary for my tastes
- Snapshotting all datasets individually, recursively, with a 2-week lifetime
- Backing up my config daily and sync that to cloud
- Following recommendation 4) in the OP. Keep boot environment; feature flags only when I know I will never ever go back.
- Using "boring" hardware - Intel NIC, Supermicro board, ECC memory, SSD boot. 9 out of 10 FreeNAS/TrueNAS issues are hardware-related.

I haven't had issues that caused me to have to go back, and, a boot environment makes it trivial to do so, and the snapshot means that if there's a change to iocage that breaks things (looking at you, NAT setting in 11.3), that's easily and rapidly rolled back.

For my home use, upgrades have been uneventful. And, things can go wrong, and hence all the caveats.
I
Path to success for system upgrades

Too many folks here just blindly upgrade, and find themselves in a pickle afterwards (jails no longer working, encrypted pools not unlocking, or other nastiness). All too often, their predicaments are addressed in the upgrade's release notes, or in the Guide section 1.1 on the changes adopted in the new version. So before you blithely click Upgrade, please consider the following:
  1. Read the release notes carefully, along with the Guide for the new version, section 1.1. Look for any gotchas, deprecated features, or one-way actions afterwards (like upgrading ZFS pool features) that would prevent going back. Read them again. Read them a third time. In fact you should re-read them both several times fully until you understand all the ways the developers have foreseen you could run into difficulty after the upgrade.
  2. Prepare an action plan to mitigate all the gotchas in your specific installation.
    • In particular, backup your GELI keys, recovery keys, SSH keys, and system configuration.
    • It's prudent to have replacement media on hand for your boot pool, along with a USB 2.0 thumb drive prepped with the installer for the new version.
    • Have printed screenshots of all relevant configuration screens, in case your configuration backup doesn't work, and you need to re-enter everything again by hand. Make sure these are current.
    • Likewise, have screenshots of the relevant jail/plugin/VM configurations, in case you need to rebuild these from scratch.
    • Backup your pool, in case you need to reconstitute it.
    • Test your backups to make sure they work.
    • Review the steps needed to reboot back into your current version from the Guide, section 2.5.5.
  3. All this prep work may seem like overkill, but if something goes wrong, you'll have a plan to address it, and as a last resort, revert back to the last known working configuration.
  4. Once you're in the new version, and have solved all the post-upgrade issues, leave the old boot environment in place, and don't upgrade any ZFS feature flags for at least a week. If you find an issue that appears after the immediate aftermath, you'll still have a way to revert back to the last known working configuration. After this feeling out period, you can consider making one-way changes that will prevent you from going back.
Remember, a system upgrade is a high-risk event. Many things can go wrong, and an ounce of prevention is better than a pound of cure.
Is it not beter that Trunas get an simpel helper ther for Backing up 1000 things ``
Like a tracker when i install i jail it get higlited and if i do an systemsave (wathever you cal it befor backup) and it saves also the jail config and teh plex config in the same bundel...
Othevice it is totaly usles program if i do need to hav an alftime job just to kep up whit the bugs and al tha backup tha i have to do when it vas my intention to get the freenas is an esay reliebel Raid that i can trust on ...
But after upgrading to 11.3 now i RELLY start to mistrust in Freenas/Trunas
I cant go from Frenass 9 somthing to 12 trunas so i have to take midelvay and then Got rely stuck
Cant updat true internet cant accsess the net,, Download and try manuel way and it refuse to higligt the instal buton and i can read one milin paper about the Destroied cerificate shity but non can rely exsplain how to solv it
So that you are saing that Trunas is not any prog to get if im woried about my DATA
It is nobudy whit an life and an orinary work that can do all that for an upgrade.
Regards
Berna that just giv up on Freenas/Trunas it is just crap almost as bad as WD stuf
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
If you go to each new release in a timely manner, updates are mostly painless. It is your responsibility to keep your software at a maintained and supported version. The fundamental subsystem for jails was completely rewritten in FreeNAS 11. The time to migrate your FreeNAS 9 jails to the new system was THEN. Not NOW. This was clearly documented and migration helper scripts provided.

Keep your software current, folks. I have little empathy for someone complaining that there is no easy upgrade path for a system that is 10 years old. Especially when it is free and open source. PHP releases have a lifetime of two years. Plan for updates at least every two years for every open source product you depend on. Better make that one year.
 

Berna

Dabbler
Joined
Mar 23, 2018
Messages
11
It is even VORS now.
I cant upgrade the cor produkt whitout droping all things att hand and devote 2 weeks resherce what went wrong.
And i only have an file server por gay that have jail or plugins.
Freenas/trunas is prbebly dead
What ever i trye i have a big wall obstical to go around work around and frustration
Now it start agen i get ip http:xxx.xxx.xxx.0 http:xxx.xxx.xxx.41 https:xxx.xxx.xxx.0 https:xxx.xxx.xxx.41
On the samt F nick and no explanation whay if i unplug the tp cabel and put it back it got one of eatch http and https the one whit 41 and network works
IS that an harwher problem or ??
So no you are geting wohre and whorch on the support side
 

Berna

Dabbler
Joined
Mar 23, 2018
Messages
11
same agen YES good stuff
It have worked sins 2017 jan i built it and it hav only been down a few times becuse of hard wher and then now after im being forsed to upgrad it is totoly whorthles
I hope i can get it to wor so far that i get an new somthing else and transfer data to that place
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
You should have upgraded to each release when it was published and the previous one declared EOL. You are aware what "end of life" is supposed to mean?
 
Joined
Oct 22, 2019
Messages
3,641
Keep your software current, folks.
I think one reason people don't like to update their system every single time there's an update or patch is because this requires a reboot.

I know the Linux kernel is capable of "live patching", which spares a reboot when you simply need to apply security updates and certain bugfixes (see: RedHat's "kpatch" and Oracle's "Ksplice".)

I don't believe FreeBSD has any such mechanism? (...yet?)

What is hindering "live patching" from becoming more commonplace in general?

Imagine seeing the notification "A security update is available for TrueNAS", then applying it, and that's it! No need to reboot. You've updated a system with important security patches for the kernel and zero downtime. :cool:
 

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
@winnielinnie Seriously ... what is the problem with a reboot? Less than 5 minutes of downtime. And a clean state of the system and the certainty that all services will come up in case of an unexpected reboot, should that ever happen.

I always reboot my systems after every update even if the update in question does not mandate it!

I as a "professional operator", i.e. systems and network engineer, love reboots. Nothing like that fresh system smell ... :tongue:
 
Top