Upgrade from Bluefin

[jlw52761]

It looks like there's a lot of issues with both upgrades and fresh installs of Cobia. My question is, why is this still being pushed and Bluefin pretty much dead in the water, even though it's the stable version?
Am I wrong on this? Would love to see some security updates to Bluefin so I feel a little better staying on that version until I get better warm-fuzzies about Cobia's stability and ability to be upgraded/migrated to.
FWIW, I run a couple of non-critical VMs and a couple of Apps, including my Frigate NVR, but more importantly this is my iSCSI target for my hypervisor cluster, so can't have it not being stable.

 

[ABain]

One of the big challenges is that the forums and reports rarely have information on the successful systems running out there (it does happen and we love to see these). Cobia today has >50K deployments active. BlueFin today has < 40K and falling, with users actively migrating to Cobia or DragonFish BETA.

Regarding the security updates, the reality is that many are just not available for the earlier versions of components that are in BlueFin.

On the risk of upgrading, you can roll back from Cobia to BlueFin by having a copy of your BlueFin boot environment (make sure the keep flag is set), see this link on managing boot environments.

We highly recommend downloading a copy of your config file before upgrading and it is essential you do not upgrade pools post-upgrade (as this would render them incompatible with earlier releases).

Note: As you have Apps installed and the upgrade from BlueFin to Cobia for these cannot be undone by just rolling back to an earlier release boot environment, you would need to follow this guide to ensure you can recover these in BlueFin should you need to rollback.

 

[jlw52761]

Absolutely agree on the notion of not enough reporting of the good installs, that definitely is a challenge for any systems or software developer, and I do apologize for coming of strong in my initial post. My concern is with the amount of dead in the water systems after the upgrade being reported. The posts I was reading on the failures seem to be pretty fundamental things that I wouldn't expect to fail as part of an upgrade, like fail to boot and such. Granted, there's no mention of the hardware, if the firmware is up to date, etc, but Linux is usually pretty forgiving on these matters.

I don't have a separate TrueNAS system to replicate to yet, that's a month or more out, and thankfully the configuration of the Apps isn't terribly concerning after I thought more about it because the data and configurations of the apps in the Apps is stored in their own datasets. So possibly a screenshot to document any custom settings in the App would suffice, but I may also be able to spin up a small virtualized TrueNAS on my ESXi environment that can take the replication jobs of the Apps.

I'm assuming there's no issues with rollback in regards to the VMs running on the system?

As far as boot environments, oh yeah, you bet there will be a snapshot before any upgrades.

As for the config backup, not sure I know where to grab that from to store off the system in case of catastrophic failure.

 

[PhilD13]

I have a Bluefin system and a Cobia system. Both are in 24/7/365 use and after setup are stable. The Bluefin (main) system works fine for what I'm using it for which is daily computer backups, a couple apps, and serving a few files. The Cobia (backup) system did not have any install issues on old hardware but has given some compatibility (they changed permissions/users) issues between Cobia and Bluefin systems.

Mileage may vary, but be aware that I was unable to setup replication between the Bluefin and Cobia systems. There is documentation and a guide within the documentation pages on how to setup the replication between Bluefin and Cobia but when I did the install, I ran out of time to troubleshoot it and so I punted and went with a Rsync task which works fine for what I need. I would suggest reading the documentation on replication between Cobia and Bluefin before attempting to set it up.

I don't have any VM's so I can't say on the rollbacks how they would be handled, but I would think cloning it to a regular data dataset for safekeeping may work if a rollback fails?

As far as updates, I have updated a Truenas VM running on Virtualbox through several versions of Bluefin to Cobia versions and now Dragonfish Beta and the upgrade each time has worked fine.

 

[ABain]

'm assuming there's no issues with rollback in regards to the VMs running on the system?

As far as boot environments, oh yeah, you bet there will be a snapshot before any upgrades.

As for the config backup, not sure I know where to grab that from to store off the system in case of catastrophic failure.

So, on the VMs, I'm going to take a look at this today as this has piqued my interest. I expect this to work, but as I myself have never done this, I want to find out.

On where to get the config backup System > General > Save Config:

 

[PhilD13]

On Bluefin and Cobia the system config is at: System Settings > General > Manage Configuration > Download File
The Save Configuration popup dialog will display, Make the desired selections as instructed in the dialog and save it to your computer.

 

[jlw52761]

Currently running this on a Dell PowerEdge R530 with up to date firmware, so hopefully everything's on the ACL for the Linux kernel and distro being used in Scale.

I do have my old system laying on a desk ready to be dismantled, I may break out that old beast as a replication target and test the upgrade there, then use it as a hot backup. It's a beast for sure, being an "old" M2400 Avamar Storage Node, but it's got 12 drive slots so there's that.

That's going to take some time and effort to get setup, even without the drives that's 70lbs of iron to sling, then add the drives and well, it's a back breaker. Might need to think of some other lightweight alternative, but if the upgrade goes sideways and I can't boot back into Bluefin, which there's been some of those posts, well a rebuild will hurt, but the boot pool is on separate disks from the data pool, so in theory I should be able to get the data pool imported back and restore the apps pool. I will just need to keep this system on Bluefin.

May even be able to add a couple of drives to my ESXi host as local storage and run a small virtual TrueNAS system there to address the whole Apps issue.

After all that babbling, here's a question. If I have to boot back to Bluefin after a failed upgrade in Cobia, since my data and app config is on their own datasets, can't I just recreate the apps from hand? I'm assuming the config backup doesn't have that information in it. The only outstanding question is the VM's, and I may be able to reduce the amount of those by shifting some services around...

 

[ABain]

Currently running this on a Dell PowerEdge R530 with up to date firmware, so hopefully everything's on the ACL for the Linux kernel and distro being used in Scale.

I do have my old system laying on a desk ready to be dismantled, I may break out that old beast as a replication target and test the upgrade there, then use it as a hot backup. It's a beast for sure, being an "old" M2400 Avamar Storage Node, but it's got 12 drive slots so there's that.

That's going to take some time and effort to get setup, even without the drives that's 70lbs of iron to sling, then add the drives and well, it's a back breaker. Might need to think of some other lightweight alternative, but if the upgrade goes sideways and I can't boot back into Bluefin, which there's been some of those posts, well a rebuild will hurt, but the boot pool is on separate disks from the data pool, so in theory I should be able to get the data pool imported back and restore the apps pool. I will just need to keep this system on Bluefin.

May even be able to add a couple of drives to my ESXi host as local storage and run a small virtual TrueNAS system there to address the whole Apps issue.

After all that babbling, here's a question. If I have to boot back to Bluefin after a failed upgrade in Cobia, since my data and app config is on their own datasets, can't I just recreate the apps from hand? I'm assuming the config backup doesn't have that information in it. The only outstanding question is the VM's, and I may be able to reduce the amount of those by shifting some services around...

The VMs should be fine.
for the Apps here is the link to the steps to take before upgrading to ensure you can restore them if you roll back:

23.10 (Cobia) Upgrades

Overview and processes for upgrading from earlier SCALE major versions and from 23.10 to newer major versions.

www.truenas.com

 

[jlw52761]

The VMs should be fine.
for the Apps here is the link to the steps to take before upgrading to ensure you can restore them if you roll back:

23.10 (Cobia) Upgrades

Overview and processes for upgrading from earlier SCALE major versions and from 23.10 to newer major versions.

www.truenas.com

So I really only need to focus on the ix-applications data and getting those replicated somewhere else, everything else should be good to go if I have to boot back into Bluefin? That does make some things a little easier I suppose. Worst case, I got it down to a couple of apps that should be easy to recreate since I have the actual application configs and data on different datasets outside of the ix-applications dataset.

 

[billbillw]

I did the upgrade this morning and it was, as far as I can tell, flawless. No issues to speak of. Mine is a relatively simple setup one SSD pool for Apps and a Storage Pool. I only run a handful of apps. I mainly use it for SMB and Plex. After the upgrade, I took a few minutes to upgrade the apps and that went without problem as well.

I went from the last Bluefin to Cobia 23.10.2. It took no more than 15 minutes for everything to come back up.

 

[Dave41]

I had been using Pi-Hole on Truenas Scale for about a year until upgrading from Bluefin to Cobia (Version:23.10.2) in February. I struggled some with Nextcloud and Syncthing after the upgrade but got both of them running without reinstalling them. Pi-Hole (TrueCharts version) on the other hand did not survive. Other apps will install OK, but since the upgrade TrueCharts and TrueNas versions of Pi-Hole will not install they both fail with the following cryptic error message.
[EFAULT] Failed to install App: Error: INSTALLATION FAILED: create: failed to create: secrets "sh.helm.release.v1.pihole.v1" is forbidden: unable to create new content in namespace ix-pihole because it is being terminated
Anyone have any ideas how to FIX this error???

 

[LarsR]

Sounds like your app is stuck in limbo... check with k3s kubectl get pods -A if it really is stuck. If it is stuck you can try to delete it with sudo k3s kubectl delete ns ix-appname (replace appname with the name of your pihole app) and then try installing it again. It could be that you have to press save twice during the app creation

 

[Dave41]

I ran "k3s kubectl get pods -A"and the last three lines that appeared seemed to be Pi-Hole related. I then tried try to delete it with "sudo k3s kubectl delete ns ix-pihole" and received namespace "ix-pihole" deleted not sure if it did anything as the pods display still has the same three lines at the bottom and the app still fails to install with the same old message as above.

 

[jlw52761]

I went ahead with the upgrade and it took a couple of reboots before the apps settled themselves out. I am noticing EVERYTHING is slow, much slower than on Bluefin, including iSCSI.

Not very happy with that I can say, and no, the CPU's are not pegged at all, but my disks all got reordered so that's not very helpful. My SLOG used to be sdh, now it's sdd, so had to update all my documentation on that one. ZFS cache is maxed out at 63.9GB used, so not thinking that's the issue. I have noticed that my daily scrub went from just a few minutes to ~10 hours, so that's probably a major factor in the slowness. Wonder why the scrub takes so much longer now?

Honestly, kinda wished I would've stayed on Bluefin at this particular moment. I lost my vCenter as part of the upgrade because of the slowness and iSCSI timeouts that occurred.

 

[PhilD13]

It's why I said earlier your mileage may vary. Some stuff works okay in Cobia and some stuff not so much. I think when Dragonfish is officially released and you upgrade to it, you will find many issues present in Cobia have gone away and things work better and smoother. The RC release works and performs much better than the latest Cobia does and various reports around confirm that. At least on my limited testing install in Virtualbox I upgraded it from Cobia to Dragonfish-RC and it works much better than it does using Cobia.

I'm planning on upgrading my Bluefin server over to Dragonfish once the release is proven to not have any major issues after release. My Cobia will be getting updated as well and probably sooner than the Bluefish server.