Possible Boot Pool failure following upgrade?

[NumberSix]

Plugins and Jails require the "iocage" dataset to reside. Upon creation of your first Plugin or Jail, it will prompt you where you want to hold the iocage dataset.

My hunch tells you you chose (or left it at) "System". Because for all intents and purposes, the dedicated "System" pool you created is a data pool, according to TrueNAS. (In your setup, "System" is just the name you gave it, but from TrueNAS's perspective it's "data pool number two".)

I see.
Crazy last words of a desperate man then: This is a 'what if'.

What if I were to try to use the backup of the old system pool on the SSD (before the upgrade, that is, before it also contained the boot and log and swap) to over-write what's on the SSD now, if that's even possible, then to boot from the degraded but still oerational USB. Effectively, to roll back to where I was yesterday. Then I might be in a position to do 'something' to change how and where the iocage was, such that, when I then go through the transition to full SSD reliance again, I do so WITH my plugins intact?

Is that too insane to contemplate, or might there be a very workable plan in that?

 

[winnielinnie]

Why not just reconfigure a Plugin/Jail for Syncthing and Grafana? Technically, you could do a manual replication from your iocage backups (if they really exist), but it seems like too much dancing.

As for the "Syncthing" dataset you're seeing, that's a "data" dataset that your Syncthing plugin/jail was configured to use as a mount point for synchronizing files.

Think of a plugin/jail as its own server. The iocage dataset is where your "mini" servers all live. No important data should ever live inside there.

• A Plex jail? But important data (movies, music, photos, etc) are on an actual data dataset, that the Plex jail uses as a mount point to access it.
• A qBittorrent jail? But important data (torrents, downloads, etc) are on an actual data dataset, that the qBittorrent jail uses as a mount point to access it.
• A Syncthing jail? But important data (documents, multimedia, stuff to sync, etc) are on an actual data dataset, that the Syncthing jail uses as a mount point to access it.

The noteworthy stuff in a plugin/jail under iocage is the configuration and custom settings. It sucks to lose those, but you can include iocage in a backup/replication task as well.

 

[NumberSix]

Why not just reconfigure a Plugin/Jail for Syncthing and Grafana? Technically, you could do a manual replication from your iocage backups (if they really exist), but it seems like too much dancing.

Hi Winnie
That's another genius shortcut! Rather that the pain of trying to roll anything back, use the backup of the old 'iocage portion of' System'. Dancing? Would it be more complex than copying my backup of the iocage folder (which I just checked and I have it and everything under it) over the current 'live' iocage folder?

Also I was taking regular recursive Snapshots of the iocage (as with everything else) and I just checked; I have all those Sapshots too (although don't know if they will save me, or how to use them if they could).

Could restoring the iocage be so simple as over-writing a directory structure?! And/or doing something with the Snapshots??! I am so excited at the possibility!

Here's what I see of my NAS pool from Windows:

Notice "iocage" was created yesterday, after I moved all system related elements to the SSD. So that's TrueNAS's active iocage. Currently it contains no jails and no plugins. It just exists.

Now turning attention to the last item in the above list, 'System-Backups' is just that - my backups of everything from the former SSD. So if I drill down into it I see the following:

So, within 'System-Backups', I have 'iocage' and all it's associated sub-directories and files.

Can I copy the one folder over the other, and will that fix my loss? Can I do any damage to the wider system by attempting that? And lastly, if you can give that a green light, is there any special shell command I should use to do it, or can I copy A over B from Windows for the same result? Or can/should my Snapshots save me by some means?

I keenly await your thoughts on these two options! :)

Thank you Winnie.

 

[winnielinnie]

Hi Winnie
That's another genius shortcut! Rather that the pain of trying to roll anything back, use the backup of the old 'iocage portion of' System'. Dancing? Would it be more complex than copying my backup of the iocage folder (which I just checked and I have it and everything under it) over the current 'live' iocage folder?

Also I was taking regular recursive Snapshots of the iocage (as with everything else) and I just checked; I have all those Sapshots too (although don't know if they will save me, or how to use them if they could).

It might be safer to just create the new Plugins/Jails again, make sure they can run, and then finally copy the relevant config folders/files from the backup (using the Shell or command-line).

Each Plugin/Jail will be different. It depends where Syncthing and Grafana save their configs.

The "root" folder nested under a jail's exclusive dataset is where the filesystem tree starts. So for example, take qBittorrent.

The config folder is located here, which holds your settings for qBittorrent:
/mnt/poolname/iocage/jails/qbittorrent/root/var/db/qbittorrent

From within the jail, it appears as /var/db/qbittorrent

Not to mention the "service" files, which are usually created upon Plugin creation and/or package installation (if done manually.)

They might be under (from within the jail, remember):
/usr/local/etc/rc.d/
or
/etc/rc.d/



You also have to make sure you retain the ownership and permissions of the config files.

This is why it might be easier to take this as a lesson and create the Plugins/Jails and re-configure everything again.

 

[NumberSix]

Each Plugin/Jail will be different. It depends where Syncthing and Grafana save their configs.

Agreed. But since I'd be copying a working saved iocage structure over the current one, these two plugin's and the way they store their files, relative to the /iocage directory would be identical. Just because the new /iocage is located on my HDD and not the SDD, there's no reason why we should think the structural relativity of other files deeper down the tree than /iocage would change. Surely?

So, although I don't know where they store their respective configs, or their respective service files, I do know that the path from /iocage must logically be the same as it was when I last took a copy of /iocage, about a week ago. What am I missing here, that you disagree?

You also have to make sure you retain the ownership and permissions of the config files.

I was imagining that the backup-up /iocage copy would include storing the files and folders with it's original ownerships and permissions. Can I not assume that? Otherwise a backup copy is not truely a copy?

This is why it might be easier to take this as a lesson and create the Plugins/Jails and re-configure everything again.

(I can see that prospect looming on the horizon, of course, but not before I'm certain that the saved backups and snapshots I carefully squirreled away really have got no value to help me now I need them.)

 

[winnielinnie]

I was imagining that the backup-up /iocage copy would include storing the files and folders with it's original ownerships and permissions. Can I not assume that? Otherwise a backup copy is not truely a copy?

You implied you were going to do this over a network share (Windows, SMB), or maybe I read it wrong. As long as you do it via the Shell or SSH, you should be fine. But there's no reason to dump everything into place. Besides, I'm not sure they will register as active plugins, since there's a config that TrueNAS uses itself to keep track of your jails/plugins. It doesn't simply "scan" iocage for jails and plugins.

So, although I don't know where they store their respective configs, or their respective service files, I do know that the path from /iocage must logically be the same as it was when I last took a copy of /iocage, about a week ago. What am I missing here, that you disagree?

If you want, you can try to destroy the new (essentially empty) iocage dataset, and then do a full replication of the backed-up iocage to nest directly underneath your top-level root dataset ("NAS").

 

[NumberSix]

If you want, you can try to destroy the new (essentially empty) iocage dataset, and then do a full replication of the backed-up iocage to nest directly underneath your top-level root dataset ("NAS").

Thank you Winnie.
I might re-install but not configure those 2 plugins, just so TrueNAS knows I have them, then try over-writing the iocage (from shell) and see if that saves the day. Otherwise Plan B; delete the iocage data set and start over, reinstalling from scratch (Syncthing anyway. The version of Grafana included in the TrueNAS interface is much documented not to work, but some kind sould wrote a detailed 'How to' entry in this forum on installing it from an external source. I did, but wow was it complicated, so I might live without that one).

This experiment needs to wait for a week or so as I need to switch my attention elsewhere for now, but I'd love to get back to you with an update in due course. Thanks again Winnie. I do appreciate your support!!

 

[NumberSix]

If you want, you can try to destroy the new (essentially empty) iocage dataset, and then do a full replication of the backed-up iocage to nest directly underneath your top-level root dataset ("NAS").

Hi!
Back from my travels, I tried as you suggested, using the 'rsync' command. Actually I varied it by installing but not configuring the three lost plugins first, so that TrueNAS and any records it keeps about plugins other than in the jails, would be aware of what I was trying to set up. Then I used rsync to over-write the partly setup jails with my backups. Despite adding the -aR flags and even --delete to wipe out extraneous files in the target, after a very long write back process, it reported it had "completed with errors". Long story short, I had to delete the iocage and replace it with a new empty one. I doubt I have the energy to rebuild everything from scratch, plus my faith in the value of TrueNAS' own backups has been demonstrated to be misplaced, which is very regrettable as of course it strikes at the heart of TrueNAS's entire raison d'être.

Thank you for your keen help and support throughout though :)

 

[winnielinnie]

I tried as you suggested, using the 'rsync' command.

I never said anything about rsync.

I was suggesting that you try to replicate your backed-up iocage dataset to nest directly underneath your NAS top-level root dataset.

From post #26 above:

If you want, you can try to destroy the new (essentially empty) iocage dataset, and then do a full replication of the backed-up iocage to nest directly underneath your top-level root dataset ("NAS").

 

[NumberSix]

I never said anything about rsync.

I was suggesting that you try to replicate your backed-up iocage dataset to nest directly underneath your NAS top-level root dataset.

From post #26 above:

Ah sorry. I was speaking too loosely by far. I simply meant I tried to use the backup.