Rolling environment back, loses certain configuration stuff

[BloodyIron]

Okay so some of you may be familiar with my 9.3->11.1-U4 (and 9.10) stability issues. So I rolled back to 9.3 which has been super stable for me (until I figure out why it's unstable to begin with).

But the problem is, rolling back, I lost a bunch of configs that are... actually important. I don't yet know the full extent of what I've lost, so I'm going to list a few I have seen thus far. I wasn't sure if I should put this in the bug tracker, or how to file it, so I am posting on the forums first to get some advice on if I should file it, and how.

Okay, here's what I've lost so far (btw when I upgraded from 9.3->11.1-U4 these configs were kept, I lost them when I rolled back the boot environment):

1. My LAGG configuration is completely wiped away (this is annoying).
2. At least one local user was removed (which I used for an SMB share auth).
3. An SMB share was completely removed, but the dataset it was pointing to is still there (as it should be).

Now, my understanding is I should not have lost these configuration facets. So, what's up with that?

Please advise as to whether I should report this as a bug or not, and how, etc. Since I know 9.3 is not in main support.

 

[Ericloewe]

That's... weird. Your old environment should be exactly as it was before the update. Did you import a config file, by any chance?

If not, backup the config as it is now and go spelunking in the system dataset to see if you have any 9.3 configs in there and try them out.

 

[BloodyIron]

No, I didn't import any config files. My understanding was the config wouldn't be changed.

I have the System Dataset as one of the pools. I'll have to check in there...

Are configs periodically saved to the System Dataset? Or are such config files generated when updates are applied? I cann't seem to find any config files in the System Dataset so far :( (I take it, I'm supposed to be looking in the ".freenas" folder and subfolders? That is where I looked)

That's... weird. Your old environment should be exactly as it was before the update. Did you import a config file, by any chance?

If not, backup the config as it is now and go spelunking in the system dataset to see if you have any 9.3 configs in there and try them out.

 

[Ericloewe]

Are configs periodically saved to the System Dataset?

Yes.

I cann't seem to find any config files in the System Dataset so far :( (I take it, I'm supposed to be looking in the ".freenas" folder and subfolders? That is where I looked)

I don't know where exactly, but searching the forums is certain to yield the answer.

 

[BloodyIron]

Yeah not finding any .db files. This is concerning :/

Checked in the .freenas folder with a grep recursive command, no go. And that's on the pool that FreeNAS is set to use for System Dataset.

Yes.


I don't know where exactly, but searching the forums is certain to yield the answer.

 

[MrToddsFriends]

Yeah not finding any .db files. This is concerning :/

Maybe the following information is of help, gathered on a system currently running FreeNAS-11.1-U4.

Showing the "configs mountpoint" (containing a longish system specific hex id):

Code:

~ # mount | grep configs
volvar/.system/configs-f36704f2fe794cb6a75657843255655d on /var/db/system/configs-f36704f2fe794cb6a75657843255655d (zfs, local, nfsv4acls)

Listing my "configs folder", showing many subfolders (together with some history of my system) which in turn contain daily backups of .db files

Code:

~ # ls -al /var/db/system/configs-f36704f2fe794cb6a75657843255655d/
total 277
drwxr-xr-x  42 root  wheel  42 Mar 24 03:45 .
drwxr-xr-x  10 root  wheel  13 Apr 26 01:26 ..
drwxr-xr-x   2 root  wheel  21 Jul  4  2017 FreeNAS-11.0-RELEASE (a2dc21583)
drwxr-xr-x   2 root  wheel  19 Jul 21  2017 FreeNAS-11.0-U1 (aa82cc58d)
drwxr-xr-x   2 root  wheel  49 Sep  6  2017 FreeNAS-11.0-U2 (e417d8aa5)
drwxr-xr-x   2 root  wheel  22 Sep 26  2017 FreeNAS-11.0-U3 (c5dcf4416)
drwxr-xr-x   2 root  wheel  15 Dec 23 03:45 FreeNAS-11.0-U4 (54848d13b)
drwxr-xr-x   2 root  wheel  28 Jan 18 03:45 FreeNAS-11.1-RELEASE (dc7d195f4)
drwxr-xr-x   2 root  wheel  36 Feb 21 03:45 FreeNAS-11.1-U1 (f7e246b8f)
drwxr-xr-x   2 root  wheel  30 Mar 21 03:45 FreeNAS-11.1-U2 (c636d1f4b)
drwxr-xr-x   2 root  wheel   4 Mar 23 03:45 FreeNAS-11.1-U3 (bbd4bcb95)
drwxr-xr-x   2 root  wheel  36 Apr 26 03:45 FreeNAS-11.1-U4 (89e3d93bc)
drwxr-xr-x   2 root  wheel   6 Mar 26  2016 FreeNAS-9.10-RELEASE (2def9c8)
drwxr-xr-x   2 root  wheel  19 Apr 12  2016 FreeNAS-9.10-STABLE-201603252134 (412fb1c)
drwxr-xr-x   2 root  wheel   5 Apr 15  2016 FreeNAS-9.10-STABLE-201604111739 (896cc83)
drwxr-xr-x   2 root  wheel   6 Apr 19  2016 FreeNAS-9.10-STABLE-201604140622 (8781269)
drwxr-xr-x   2 root  wheel  10 Apr 27  2016 FreeNAS-9.10-STABLE-201604181743 (74ef270)
drwxr-xr-x   2 root  wheel   8 May  3  2016 FreeNAS-9.10-STABLE-201604261518 (881b70d)
drwxr-xr-x   2 root  wheel  24 May 25  2016 FreeNAS-9.10-STABLE-201605021851 (35c85f7)
drwxr-xr-x   2 root  wheel  16 Jun  8  2016 FreeNAS-9.10-STABLE-201605240427 (64fcd8e)
drwxr-xr-x   2 root  wheel  22 Jun 28  2016 FreeNAS-9.10-STABLE-201606072003 (696eba7)
drwxr-xr-x   2 root  wheel  39 Aug  4  2016 FreeNAS-9.10-STABLE-201606270534 (dd17351)
drwxr-xr-x   2 root  wheel  63 Oct  4  2016 FreeNAS-9.10.1 (d989edd)
drwxr-xr-x   2 root  wheel  47 Nov 18  2016 FreeNAS-9.10.1-U2 (f045a8b)
drwxr-xr-x   2 root  wheel  34 Dec 20  2016 FreeNAS-9.10.1-U4 (ec9a7d3)
drwxr-xr-x   2 root  wheel  25 Jan 12  2017 FreeNAS-9.10.2 (a476f16)
drwxr-xr-x   2 root  wheel  56 Mar  7  2017 FreeNAS-9.10.2-U1 (86c7ef5)
drwxr-xr-x   2 root  wheel  62 May  6  2017 FreeNAS-9.10.2-U2 (e1497f2)
drwxr-xr-x   2 root  wheel  22 May 26  2017 FreeNAS-9.10.2-U3 (e1497f269)
drwxr-xr-x   2 root  wheel  16 Jun  9  2017 FreeNAS-9.10.2-U4 (27ae72978)
drwxr-xr-x   2 root  wheel   8 Jun 15  2017 FreeNAS-9.10.2-U5 (561f0d7a1)
drwxr-xr-x   2 root  wheel   8 Sep 22  2015 FreeNAS-9.3-STABLE-201509160044
drwxr-xr-x   2 root  wheel   9 Sep 29  2015 FreeNAS-9.3-STABLE-201509220011
drwxr-xr-x   2 root  wheel  33 Oct 30  2015 FreeNAS-9.3-STABLE-201509282017
drwxr-xr-x   2 root  wheel   6 Nov  3  2015 FreeNAS-9.3-STABLE-201510290351
drwxr-xr-x   2 root  wheel   4 Nov  5  2015 FreeNAS-9.3-STABLE-201511020249
drwxr-xr-x   2 root  wheel  26 Nov 29  2015 FreeNAS-9.3-STABLE-201511040813
drwxr-xr-x   2 root  wheel  16 Dec 13  2015 FreeNAS-9.3-STABLE-201511280648
drwxr-xr-x   2 root  wheel  39 Jan 19  2016 FreeNAS-9.3-STABLE-201512121950
drwxr-xr-x   2 root  wheel  17 Feb  3  2016 FreeNAS-9.3-STABLE-201601181840
drwxr-xr-x   2 root  wheel   3 Feb  4  2016 FreeNAS-9.3-STABLE-201602020212
drwxr-xr-x   2 root  wheel  49 Mar 22  2016 FreeNAS-9.3-STABLE-201602031011

Listing all daily backup .db files from my currently used FreeNAS version (11.1-U4):

Code:

~ # ls -al "/var/db/system/configs-f36704f2fe794cb6a75657843255655d/FreeNAS-11.1-U4 (89e3d93bc)"
total 3978
drwxr-xr-x   2 root  wheel	  36 Apr 26 03:45 .
drwxr-xr-x  42 root  wheel	  42 Mar 24 03:45 ..
-rw-r-----   1 root  wheel  407552 Mar 24 03:45 20180324.db
-rw-r-----   1 root  wheel  407552 Mar 25 03:45 20180325.db
-rw-r-----   1 root  wheel  407552 Mar 26 03:45 20180326.db
-rw-r-----   1 root  wheel  407552 Mar 27 03:45 20180327.db
-rw-r-----   1 root  wheel  407552 Mar 28 03:45 20180328.db
-rw-r-----   1 root  wheel  407552 Mar 29 03:45 20180329.db
-rw-r-----   1 root  wheel  407552 Mar 30 03:45 20180330.db
-rw-r-----   1 root  wheel  407552 Mar 31 03:45 20180331.db
-rw-r-----   1 root  wheel  407552 Apr  1 03:45 20180401.db
-rw-r-----   1 root  wheel  407552 Apr  2 03:45 20180402.db
-rw-r-----   1 root  wheel  407552 Apr  3 03:45 20180403.db
-rw-r-----   1 root  wheel  407552 Apr  4 03:45 20180404.db
-rw-r-----   1 root  wheel  407552 Apr  5 03:45 20180405.db
-rw-r-----   1 root  wheel  407552 Apr  6 03:45 20180406.db
-rw-r-----   1 root  wheel  407552 Apr  7 03:45 20180407.db
-rw-r-----   1 root  wheel  407552 Apr  8 03:45 20180408.db
-rw-r-----   1 root  wheel  407552 Apr  9 03:45 20180409.db
-rw-r-----   1 root  wheel  407552 Apr 10 03:45 20180410.db
-rw-r-----   1 root  wheel  407552 Apr 11 03:45 20180411.db
-rw-r-----   1 root  wheel  407552 Apr 12 03:45 20180412.db
-rw-r-----   1 root  wheel  407552 Apr 13 03:45 20180413.db
-rw-r-----   1 root  wheel  407552 Apr 14 03:45 20180414.db
-rw-r-----   1 root  wheel  407552 Apr 15 03:45 20180415.db
-rw-r-----   1 root  wheel  407552 Apr 16 03:45 20180416.db
-rw-r-----   1 root  wheel  407552 Apr 17 03:45 20180417.db
-rw-r-----   1 root  wheel  407552 Apr 18 03:45 20180418.db
-rw-r-----   1 root  wheel  407552 Apr 19 03:45 20180419.db
-rw-r-----   1 root  wheel  407552 Apr 20 03:45 20180420.db
-rw-r-----   1 root  wheel  407552 Apr 21 03:45 20180421.db
-rw-r-----   1 root  wheel  407552 Apr 22 03:45 20180422.db
-rw-r-----   1 root  wheel  407552 Apr 23 03:45 20180423.db
-rw-r-----   1 root  wheel  407552 Apr 24 03:45 20180424.db
-rw-r-----   1 root  wheel  407552 Apr 25 03:45 20180425.db
-rw-r-----   1 root  wheel  407552 Apr 26 03:45 20180426.db

Finally counting all "*.db" files present in the "configs folder" with all of its subfolders

Code:

~ # find /var/db/system/configs-f36704f2fe794cb6a75657843255655d/ -name "*.db" | wc -l
	 878

 

[BloodyIron]

Dang there's configs in them thar folders!

Oddly enough, my 9.3 db config backups stop at May 23 2017.... Might be new enough, but I don't know why it stopped...

Thanks for the info! But any idea why it stopped backing them up?

I was going to restore from one of these, but May 23 2017 is quite old. I'd like to identify why it hasn't been backing up, or if there's another location, since I'd prefer not to lose config stuff. The user account I lost was made within the last month. D:

What if I restored a 9.10 or 11.1-U4 config that's newer? I haven't upgraded the ZFS version/features, or made any configs that are centric to 9.10 or 11.1-U4, so... I anticipate it should work?

Maybe the following information is of help, gathered on a system currently running FreeNAS-11.1-U4.

Showing the "configs mountpoint" (containing a longish system specific hex id):

Code:

~ # mount | grep configs
volvar/.system/configs-f36704f2fe794cb6a75657843255655d on /var/db/system/configs-f36704f2fe794cb6a75657843255655d (zfs, local, nfsv4acls)

Listing my "configs folder", showing many subfolders (together with some history of my system) which in turn contain daily backups of .db files

Code:

~ # ls -al /var/db/system/configs-f36704f2fe794cb6a75657843255655d/
total 277
drwxr-xr-x  42 root  wheel  42 Mar 24 03:45 .
drwxr-xr-x  10 root  wheel  13 Apr 26 01:26 ..
drwxr-xr-x   2 root  wheel  21 Jul  4  2017 FreeNAS-11.0-RELEASE (a2dc21583)
drwxr-xr-x   2 root  wheel  19 Jul 21  2017 FreeNAS-11.0-U1 (aa82cc58d)
drwxr-xr-x   2 root  wheel  49 Sep  6  2017 FreeNAS-11.0-U2 (e417d8aa5)
drwxr-xr-x   2 root  wheel  22 Sep 26  2017 FreeNAS-11.0-U3 (c5dcf4416)
drwxr-xr-x   2 root  wheel  15 Dec 23 03:45 FreeNAS-11.0-U4 (54848d13b)
drwxr-xr-x   2 root  wheel  28 Jan 18 03:45 FreeNAS-11.1-RELEASE (dc7d195f4)
drwxr-xr-x   2 root  wheel  36 Feb 21 03:45 FreeNAS-11.1-U1 (f7e246b8f)
drwxr-xr-x   2 root  wheel  30 Mar 21 03:45 FreeNAS-11.1-U2 (c636d1f4b)
drwxr-xr-x   2 root  wheel   4 Mar 23 03:45 FreeNAS-11.1-U3 (bbd4bcb95)
drwxr-xr-x   2 root  wheel  36 Apr 26 03:45 FreeNAS-11.1-U4 (89e3d93bc)
drwxr-xr-x   2 root  wheel   6 Mar 26  2016 FreeNAS-9.10-RELEASE (2def9c8)
drwxr-xr-x   2 root  wheel  19 Apr 12  2016 FreeNAS-9.10-STABLE-201603252134 (412fb1c)
drwxr-xr-x   2 root  wheel   5 Apr 15  2016 FreeNAS-9.10-STABLE-201604111739 (896cc83)
drwxr-xr-x   2 root  wheel   6 Apr 19  2016 FreeNAS-9.10-STABLE-201604140622 (8781269)
drwxr-xr-x   2 root  wheel  10 Apr 27  2016 FreeNAS-9.10-STABLE-201604181743 (74ef270)
drwxr-xr-x   2 root  wheel   8 May  3  2016 FreeNAS-9.10-STABLE-201604261518 (881b70d)
drwxr-xr-x   2 root  wheel  24 May 25  2016 FreeNAS-9.10-STABLE-201605021851 (35c85f7)
drwxr-xr-x   2 root  wheel  16 Jun  8  2016 FreeNAS-9.10-STABLE-201605240427 (64fcd8e)
drwxr-xr-x   2 root  wheel  22 Jun 28  2016 FreeNAS-9.10-STABLE-201606072003 (696eba7)
drwxr-xr-x   2 root  wheel  39 Aug  4  2016 FreeNAS-9.10-STABLE-201606270534 (dd17351)
drwxr-xr-x   2 root  wheel  63 Oct  4  2016 FreeNAS-9.10.1 (d989edd)
drwxr-xr-x   2 root  wheel  47 Nov 18  2016 FreeNAS-9.10.1-U2 (f045a8b)
drwxr-xr-x   2 root  wheel  34 Dec 20  2016 FreeNAS-9.10.1-U4 (ec9a7d3)
drwxr-xr-x   2 root  wheel  25 Jan 12  2017 FreeNAS-9.10.2 (a476f16)
drwxr-xr-x   2 root  wheel  56 Mar  7  2017 FreeNAS-9.10.2-U1 (86c7ef5)
drwxr-xr-x   2 root  wheel  62 May  6  2017 FreeNAS-9.10.2-U2 (e1497f2)
drwxr-xr-x   2 root  wheel  22 May 26  2017 FreeNAS-9.10.2-U3 (e1497f269)
drwxr-xr-x   2 root  wheel  16 Jun  9  2017 FreeNAS-9.10.2-U4 (27ae72978)
drwxr-xr-x   2 root  wheel   8 Jun 15  2017 FreeNAS-9.10.2-U5 (561f0d7a1)
drwxr-xr-x   2 root  wheel   8 Sep 22  2015 FreeNAS-9.3-STABLE-201509160044
drwxr-xr-x   2 root  wheel   9 Sep 29  2015 FreeNAS-9.3-STABLE-201509220011
drwxr-xr-x   2 root  wheel  33 Oct 30  2015 FreeNAS-9.3-STABLE-201509282017
drwxr-xr-x   2 root  wheel   6 Nov  3  2015 FreeNAS-9.3-STABLE-201510290351
drwxr-xr-x   2 root  wheel   4 Nov  5  2015 FreeNAS-9.3-STABLE-201511020249
drwxr-xr-x   2 root  wheel  26 Nov 29  2015 FreeNAS-9.3-STABLE-201511040813
drwxr-xr-x   2 root  wheel  16 Dec 13  2015 FreeNAS-9.3-STABLE-201511280648
drwxr-xr-x   2 root  wheel  39 Jan 19  2016 FreeNAS-9.3-STABLE-201512121950
drwxr-xr-x   2 root  wheel  17 Feb  3  2016 FreeNAS-9.3-STABLE-201601181840
drwxr-xr-x   2 root  wheel   3 Feb  4  2016 FreeNAS-9.3-STABLE-201602020212
drwxr-xr-x   2 root  wheel  49 Mar 22  2016 FreeNAS-9.3-STABLE-201602031011

Listing all daily backup .db files from my currently used FreeNAS version (11.1-U4):

Code:

~ # ls -al "/var/db/system/configs-f36704f2fe794cb6a75657843255655d/FreeNAS-11.1-U4 (89e3d93bc)"
total 3978
drwxr-xr-x   2 root  wheel	  36 Apr 26 03:45 .
drwxr-xr-x  42 root  wheel	  42 Mar 24 03:45 ..
-rw-r-----   1 root  wheel  407552 Mar 24 03:45 20180324.db
-rw-r-----   1 root  wheel  407552 Mar 25 03:45 20180325.db
-rw-r-----   1 root  wheel  407552 Mar 26 03:45 20180326.db
-rw-r-----   1 root  wheel  407552 Mar 27 03:45 20180327.db
-rw-r-----   1 root  wheel  407552 Mar 28 03:45 20180328.db
-rw-r-----   1 root  wheel  407552 Mar 29 03:45 20180329.db
-rw-r-----   1 root  wheel  407552 Mar 30 03:45 20180330.db
-rw-r-----   1 root  wheel  407552 Mar 31 03:45 20180331.db
-rw-r-----   1 root  wheel  407552 Apr  1 03:45 20180401.db
-rw-r-----   1 root  wheel  407552 Apr  2 03:45 20180402.db
-rw-r-----   1 root  wheel  407552 Apr  3 03:45 20180403.db
-rw-r-----   1 root  wheel  407552 Apr  4 03:45 20180404.db
-rw-r-----   1 root  wheel  407552 Apr  5 03:45 20180405.db
-rw-r-----   1 root  wheel  407552 Apr  6 03:45 20180406.db
-rw-r-----   1 root  wheel  407552 Apr  7 03:45 20180407.db
-rw-r-----   1 root  wheel  407552 Apr  8 03:45 20180408.db
-rw-r-----   1 root  wheel  407552 Apr  9 03:45 20180409.db
-rw-r-----   1 root  wheel  407552 Apr 10 03:45 20180410.db
-rw-r-----   1 root  wheel  407552 Apr 11 03:45 20180411.db
-rw-r-----   1 root  wheel  407552 Apr 12 03:45 20180412.db
-rw-r-----   1 root  wheel  407552 Apr 13 03:45 20180413.db
-rw-r-----   1 root  wheel  407552 Apr 14 03:45 20180414.db
-rw-r-----   1 root  wheel  407552 Apr 15 03:45 20180415.db
-rw-r-----   1 root  wheel  407552 Apr 16 03:45 20180416.db
-rw-r-----   1 root  wheel  407552 Apr 17 03:45 20180417.db
-rw-r-----   1 root  wheel  407552 Apr 18 03:45 20180418.db
-rw-r-----   1 root  wheel  407552 Apr 19 03:45 20180419.db
-rw-r-----   1 root  wheel  407552 Apr 20 03:45 20180420.db
-rw-r-----   1 root  wheel  407552 Apr 21 03:45 20180421.db
-rw-r-----   1 root  wheel  407552 Apr 22 03:45 20180422.db
-rw-r-----   1 root  wheel  407552 Apr 23 03:45 20180423.db
-rw-r-----   1 root  wheel  407552 Apr 24 03:45 20180424.db
-rw-r-----   1 root  wheel  407552 Apr 25 03:45 20180425.db
-rw-r-----   1 root  wheel  407552 Apr 26 03:45 20180426.db

Finally counting all "*.db" files present in the "configs folder" with all of its subfolders

Code:

~ # find /var/db/system/configs-f36704f2fe794cb6a75657843255655d/ -name "*.db" | wc -l
	 878

 

[MrToddsFriends]

Oddly enough, my 9.3 db config backups stop at May 23 2017.... Might be new enough, but I don't know why it stopped...

Unfortunately I have no idea. Maybe because you/someone reconfigured the system to store the .db files somewhere else at that point in time. Might have happened without knowing that .db restoration becomes more difficult after this change ...

 

[BloodyIron]

I'm the only person that has access to these systems. And I know for a fact no config restorations have happened in the last 2 years.

Unfortunately I have no idea. Maybe because you/someone reconfigured the system to store the .db files somewhere else at that point in time. Might have happened without knowing that .db restoration becomes more difficult after this change ...

 

[Stux]

Its possible that your config backup issues are related to your issues with the config not having all the config...

As far as I know, the backup environments are clones... and I'd expect that it should be exactly as it was when you upgraded. Of course, perhaps you've discovered a 9.3 bug. I doubt that'd be fixed ;)

 

[pschatz100]

I presume you don't back up your configuration when you back up your data? I back up my configuration before each update and also once a month to an external drive. That way, if I ever need to do a fresh install, I can simply restore the config. It saved me once when I was still booting off USB flash drives and one of them failed.

You might consider doing this after you get your system sorted out.

 

[BloodyIron]

When setting the System Dataset in the settings section, the configs are _supposed_ to get backed up every day, as per built-in FreeNAS functionality. However, it seemed to stop doing that at some point for a reason even the devs don't quite understand.

I presume you don't back up your configuration when you back up your data? I back up my configuration before each update and also once a month to an external drive. That way, if I ever need to do a fresh install, I can simply restore the config. It saved me once when I was still booting off USB flash drives and one of them failed.

You might consider doing this after you get your system sorted out.

 

[pschatz100]

Actually, I meant backing up the configuration to an external device. Just like making periodic backups of my data and saving to an external device, I make periodic backups of my configuration and save them. You do this from the General settings under the "System" tab. Once you do this, it is really easy to make a fresh install and then upload a saved configuration to get your system back.