Datasets gets rm'ed when changing anything related to "System Dataset" on reboot

[marcusorwen]

Hi,
Been having no problems for years but now just recently when i started to upgrade my system with more storage and then ultimately moving my system dataset and everything else from the smaller volumes to the new big one i have encountered some weird issues whenever i try to change the location of the system dataset or any of its settings (like syslog/collectd dataset) on reboot the system starts to act up and starts to remove all my data on the volume that is hosting the dataset before (even if the volume is the same volume).

My system specs are as follows:
Build: FreeNAS-9.10.1 (d989edd)
Platform: Intel(R) Xeon(R) CPU X3470 @ 2.93GHz
Memory: 16346MB DDR3 ECC
Drives: 3x2TB WD Red, 5x3TB WD Red and 1 WD Blue 320GB (currently only for storing temporary data).

This last time i was lucky enough to stop the remove process from running all the way through during reboot and restore my configuration backup before i changed the system dataset setting (this time only enabling logging on the dataset). but i lost all my jails (currently in the process of recreating them again), and some of my data but luckily the snapshots are still there so i have gotten all that back.

Any assistance with this is much appreciated (even if it needs me to reinstall and reconfigure the whole system from the ground up), i have not found anything related to this before hence why I am making this post.

 

[Sakuru]

Go to System --> Update, and run Verify Install
Also, please post the output of zpool history in [CODE] tags.

 

[marcusorwen]

Code:

The following Inconsistencies were found in your Current Install:

List of Files/Directories/Symlinks not Found:

/compat/linux/proc/.donotremove
List of Permission Errors:

/compat/linux/proc Expected MODE: 0755, Got: 0555

im having issues posting the zpool history log in the post (its 6000 lines long but most of it is snapshot information), so i put it in a file attatchment instead.

 

[Sakuru]

The following Inconsistencies were found in your Current Install: List of Files/Directories/Symlinks not Found: /compat/linux/proc/.donotremove List of Permission Errors: /compat/linux/proc Expected MODE: 0755, Got: 0555

Ok, that's good. Those 2 errors are normal for 9.10.1.

I don't see anything too strange in your zpool history. I see that you created a .system dataset in Pingu then deleted it about 9 hours later.

Code:

2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system
2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system/cores
2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system/samba4
2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system/syslog-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.00:03:20 zfs create -o mountpoint=legacy Pingu/.system/rrd-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.00:03:25 zfs create -o mountpoint=legacy Pingu/.system/configs-7f4d67ae16c94917b949456bb9f364ad

2016-08-07.09:44:58 zfs destroy Pingu/.system/configs-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.09:45:07 zfs destroy Pingu/.system/cores
2016-08-07.09:45:16 zfs destroy Pingu/.system/rrd-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.09:45:23 zfs destroy Pingu/.system/samba4
2016-08-07.09:45:33 zfs destroy Pingu/.system/syslog-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.09:45:40 zfs destroy Pingu/.system

starts to remove all my data on the volume

Are you saying that FreeNAS starts deleting everything in Pingu when you reboot?

 

[marcusorwen]

Ok, that's good. Those 2 errors are normal for 9.10.1.

I don't see anything too strange in your zpool history. I see that you created a .system dataset in Pingu then deleted it about 9 hours later.

Code:

2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system
2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system/cores
2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system/samba4
2016-08-07.00:03:19 zfs create -o mountpoint=legacy Pingu/.system/syslog-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.00:03:20 zfs create -o mountpoint=legacy Pingu/.system/rrd-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.00:03:25 zfs create -o mountpoint=legacy Pingu/.system/configs-7f4d67ae16c94917b949456bb9f364ad

2016-08-07.09:44:58 zfs destroy Pingu/.system/configs-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.09:45:07 zfs destroy Pingu/.system/cores
2016-08-07.09:45:16 zfs destroy Pingu/.system/rrd-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.09:45:23 zfs destroy Pingu/.system/samba4
2016-08-07.09:45:33 zfs destroy Pingu/.system/syslog-7f4d67ae16c94917b949456bb9f364ad
2016-08-07.09:45:40 zfs destroy Pingu/.system

Are you saying that FreeNAS starts deleting everything in Pingu when you reboot?

That must have been when i swapped the dataset back and forth between the volume that nolonger exist on the system (i had a old volume named Shadow before that i wanted to get rid of and moved the dataset over to Pingu).

The wierd thing is everything is fine on any other volume that dosent contain the system dataset if i change any setting.

But as soon as i change anything (just has to be checking a box or unchecking) the dataset that is the source gets deleted, not just the .system one but also all files in any other dataset on that volume. (i can see it trying to delete jails that are running and getting permission denied on certain run folders), i do not know what is causing the system to try and remove everything, also something to note is that whenever i try and change anything it dosent detect /var/log during the reboot so that is also something that is really wierd (maybe not related to this tho, but it has something to do with the system dataset atleast).

But it does not happen if i do not touch anything related to system dataset, then i can reboot just fine and nothing gets removed.

 

[marcusorwen]

I have one question, i know what cores, rrd-xxxxxx..., samba4 and syslog-xxxx... is but what is configs-xxxx....?

Cause im leaning towards just removing the system dataset reinstall freenas and import my volumes and reconfigure freenas from scratch and only keeping data and jails.

Or is there a recommended way of doing this?

 

[danb35]

what is configs-xxxx....?

FreeNAS keeps backups of your configuration database automatically, and that's where it stores them.

 

[marcusorwen]

FreeNAS keeps backups of your configuration database automatically, and that's where it stores them.

Thanks well then i know, that should really be stated more clearly in the docs what the different datasets are used for, or atleast that it stores backup configurations aswell.

Then i should be fine removing them and reinstalling FreeNAS and not restoring a configuration backup and then import the volumes and reconfiguring the jails location (so i would keep everything except like freenas specific settings)?