7TB of data deleted by freenas-update

[sadai]

I did a freenas-update on cli because I needed to see what is going on and relocated the download directory via '-C /mnt/data/share/' to my data directory with subdirectories containing about 7 TB of data.
After the reboot my SMB share was not available anymore (which was the above specified directory), now it seems as the updater did something like rm -rf '/mnt/data/share' to cleanup the update files but I cannot imagine it would do it in such a stupid way. What else could have happened?

Update was from version 9.10.2-U2 to U3

 

[Robert Trevellyan]

What else could have happened?

There might be a new dataset mounted in place of the original folder.

 

[SweetAndLow]

Output of zfs list?

Sent from my Nexus 5X using Tapatalk

 

[sadai]

Code:

[root@freenas] ~# zfs list
NAME														 USED  AVAIL  REFER  MOUNTPOINT
data														3.80G  7.65T  1.32G  /mnt/data
data/.system												38.1M  7.65T   279K  legacy
data/.system/configs-06013090bca44255afe5eba672a60979		570K  7.65T   570K  legacy
data/.system/cores										  22.2M  7.65T  22.2M  legacy
data/.system/rrd-06013090bca44255afe5eba672a60979			209K  7.65T   209K  legacy
data/.system/samba4										 5.02M  7.65T  5.02M  legacy
data/.system/syslog-06013090bca44255afe5eba672a60979		9.88M  7.65T  9.88M  legacy
data/jails												  2.41G  7.65T   273K  /mnt/data/jails
data/jails/.warden-template-pluginjail					   896M  7.65T   893M  /mnt/data/jails/.warden-template-pluginjail
data/jails/.warden-template-pluginjail-9.3-x64			   822M  7.65T   818M  /mnt/data/jails/.warden-template-pluginjail-9.3-x64
data/jails/owncloud_1										751M  7.65T  1.60G  /mnt/data/jails/owncloud_1
freenas-boot												1.28G  2.28G	31K  none
freenas-boot/ROOT										   1.25G  2.28G	25K  none
freenas-boot/ROOT/9.10.2-U3								 1.24G  2.28G   639M  /
freenas-boot/ROOT/FreeNAS-4368ef3bd4b6c54d8c8fa7f13a9db6b0  1.27M  2.28G   637M  /
freenas-boot/grub										   26.7M  2.28G  6.34M  legacy

Code:

[root@freenas] ~# zpool history
History for 'data':
....
2017-04-23.19:26:51 zpool import -o cachefile=none -R /mnt -f data
2017-04-23.19:26:51 zpool set cachefile=/data/zfs/zpool.cache data
2017-04-23.19:31:15 zpool import -o cachefile=none -R /mnt -f data
2017-04-23.19:31:15 zpool set cachefile=/data/zfs/zpool.cache data
2017-04-23.19:54:15 zfs set org.freenas:description= data
2017-04-23.19:54:16 zfs inherit compression data
2017-04-23.19:54:16 zfs inherit atime data
2017-04-23.19:54:16 zfs inherit dedup data
2017-04-23.19:54:16 zfs set reservation=0 data
2017-04-23.19:54:16 zfs set refreservation=0 data
2017-04-23.19:54:16 zfs set quota=none data
2017-04-23.19:54:16 zfs set refquota=none data
2017-04-23.19:54:21 zfs set aclmode=passthrough data
2017-04-23.20:31:07 zpool import -o cachefile=none -R /mnt -f data
2017-04-23.20:31:07 zpool set cachefile=/data/zfs/zpool.cache data
...
History for 'freenas-boot':
...
2017-04-23.18:50:19 zfs destroy -r freenas-boot/ROOT/FreeNAS-9.3-STABLE-201511040813
2017-04-23.18:50:27 zfs destroy freenas-boot/ROOT/FreeNAS-4368ef3bd4b6c54d8c8fa7f13a9db6b0@2017-04-14-11:42:15
2017-04-23.18:51:20 zfs destroy -r freenas-boot/ROOT/FreeNAS-9.3-STABLE-201604150515
2017-04-23.18:51:26 zfs destroy freenas-boot/ROOT/FreeNAS-4368ef3bd4b6c54d8c8fa7f13a9db6b0@2017-04-14-12:17:48
2017-04-23.18:53:07 zfs snapshot -r freenas-boot/ROOT/FreeNAS-4368ef3bd4b6c54d8c8fa7f13a9db6b0@2017-04-23-18:53:02
2017-04-23.18:53:11 zfs clone -o canmount=off -o mountpoint=/ freenas-boot/ROOT/FreeNAS-4368ef3bd4b6c54d8c8fa7f13a9db6b0@2017-04-23-18:53:02 freenas-boot/ROOT/9.10.2-U3
2017-04-23.18:53:25 zfs set beadm:nickname=9.10.2-U3 freenas-boot/ROOT/9.10.2-U3
2017-04-23.19:08:57 zfs set canmount=noauto freenas-boot/ROOT/9.10.2-U3
2017-04-23.19:08:58 zfs set mountpoint=/tmp/BE-9.10.2-U3.Hl8AyZxT freenas-boot/ROOT/9.10.2-U3
2017-04-23.19:09:04 zfs set mountpoint=/ freenas-boot/ROOT/9.10.2-U3
2017-04-23.19:09:07 zpool set bootfs=freenas-boot/ROOT/9.10.2-U3 freenas-boot
2017-04-23.19:09:12 zfs set canmount=noauto freenas-boot/ROOT/FreeNAS-4368ef3bd4b6c54d8c8fa7f13a9db6b0
2017-04-23.19:09:16 zfs set canmount=noauto freenas-boot/ROOT/Initial-Install
2017-04-23.19:09:18 zfs set canmount=noauto freenas-boot/ROOT/default
2017-04-23.19:09:22 zfs promote freenas-boot/ROOT/9.10.2-U3
2017-04-23.19:34:46 zfs destroy -r freenas-boot/ROOT/default
2017-04-23.19:34:58 zfs destroy freenas-boot/ROOT/9.10.2-U3@2015-11-15-17:28:29
2017-04-23.19:35:22 zfs destroy -r freenas-boot/ROOT/Initial-Install
2017-04-23.19:35:38 zfs destroy freenas-boot/ROOT/9.10.2-U3@2015-11-04-21:42:53

I started the update at about 18:50 and lasted until ~19:10 which is suspicious, does anybody know how long this update takes under normal conditions (from webgui)?
I think it hang most of the time after "Activated successfully" (which is 'beadm activate'), but wasn't able to find out what is happening after this step.

Many thanks for your help.

 

[Vito Reiter]

The data was definitely deleted it looks like, I'm unsure why you would set the update directory for the operating system on your data volume, and more unsure why you wouldn't make it its own directory. Now, this is assuming that your total amount of space is 7.65T~ if you have like 15T that should be available then the data may still be there..? I would open up a shell and see if you can't navigate to that directory and find the data. Doing what you did was extremely dangerous, could you elaborate on why you wanted to do it that way anyway?

Edit Note: Redundancy is NOT a backup.

 

[PhilipS]

I know this doesn't help now, but for others and the future, having snapshots set up would have allowed you to rollback.

 

[sadai]

My initial problem was not enough space on the flash drive, so I assumed I can save some space by relocating the update files with the -C option "cache_dir" which doesn't sound dangerous, just another directory to put the update files ...
I then found out that I need to delete some boot entries but used the same command again because I already downloaded the files instead using the webgui ...
In my opinion -C should be renamed/ get another description what it is actually doing and if it deletes files then only the ones that were downloaded and not the whole parent folder! Or even better always create a subdirectory in the specified folder. I still have no idea why this was dangerous.

Doing cd /mnt/data/share gives the expected result: No such file or directory.

I accepted the risk of a hardware failure but wasn't aware that an update can delete all my data :(

 

[Vito Reiter]

I did a freenas-update on cli because I needed to see what is going on and relocated the download directory via '-C /mnt/data/share/' to my data directory with subdirectories containing about 7 TB of data.

This MAY (and I don't know for sure) have worked if you made a new directory in your volume specified for that purpose, instead, you directly used a parent directory. It sucks that you made that mistake buddy but it happened, in the future have backups, snapshots, etc. and remember that this could've been avoided with a $10 pack of 3 8GB flash drives which I'm sure would've been worth saving your data. But I'm positive it got deleted because you specified a main parent directory as the cache_dir and not it's own location.

 

[sef]

But I'm positive it got deleted because you specified a main parent directory as the cache_dir and not it's own location.

That is correct. The -C option tells it what directory to store the downloaded files in, and then it cleans up after a successful update. It doesn't keep track of which files were downloaded because it can get interrupted and restarted.

If you had periodic snapshots, then you can just rollback from that.

Also, if you're so low on disk space on your boot devices, you probably want to move the system dataset to a data pool (and the default location for the update downloads is /var/db/system/update).

I generally use freenas-update -v -C /tmp/update check.

 

[anodos]

That is correct. The -C option tells it what directory to store the downloaded files in, and then it cleans up after a successful update. It doesn't keep track of which files were downloaded because it can get interrupted and restarted.

If you had periodic snapshots, then you can just rollback from that.

Also, if you're so low on disk space on your boot devices, you probably want to move the system dataset to a data pool (and the default location for the update downloads is /var/db/system/update).

I generally use freenas-update -v -C /tmp/update check.

It doesn't sound like setting the cache directory as somewhere in the pool makes sense. Perhaps there needs to be additional sanity checking in freenas-update.

 

[sef]

Many command-line tools let you shoot yourself in the foot. This is one of them.