Strange disappearance of entire directory structure

Elo · Jun 19, 2017

I have recently lost an entire directory structure with thousands of files (Video Material). The system has

Code:

NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
OB-NAS-JAILS  3.62T   384G  3.25T		 -	 1%	10%  1.00x  ONLINE  /mnt
Pool_1		16.2T  2.56T  13.7T		 -	 4%	15%  1.00x  ONLINE  /mnt
Pool_2		19.0T  6.19T  12.8T		 -	 2%	32%  1.00x  ONLINE  /mnt
freenas-boot  7.44G  5.52G  1.91G		 -	  -	74%  1.00x  ONLINE  -

There is no no major problem with any of the pools. The Pool in question is Pool_2 which has a few major directories in it and in one of these all data (except 4 sub-directories) suddenly disappeared. The server is not accessible through SSH from the internet. The files in question is accessible through J River Media Center (MC) from the internet. If they had been deleted through this , then thy would also disappear from the MC media library (and they have not). I am the only one having the root PW and thus ability to enter through SSH on the LAN. The files can however be deleted as SMB files from a windows machine ( and I can not rule out that this has happened although it is very unlikely. I am running FreeNAS-9.10.2-U4. I am not in depth familiar with the file system and I hope someone can point me in the right direction by answering some of the following questions:

Is there a log (logs) where I can find out how these files was removed?
Is it possible or even likely the a power failure can lead to these problem (we had several ones some days ago. I have a UPS due to previous problems with power failure but I do not know the status of the battery or the UPS capability as surge protection)
If the data is still on the disks can I restore by:

# Zpool destroy Pool_2
# Zpool import -D Pool_2
Install a previous system configuration backup
Boot from a earlier boot point

I am sorry for these questions but I know from earlier experience that cut and try can destroy more than it helps when I do not have sufficient knowledge

fracai · Jun 19, 2017

If you have snapshots, you can restore quickly from those.
Can you also post the output of "zfs list"? maybe "zfs list -o space"

Elo · Jun 19, 2017

Hi... No snapshot I am afraid :-(
ZFS list as bellow in the second next posting!
zfs list -o space returns error and requires a defined property. Which one would help?

SweetAndLow · Jun 19, 2017

Elo said:
Hi... No snapshot I am afraid :-(
ZFS list as bellow
zfs list -o space returns error and requires a defined property. Which one would help?

[root@OB-NAS-MAIN] /mnt# zfs list
NAME USED AVAIL REFER MOUNTPOINT
OB-NAS-JAILS 384G 3.14T 375G /mnt/OB-NAS-JAILS
OB-NAS-JAILS/.system 489M 3.14T 402M legacy
OB-NAS-JAILS/.system/configs-a65c52f7cbea4095ad912fe7c84fed72 61.4M 3.14T 61.4M legacy
OB-NAS-JAILS/.system/cores 14.0M 3.14T 14.0M legacy
OB-NAS-JAILS/.system/rrd-a65c52f7cbea4095ad912fe7c84fed72 144K 3.14T 144K legacy
OB-NAS-JAILS/.system/samba4 708K 3.14T 708K legacy
OB-NAS-JAILS/.system/syslog-a65c52f7cbea4095ad912fe7c84fed72 10.1M 3.14T 10.1M legacy
OB-NAS-JAILS/.warden-template-pluginjail 469M 3.14T 469M /mnt/OB-NAS-JAILS/.warden-template-pluginjail
OB-NAS-JAILS/.warden-template-pluginjail--x64 449M 3.14T 449M /mnt/OB-NAS-JAILS/.warden-template-pluginjail--x64
OB-NAS-JAILS/.warden-template-pluginjail-9.2-x64 721M 3.14T 721M /mnt/OB-NAS-JAILS/.warden-template-pluginjail-9.2-x64
OB-NAS-JAILS/.warden-template-pluginjail-9.3-x64 449M 3.14T 449M /mnt/OB-NAS-JAILS/.warden-template-pluginjail-9.3-x64
OB-NAS-JAILS/.warden-template-standard 1.61G 3.14T 1.61G /mnt/OB-NAS-JAILS/.warden-template-standard
OB-NAS-JAILS/.warden-template-standard-9.3-x64 1.57G 3.14T 1.57G /mnt/OB-NAS-JAILS/.warden-template-standard-9.3-x64
OB-NAS-JAILS/.warden-template-standard-x86 205M 3.14T 205M /mnt/OB-NAS-JAILS/.warden-template-standard-x86
OB-NAS-JAILS/Owncloudjail 1.48G 3.14T 2.30G /mnt/OB-NAS-JAILS/Owncloudjail
OB-NAS-JAILS/maraschino_1 129M 3.14T 594M /mnt/OB-NAS-JAILS/maraschino_1
OB-NAS-JAILS/nextcloud 1.67G 3.14T 2.49G /mnt/OB-NAS-JAILS/nextcloud
OB-NAS-JAILS/oc-data 96K 3.14T 96K /mnt/OB-NAS-JAILS/oc-data
OB-NAS-JAILS/oc-db 1.68M 3.14T 1.68M /mnt/OB-NAS-JAILS/oc-db
Pool_1 1.71T 8.78T 816G /mnt/Pool_1
Pool_1/BTDownloads 53.7G 46.3G 53.7G /mnt/Pool_1/BTDownloads
Pool_1/db 137M 8.78T 137M /mnt/Pool_1/db
Pool_1/files 363G 8.78T 363G /mnt/Pool_1/files
Pool_1/ncdb 108M 8.78T 108M /mnt/Pool_1/ncdb
Pool_1/ncfiles 11.0G 8.78T 11.0G /mnt/Pool_1/ncfiles
Pool_1/nextcloud_db 264M 8.78T 264M /mnt/Pool_1/nextcloud_db
Pool_1/nextcloud_files 503G 8.78T 503G /mnt/Pool_1/nextcloud_files
Pool_2 6.19T 12.3T 6.19T /mnt/Pool_2
freenas-boot 5.52G 1.68G 31K none
freenas-boot/ROOT 5.35G 1.68G 25K none
freenas-boot/ROOT/9.10-STABLE-201606270534 14.9M 1.68G 609M /
freenas-boot/ROOT/9.10.1 38K 1.68G 630M /
freenas-boot/ROOT/9.10.1-U1 16.2M 1.68G 637M /
freenas-boot/ROOT/9.10.1-U2 122K 1.68G 641M /
freenas-boot/ROOT/9.10.1-U3 116K 1.68G 638M /
freenas-boot/ROOT/9.10.1-U4 17.6M 1.68G 639M /
freenas-boot/ROOT/9.10.2 20.7M 1.68G 656M /
freenas-boot/ROOT/9.10.2-U1 20.6M 1.68G 657M /
freenas-boot/ROOT/9.10.2-U2 22.1M 1.68G 659M /
freenas-boot/ROOT/9.10.2-U3 24.8M 1.68G 663M /
freenas-boot/ROOT/9.10.2-U4 5.18G 1.68G 663M /
freenas-boot/ROOT/FreeNAS-1ac5f24e172b4785efcab5401aa5507f 20.8M 1.68G 477M /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201604150515 12.7M 1.68G 538M /
freenas-boot/ROOT/Initial-Install 1K 1.68G 506M legacy
freenas-boot/ROOT/default 2.49M 1.68G 510M legacy
freenas-boot/grub

Use code tags when posting output, this is basically unreadable.

Sent from my Nexus 5X using Tapatalk

Elo · Jun 19, 2017

SweetAndLow said:
Use code tags when posting output, this is basically unreadable.

Sent from my Nexus 5X using Tapatalk

Sorry ... Hope this helps then:

Code:

NAME															USED  AVAIL  REFER  MOUNTPOINT
OB-NAS-JAILS													384G  3.14T   375G  /mnt/OB-NAS-JAILS
OB-NAS-JAILS/.system											489M  3.14T   402M  legacy
OB-NAS-JAILS/.system/configs-a65c52f7cbea4095ad912fe7c84fed72  61.4M  3.14T  61.4M  legacy
OB-NAS-JAILS/.system/cores									 14.0M  3.14T  14.0M  legacy
OB-NAS-JAILS/.system/rrd-a65c52f7cbea4095ad912fe7c84fed72	   144K  3.14T   144K  legacy
OB-NAS-JAILS/.system/samba4									 716K  3.14T   716K  legacy
OB-NAS-JAILS/.system/syslog-a65c52f7cbea4095ad912fe7c84fed72   10.0M  3.14T  10.0M  legacy
OB-NAS-JAILS/.warden-template-pluginjail						469M  3.14T   469M  /mnt/OB-NAS-JAILS/.warden-template-pluginjail
OB-NAS-JAILS/.warden-template-pluginjail--x64				   449M  3.14T   449M  /mnt/OB-NAS-JAILS/.warden-template-pluginjail--x64
OB-NAS-JAILS/.warden-template-pluginjail-9.2-x64				721M  3.14T   721M  /mnt/OB-NAS-JAILS/.warden-template-pluginjail-9.2-x64
OB-NAS-JAILS/.warden-template-pluginjail-9.3-x64				449M  3.14T   449M  /mnt/OB-NAS-JAILS/.warden-template-pluginjail-9.3-x64
OB-NAS-JAILS/.warden-template-standard						 1.61G  3.14T  1.61G  /mnt/OB-NAS-JAILS/.warden-template-standard
OB-NAS-JAILS/.warden-template-standard-9.3-x64				 1.57G  3.14T  1.57G  /mnt/OB-NAS-JAILS/.warden-template-standard-9.3-x64
OB-NAS-JAILS/.warden-template-standard-x86					  205M  3.14T   205M  /mnt/OB-NAS-JAILS/.warden-template-standard-x86
OB-NAS-JAILS/Owncloudjail									  1.48G  3.14T  2.30G  /mnt/OB-NAS-JAILS/Owncloudjail
OB-NAS-JAILS/maraschino_1									   129M  3.14T   594M  /mnt/OB-NAS-JAILS/maraschino_1
OB-NAS-JAILS/nextcloud										 1.67G  3.14T  2.49G  /mnt/OB-NAS-JAILS/nextcloud
OB-NAS-JAILS/oc-data											 96K  3.14T	96K  /mnt/OB-NAS-JAILS/oc-data
OB-NAS-JAILS/oc-db											 1.68M  3.14T  1.68M  /mnt/OB-NAS-JAILS/oc-db
Pool_1														 1.71T  8.78T   816G  /mnt/Pool_1
Pool_1/BTDownloads											 53.7G  46.3G  53.7G  /mnt/Pool_1/BTDownloads
Pool_1/db													   137M  8.78T   137M  /mnt/Pool_1/db
Pool_1/files													363G  8.78T   363G  /mnt/Pool_1/files
Pool_1/ncdb													 108M  8.78T   108M  /mnt/Pool_1/ncdb
Pool_1/ncfiles												 11.0G  8.78T  11.0G  /mnt/Pool_1/ncfiles
Pool_1/nextcloud_db											 264M  8.78T   264M  /mnt/Pool_1/nextcloud_db
Pool_1/nextcloud_files										  503G  8.78T   503G  /mnt/Pool_1/nextcloud_files
Pool_2														 6.19T  12.3T  6.19T  /mnt/Pool_2
freenas-boot												   5.52G  1.68G	31K  none
freenas-boot/ROOT											  5.35G  1.68G	25K  none
freenas-boot/ROOT/9.10-STABLE-201606270534					 14.9M  1.68G   609M  /
freenas-boot/ROOT/9.10.1										 38K  1.68G   630M  /
freenas-boot/ROOT/9.10.1-U1									16.2M  1.68G   637M  /
freenas-boot/ROOT/9.10.1-U2									 122K  1.68G   641M  /
freenas-boot/ROOT/9.10.1-U3									 116K  1.68G   638M  /
freenas-boot/ROOT/9.10.1-U4									17.6M  1.68G   639M  /
freenas-boot/ROOT/9.10.2									   20.7M  1.68G   656M  /
freenas-boot/ROOT/9.10.2-U1									20.6M  1.68G   657M  /
freenas-boot/ROOT/9.10.2-U2									22.1M  1.68G   659M  /
freenas-boot/ROOT/9.10.2-U3									24.8M  1.68G   663M  /
freenas-boot/ROOT/9.10.2-U4									5.18G  1.68G   663M  /
freenas-boot/ROOT/FreeNAS-1ac5f24e172b4785efcab5401aa5507f	 20.8M  1.68G   477M  /
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201604150515			  12.7M  1.68G   538M  /
freenas-boot/ROOT/Initial-Install								 1K  1.68G   506M  legacy
freenas-boot/ROOT/default									  2.49M  1.68G   510M  legacy
freenas-boot/grub											   153M  1.68G  6.34M  legacy

SweetAndLow · Jun 19, 2017

Much better, so you currently have 6TB of data in your pool_2 pool. What makes you think there is data missing?

Can you try sshing into the server and ls the/mnt/pool_2 path?
Sent from my Nexus 5X using Tapatalk

Elo · Jun 20, 2017

As I perhaps clumsily tried to explain in my first post.

Pool_2 did contain several main directory trees. Two if these held video data . One is still intact with about 6T data in it and working just fine.
Accessing the pool through SSH and listing the contents shows that more than 1k files that used to be in one of the directory trees is no longer there or does not show up.
The pools is also accessible from LAN via Windows shares (SMB). The files are of course also missing from there.
The most sensible cause is that the files has been deleted by accident. This seems very unlikely and only three people have access (involving RWX) and they are not aware of doing this.
I had a power failure some days ago which has caused me strange problems before.

My question was some aide in trying to find out what has happened. Through logs or other means if possible? I have a large portion of the data in different external sources but it is a very really big job to restore it from those sources. Further I need to understand what has happened in order to mitigate the reason and so I can trust the system to work properly. Even though I have been using FreeNAS for many years and have a rather complicated setup I do not have very deep competence into the nitty gritties and especially on details on the file system. I would be very thankfull for some help as it will:

Save me a ton of work
Make my system more reliable and trustworthy through understanding the weaknesses and mitigate the problems

Especially I am wondering if it is possible to re import the Pool_2 (if the data is there). I have done this before with some luck but I am not sure what is actually being done with a zpool destroy Pool_2 command.

It took me some days before I was aware of this problem (on travel). I am running regular scrubs that might have made the situation even worse? I have had this configuration for many years and through many FreeNAS OS updates without problems (except the occasional disk problems due to wear and tear).

fracai · Jun 20, 2017

Elo said:
Hi... No snapshot I am afraid :-(
ZFS list as bellow in the second next posting!
zfs list -o space returns error and requires a defined property. Which one would help?

Hmmm, "zfs list -o space" works fine for me without any additional properties. Are you sure you typed or copied the command correctly?
Without snapshots I don't have much further to add.

SweetAndLow · Jun 20, 2017

Elo said:
As I perhaps clumsily tried to explain in my first post.

Pool_2 did contain several main directory trees. Two if these held video data . One is still intact with about 6T data in it and working just fine.

Accessing the pool through SSH and listing the contents shows that more than 1k files that used to be in one of the directory trees is no longer there or does not show up.

The pools is also accessible from LAN via Windows shares (SMB). The files are of course also missing from there.

The most sensible cause is that the files has been deleted by accident. This seems very unlikely and only three people have access (involving RWX) and they are not aware of doing this.

I had a power failure some days ago which has caused me strange problems before.

My question was some aide in trying to find out what has happened. Through logs or other means if possible? I have a large portion of the data in different external sources but it is a very really big job to restore it from those sources. Further I need to understand what has happened in order to mitigate the reason and so I can trust the system to work properly. Even though I have been using FreeNAS for many years and have a rather complicated setup I do not have very deep competence into the nitty gritties and especially on details on the file system. I would be very thankfull for some help as it will:

Save me a ton of work

Make my system more reliable and trustworthy through understanding the weaknesses and mitigate the problems

Especially I am wondering if it is possible to re import the Pool_2 (if the data is there). I have done this before with some luck but I am not sure what is actually being done with a zpool destroy Pool_2 command.

It took me some days before I was aware of this problem (on travel). I am running regular scrubs that might have made the situation even worse? I have had this configuration for many years and through many FreeNAS OS updates without problems (except the occasional disk problems due to wear and tear).

Reimporting The pool will not do anything to recover deleted data. The destroy command will destroy everything so I'm not sure why you want to use it. A power outage can not cause files to get deleted. Scrubs make zero difference when someone has deleted the data. You keep coming up with freenas features that deleted the data but this isn't possible, some human deleted the data or misplaced it.

There are 2 command that will show you what was executed in the system. zpool history and history. One is for datasets the other is for shell commands. This might lead to an answer. If it was deleted over smb you can't really find out unless one of the 3 people fess up.

Sent from my Nexus 5X using Tapatalk

Elo · Jun 20, 2017

fracai said:
Hmmm, "zfs list -o space" works fine for me without any additional properties. Are you sure you typed or copied the command correctly?
Without snapshots I don't have much further to add.

Thanks.... here is what you asked for (was a misplaced space). Thanks for your interest and taking the time to answer...

Code:

NAME														   AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
OB-NAS-JAILS												   3.14T   384G		 0	375G			  0	  9.21G
OB-NAS-JAILS/.system										   3.14T   489M		 0	402M			  0	  86.5M
OB-NAS-JAILS/.system/configs-a65c52f7cbea4095ad912fe7c84fed72  3.14T  61.5M		 0   61.5M			  0		  0
OB-NAS-JAILS/.system/cores									 3.14T  14.0M		 0   14.0M			  0		  0
OB-NAS-JAILS/.system/rrd-a65c52f7cbea4095ad912fe7c84fed72	  3.14T   144K		 0	144K			  0		  0
OB-NAS-JAILS/.system/samba4									3.14T   704K		 0	704K			  0		  0
OB-NAS-JAILS/.system/syslog-a65c52f7cbea4095ad912fe7c84fed72   3.14T  10.1M		 0   10.1M			  0		  0
OB-NAS-JAILS/.warden-template-pluginjail					   3.14T   469M	   88K	469M			  0		  0
OB-NAS-JAILS/.warden-template-pluginjail--x64				  3.14T   449M	   96K	449M			  0		  0
OB-NAS-JAILS/.warden-template-pluginjail-9.2-x64			   3.14T   721M		 0	721M			  0		  0
OB-NAS-JAILS/.warden-template-pluginjail-9.3-x64			   3.14T   449M	   96K	449M			  0		  0
OB-NAS-JAILS/.warden-template-standard						 3.14T  1.61G	   80K   1.61G			  0		  0
OB-NAS-JAILS/.warden-template-standard-9.3-x64				 3.14T  1.57G	   96K   1.57G			  0		  0
OB-NAS-JAILS/.warden-template-standard-x86					 3.14T   205M		 0	205M			  0		  0
OB-NAS-JAILS/Owncloudjail									  3.14T  1.48G		 0   1.48G			  0		  0
OB-NAS-JAILS/maraschino_1									  3.14T   129M		 0	129M			  0		  0
OB-NAS-JAILS/nextcloud										 3.14T  1.67G		 0   1.67G			  0		  0
OB-NAS-JAILS/oc-data										   3.14T	96K		 0	 96K			  0		  0
OB-NAS-JAILS/oc-db											 3.14T  1.68M		 0   1.68M			  0		  0
Pool_1														 8.78T  1.71T		 0	816G			  0	   932G
Pool_1/BTDownloads											 46.3G  53.7G		 0   53.7G			  0		  0
Pool_1/db													  8.78T   137M		 0	137M			  0		  0
Pool_1/files												   8.78T   363G		 0	363G			  0		  0
Pool_1/ncdb													8.78T   108M		 0	108M			  0		  0
Pool_1/ncfiles												 8.78T  11.0G		 0   11.0G			  0		  0
Pool_1/nextcloud_db											8.78T   265M		 0	265M			  0		  0
Pool_1/nextcloud_files										 8.78T   503G		 0	503G			  0		  0
Pool_2														 12.3T  6.19T		 0   6.19T			  0	   126M
freenas-boot												   1.68G  5.52G		 0	 31K			  0	  5.52G
freenas-boot/ROOT											  1.68G  5.35G		 0	 25K			  0	  5.35G
freenas-boot/ROOT/9.10-STABLE-201606270534					 1.68G  14.9M		 0   14.9M			  0		  0
freenas-boot/ROOT/9.10.1									   1.68G	38K		 0	 38K			  0		  0
freenas-boot/ROOT/9.10.1-U1									1.68G  16.2M		 0   16.2M			  0		  0
freenas-boot/ROOT/9.10.1-U2									1.68G   122K		 0	122K			  0		  0
freenas-boot/ROOT/9.10.1-U3									1.68G   116K		 0	116K			  0		  0
freenas-boot/ROOT/9.10.1-U4									1.68G  17.6M		 0   17.6M			  0		  0
freenas-boot/ROOT/9.10.2									   1.68G  20.7M		 0   20.7M			  0		  0
freenas-boot/ROOT/9.10.2-U1									1.68G  20.6M		 0   20.6M			  0		  0
freenas-boot/ROOT/9.10.2-U2									1.68G  22.1M		 0   22.1M			  0		  0
freenas-boot/ROOT/9.10.2-U3									1.68G  24.8M		 0   24.8M			  0		  0
freenas-boot/ROOT/9.10.2-U4									1.68G  5.18G	 4.53G	665M			  0		  0
freenas-boot/ROOT/FreeNAS-1ac5f24e172b4785efcab5401aa5507f	 1.68G  20.8M		 0   20.8M			  0		  0
freenas-boot/ROOT/FreeNAS-9.3-STABLE-201604150515			  1.68G  12.7M		 0   12.7M			  0		  0
freenas-boot/ROOT/Initial-Install							  1.68G	 1K		 0	  1K			  0		  0
freenas-boot/ROOT/default									  1.68G  2.49M		 0   2.49M			  0		  0
freenas-boot/grub											  1.68G   153M	  146M   6.34M			  0		  0

Elo · Jun 20, 2017

SweetAndLow said:
Reimporting The pool will not do anything to recover deleted data. The destroy command will destroy everything so I'm not sure why you want to use it. A power outage can not cause files to get deleted. Scrubs make zero difference when someone has deleted the data. You keep coming up with freenas features that deleted the data but this isn't possible, some human deleted the data or misplaced it.

There are 2 command that will show you what was executed in the system. zpool history and history. One is for datasets the other is for shell commands. This might lead to an answer. If it was deleted over smb you can't really find out unless one of the 3 people fess up.

Sent from my Nexus 5X using Tapatalk

Thanks... I have gone trhough the last 500 commands as listed from the history command and looked through the zpool history Pool_2 output. I can not see anything that can explain the deletion of all these files.

For zpool history Pool_2 During June:

Code:

2017-05-26.03:08:16 zpool set cachefile=/data/zfs/zpool.cache Pool_2
2017-05-27.03:00:13 zpool scrub Pool_2
2017-06-08.03:00:10 zpool scrub Pool_2
2017-06-18.23:23:24 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4660761929273781653
2017-06-18.23:23:24 zpool set cachefile=/data/zfs/zpool.cache Pool_2
2017-06-19.16:27:34 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4660761929273781653
2017-06-19.16:27:34 zpool set cachefile=/data/zfs/zpool.cache Pool_2
2017-06-19.16:49:11 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4660761929273781653
2017-06-19.16:49:11 zpool set cachefile=/data/zfs/zpool.cache Pool_2

For the history in the periode where the files diaapeared (I worked on automatic update og SSL Certificates when every thing was ok (you see going in and out of jails)

Code:

368  22:14   cd letsencrypt
   369  22:14   ls
   370  22:14   cd xxxxxxxx.com
   371  22:15   cp *.* /mnt/OB-NAS-JAILS/nextcloud/usr/local/etc/letsencrypt/live/xxxxxx.com/
   372  23:35   jls
   373  23:35   jexec 9 tcsh
   374  2:25	jls
   375  2:25	jexec 9 tcsh
   376  12:20   jls
   377  12:20   jexec 10 tcsh
   378  23:14   cd /mnt
   379  23:14   ls
   380  23:14   cd Pool_1
   381  23:14   ls
   382  23:14   cd ..
   383  23:15   cd Pool_2
   384  23:15   ls -la
   385  23:15   cd VIDEO
   386  23:15   ls
   387  23:15   cd Film
   388  23:15   ls -la

The two other having access has not been in the house during the time this happened. They could delete the files through J River MC22 but then the library would show these files missing so I know that has not happened. That leaves myself and anyone with non authorised access. I have not deleted the files through SMB. I will not rule out anything but is is realy strange

SweetAndLow · Jun 20, 2017

Elo said:
Thanks... I have gone trhough the last 500 commands as listed from the history command and looked through the zpool history Pool_2 output. I can not see anything that can explain the deletion of all these files.

For zpool history Pool_2 During June:

Code:
2017-05-26.03:08:16 zpool set cachefile=/data/zfs/zpool.cache Pool_2 2017-05-27.03:00:13 zpool scrub Pool_2 2017-06-08.03:00:10 zpool scrub Pool_2 2017-06-18.23:23:24 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4660761929273781653 2017-06-18.23:23:24 zpool set cachefile=/data/zfs/zpool.cache Pool_2 2017-06-19.16:27:34 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4660761929273781653 2017-06-19.16:27:34 zpool set cachefile=/data/zfs/zpool.cache Pool_2 2017-06-19.16:49:11 zpool import -c /data/zfs/zpool.cache.saved -o cachefile=none -R /mnt -f 4660761929273781653 2017-06-19.16:49:11 zpool set cachefile=/data/zfs/zpool.cache Pool_2

For the history in the periode where the files diaapeared (I worked on automatic update og SSL Certificates when every thing was ok (you see going in and out of jails)

Code:
368 22:14 cd letsencrypt 369 22:14 ls 370 22:14 cd xxxxxxxx.com 371 22:15 cp *.* /mnt/OB-NAS-JAILS/nextcloud/usr/local/etc/letsencrypt/live/xxxxxx.com/ 372 23:35 jls 373 23:35 jexec 9 tcsh 374 2:25 jls 375 2:25 jexec 9 tcsh 376 12:20 jls 377 12:20 jexec 10 tcsh 378 23:14 cd /mnt 379 23:14 ls 380 23:14 cd Pool_1 381 23:14 ls 382 23:14 cd .. 383 23:15 cd Pool_2 384 23:15 ls -la 385 23:15 cd VIDEO 386 23:15 ls 387 23:15 cd Film 388 23:15 ls -la

The two other having access has not been in the house during the time this happened. They could delete the files through J River MC22 but then the library would show these files missing so I know that has not happened. That leaves myself and anyone with non authorised access. I have not deleted the files through SMB. I will not rule out anything but is is realy strange

Are you sure they are gone? If you look in your media player what path should they be in on the server? Usually when things magically disappear it's just users forgetting where stuff is.

Sent from my Nexus 5X using Tapatalk

Elo · Jun 20, 2017

SweetAndLow said:
Are you sure they are gone? If you look in your media player what path should they be in on the server? Usually when things magically disappear it's just users forgetting where stuff is.

Sent from my Nexus 5X using Tapatalk

I am of course sure that they are gone in the sense that they does not show when applying a ls -la command or by entering the directory in Windows. Further the user do not have to remember the path as all files are entered into a data base in MC22 where metadata , paths etc are stored. So when a user tries to use the file the player will give notice if it can not find the file or there are other errors according to the information stored in the players database.

rs225 · Jun 20, 2017

My observations:

1. Power failure will not cause ZFS to lose directories/files.
2. Snapshots would have been very helpful in this situation.
3. Either human error or a software bug in your clients is a likely cause.

With this client software, what happens if you put random data in the folders? Does the client remove it?

Elo · Jun 20, 2017

rs225 said:
My observations:

1. Power failure will not cause ZFS to lose directories/files.
2. Snapshots would have been very helpful in this situation.
3. Either human error or a software bug in your clients is a likely cause.

With this client software, what happens if you put random data in the folders? Does the client remove it?

Thanks for your observations.

I do not think that it is possible to categorically say that power failures can NOT cause loss of files but I appreciate that this is your observations. We vare not talking about a clean on off situation but a situation where the power goes on and off with a lot of surges for a while before it stabilizes.
Of course snapshots would be helpful. But I don't have them and it cant be helped.

There is still 6TB of data that is accessible to the player plus several Tbs of photos. (J River Media Player MC22). I can not see that any other data is removed during the time from last time we accessed the video data or for the last several years. SO I guess I do not know how I can test this further ....

I would rely like to not base mitigating actions on speculations but rather find out what is a probable reason through investigation of the systems logs. If this is not possible i will rebuild and make sure that i have the necessary snapshots available.

Further i like to be sure that there is no way to retrieve the data.

rs225 · Jun 20, 2017

Some filesystems can lose directories, a files entire content, etc, in a power failure. What I mean is that ZFS does not have that type of design.

Robert Trevellyan · Jun 20, 2017

One way the contents of a directory can mysteriously disappear is when something is mounted on top of it.

SweetAndLow · Jun 20, 2017

Elo said:
Thanks for your observations.

I do not think that it is possible to categorically say that power failures can NOT cause loss of files but I appreciate that this is your observations. We vare not talking about a clean on off situation but a situation where the power goes on and off with a lot of surges for a while before it stabilizes.
Of course snapshots would be helpful. But I don't have them and it cant be helped.

There is still 6TB of data that is accessible to the player plus several Tbs of photos. (J River Media Player MC22). I can not see that any other data is removed during the time from last time we accessed the video data or for the last several years. SO I guess I do not know how I can test this further ....

I would rely like to not base mitigating actions on speculations but rather find out what is a probable reason through investigation of the systems logs. If this is not possible i will rebuild and make sure that i have the necessary snapshots available.

Further i like to be sure that there is no way to retrieve the data.

If something is written to disk you can toggle the power a million times and as long as the disk is alive your data will be there. Power failure can not cause a directory to disappear. I'm flight writes may end up in an unknown state but that is all.

Sent from my Nexus 5X using Tapatalk

SweetAndLow · Jun 20, 2017

Robert Trevellyan said:
One way the contents of a directory can mysteriously disappear is when something is mounted on top of it.

I agree but they say there are no datasets or anything. Now if they said director exist but files gone then maybe something got mounted over the previous director.

Sent from my Nexus 5X using Tapatalk

Elo · Jun 20, 2017

SweetAndLow said:
I agree but they say there are no datasets or anything. Now if they said director exist but files gone then maybe something got mounted over the previous director.

Sent from my Nexus 5X using Tapatalk

The data was organized in one main directory and then each video in its own subdirectory under that. All subdirectories are gone but the main one still exists.

Would a mount show up in any of the logs?

Important Announcement for the TrueNAS Community.

Strange disappearance of entire directory structure

Contributor

Guru

Contributor

Sweet'NASty

Contributor

Sweet'NASty

Contributor

Guru

Sweet'NASty

Contributor

Contributor

Sweet'NASty

Contributor

Guru

Contributor

Guru

Pony Wrangler

Sweet'NASty

Sweet'NASty

Contributor

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Strange disappearance of entire directory structure"

Similar threads