Data disappearing. How is this happening?

[Marlberg]

In my signature you will find the relevant hardware information. The Proxmox server information can be ignored as it is not attached via share or via any other method to the TRUENAS server although it does reside on the same subnet.

Problem: I have a single plugin created jail (Plex beta) installed, two smb shares (/mnt/WizardPool1/media (for my Plex Server) and /mnt/WizardPool1/WizardHomeDS as windows shares) one NFS Share one Cloud Sync (B2) task that runs daily, a scrub task that runs bi-monthly (1,15) and in the WizardHomeDS share I have backup images of my various home pc's/devices. As a safety precaution (until I can procure another similar set of hardware) I have a 14TB external HDD attached to one of my home pc's that has a copy of all of the data from both smb shares. This is fortunate as twice now I have had to reload all of the share data from both smb shares from that external HDD. Therein lies the problem. I recopy the data down to theTrueNAS Core 12.1 server from the external HDD (8TB or so which takes about a day and a half) reinstall the plugin get back up and running then at some point in the next day or so after rebuilding I open the media or WizardHomeDS share (through Windows 10) only to find all of the data gone. No error reported on the logs. No system alerts in the gui the data is just no longer where I put it. My event logs and firewall (ASUS router firewall with verbose logging turned on) show no intrusions or suspicious activity Wireshark shows no weird network traffic, I just wake up that morning to two totally empty datasets and mount points. Any ideas on where to start looking to trace down why this is happening? Tell me what you need to see from me and I'll try to post it. I hate having to reload the data every few days.

 

[Marlberg]

I don't believe I am much closer to discovering where 8TB of data just seems to disappear to. About the only thing timing wise that might be responsible (and that is a long shot) was a simultaneous backup of one of my pc's to one of the SMB shares using Macrium Reflect 7.3 and scheduled SMART Test on all disks on the striped vdev's. No errors in the logs no disk errors nothing sinister in any of the network logs from wireshark. Frankly stumped on this.

 

[Marlberg]

Update 2: Never found out what caused it systems been running stable with data being right where it is supposed to be for two weeks now. The only changes I have made is that I have not run a scrub task since the last data loss. Could a scrub task cause the data to just simply disappear?

 

[jgreco]

A scrub task shouldn't do anything that reading the relevant data wouldn't also do.

When ZFS reads bad data, it makes an attempt to correct it, and there is possibly some room for bad things to happen if there is no available source for recovery. This is bad if the data happens to be metadata (stuff like directory contents).

Scrubbing is really just a process of reading the entire pool, and letting ZFS fix any issues it finds, very similarly to what would happen if you independently read bad data off the pool.

 

[Marlberg]

Scrubbing is really just a process of reading the entire pool, and letting ZFS fix any issues it finds, very similarly to what would happen if you independently read bad data off the pool.

Yep, thats exactly what I thought too. So its still kind of a stumper then where 8 TiB of data went the SMB shares were there mount points too. just no subdirectories or data in them...Troubling...I wish I knew of a shell command that could tell me what exactly happened to make the data just delete itself...

P.S. I know the data did not do anything of its own volition but from what I have seen that's very much the way it appeared (appearances being deceptive and all that)