Jail Misconfiguration Data Destroyed? 5.3TB pool 1TB Used

[OctaneZ]

Backstory:
I upgraded from 8.3 to 9.1 after a USB stick failure 8 days ago. Auto-imported zvol, all data survived. Today in an effort to setup Transmission I was editing the Jails configuration to create a directory structure for transmission. In 8.3 that directory was:

Code:

/fourslot-data/benc/transmission

The jail edit/creation processed that and appeared to work, however the ~1TB of data in ~benc is now hidden. 1TB is still in use according to df but I can't access it.

Code:

fourslot:~ benc$ df -h
Filesystem                                                          Size    Used  Avail Capacity  Mounted on
/dev/ufs/FreeNASs1a                                                  926M    654M    198M    77%    /
devfs                                                                1.0k    1.0k      0B  100%    /dev
/dev/md0                                                            4.6M    3.8M    403k    91%    /etc
/dev/md1                                                            823k    2.0k    756k    0%    /mnt
/dev/md2                                                            149M    20M    116M    15%    /var
/dev/ufs/FreeNASs4                                                    19M      1M    17M    5%    /data
fourslot-data                                                        5.3T      1T    4.4T    19%    /mnt/fourslot-data
fourslot-data/Jail                                                  4.4T    497M    4.4T    0%    /mnt/fourslot-data/Jail
fourslot-data/Plugins                                                4.4T    546M    4.4T    0%    /mnt/fourslot-data/Plugins
fourslot-data/jails                                                  4.4T    137M    4.4T    0%    /mnt/fourslot-data/jails
fourslot-data/jails/.warden-template-9.1-RELEASE-amd64-pluginjail    4.4T    488M    4.4T    0%    /mnt/fourslot-data/jails/.warden-template-9.1-RELEASE-amd64-pluginjail
fourslot-data/jails/bit_1                                            4.4T    636M    4.4T    0%    /mnt/fourslot-data/jails/bit_1
devfs                                                                1.0k    1.0k      0B  100%    /mnt/fourslot-data/jails/bit_1/dev
procfs                                                              4.0k    4.0k      0B  100%    /mnt/fourslot-data/jails/bit_1/proc
fourslot-data/benc                                                  4.4T    44k    4.4T    0%    /mnt/fourslot-data/benc
fourslot-data/benc/transmission                                      4.4T    44k    4.4T    0%    /mnt/fourslot-data/benc/transmission

The Web interface storage tab shows:

Code:

fourslot-data    1014.5 GiB (18%)    4.3 TiB    5.3 TiB    HEALTHY

Can anyone save me and tell me how to locate this "missing" and essential 1TB of data?

I am at whits end.

 

[cyberjock]

Um, benc doesn't have 1TB of data. Look at that chart again.. It says 44kbytes used.

 

[OctaneZ]

Right, completely agree. But something in fourslot-data is still using a TB of data:

Code:

fourslot-data  5.3T 1T 4.4T 19% /mnt/fourslot-data

Which is why I am hoping that the "missing" terabyte is not destroyed [yet] just not accessible at the moment, which is why rather than futzing in jexec, etc, I am hoping someone here might have an idea about how to get to that used 19% of data.

 

[cyberjock]

I'd go and start looking in your folders for where that 1TB of data is.

If you had snapshots you could maybe do a snapshot restore from a time when the data did exist. Other than that, I have nothing else to recommend. Now you see why we harp on people about backups so much.

 

[OctaneZ]

Thanks. I have been looking, but before going to extraordinary measures, I was hoping that there was a ZFS store somewhere that (without having been scrubbed, etc) was still intact. That 1TB of data is currently inaccessible:

Code:

[benc@fourslot] /mnt/fourslot-data# du -hs * .*
1.5G    Jail
548M    Plugins
5.0k    benc
1.3G    jails
13k    jd
13k    ryan
3.6G    .
3.6G    ..
383M    .freenas

So the 1TB is still in use, but is not currently being mounted, but is still in use:

Code:

[benc@fourslot] /mnt/fourslot-data# df -h | grep 5.3T
fourslot-data    5.3T      1T    4.4T    19%    /mnt/fourslot-data

 

[cyberjock]

Well, you appear to have a folder called "Jail" and another folder called "jails". There's bonus points in FreeBSD for capitalization too.

I'd go into the "Jail" and "jails" and see what's taking up space there.

The only thing even remotely "hidden" would be snapshots. But unless you've set that up you have none.

 

[OctaneZ]

Right, I've been through "Jail" and "jails", they have the core BSD jail files, about 1GB each:​

Code:

[benc@fourslot] /mnt/fourslot-data# du -hs * .*
997M    Jail
548M    Plugins
5.0k    benc
1.3G    jails
13k    jd
13k    ryan
3.1G    .
3.1G    ..
383M    .freenas

but Storage-->Active Volumes, still says I have 1TB in use, 5.3TB total, 4.3 Available, and 1014.5 GiB (18%) in use:

 

[cyberjock]

Oh, crap. I just confused GB with TB. I'm definitely a moron today.

du and df aren't particularly useful ways for ZFS. They don't do their calculations correctly in the way that ZFS functions. You'll get a slightly more useful response if you used du -hs * instead of your command. That extra space is probably not doing you any favors.

Edit: It also appears that it completely ignores all folders that have a period as the first character. So that kind of ruins the whole exercise because you are already looking for hidden files/folders(which start with a period in the name). Looks like you're going to have to do it the old fashioned way by scouring the directories. :(

 

[OctaneZ]

Yes, now you see where I am. The link/mount is broken in

Code:

/mnt/fourslot-data

data is being used but is inaccessible. the point of using

du -hs * .*

is to show that there are no hidden files that the data is residing in. The "missing" data survived a reboot. "fourslot-data" is still showing 1TB is use. Auto-import and import volume are not showing an orphaned dataset. I destroyed my jail successfully, and the "missing" data is still intact. I have a feeling (and I am quite comfortable with) that we are in a CLI-mode recovery. 1TB is still in use, I just can't mount it or find the command to show/list/mount an orphaned ZFS dataset. There are no ZFS snapshots or replications.

 

[cyberjock]

yeah, except that doesn't show hidden files. "ls -l" does though. :) Post the output of "ls -l /mnt/fourslot-data".

 

[OctaneZ]

Code:

[benc@fourslot] /mnt/fourslot-data# ls -la
total 17
drwxr-xr-x  9 1002  wheel    9 Nov 25 09:38 ./
drwxr-xr-x  4 root  wheel  512 Dec  2 20:23 ../
drwxr-xr-x  7 root  benc    9 Oct 27  2012 .freenas/
drwxr-xr-x  6 root  wheel    8 Dec  2 20:26 Jail/
drwxr-xr-x  5 root  wheel  13 Jul 27 20:19 Plugins/
drwxr-xr-x  3 root  wheel    3 Dec  2 18:45 benc/
drwxr-xr-x  7 root  wheel    9 Nov 25 09:45 jails/
drwxr-xr-x  2 1003  1003    10 Sep 17 16:02 jd/
drwxr-xr-x  2 1002  wheel    9 Jan 24  2012 ryan/
[benc@fourslot] /mnt/fourslot-data# ls -la benc/
total 6
drwxr-xr-x  3 root  wheel  3 Dec  2 18:45 ./
drwxr-xr-x  9 1002  wheel  9 Nov 25 09:38 ../
drwxr-xr-x  2 root  wheel  3 Dec  2 18:45 transmission/

no hidden directories with data.

 

[cyberjock]

actually, .freenas is a hidden directory. But you'd have to look inside to see how much data is there.

All I can say is you are going to have to go hunting manually. It's quite possible you have a folder that root/wheel doesn't have permissions to access, so you have data that is basically "hiding".

 

[Dusan]

Is it possible that you have something else mounted at /fourslot-data/benc/transmission?
Could you please post output of "mount"?

 

[OctaneZ]

Dusan, great suggestion!!

Code:

[benc@fourslot ~]$ mount
/dev/ufs/FreeNASs1a on / (ufs, local, read-only)
devfs on /dev (devfs, local, multilabel)
/dev/md0 on /etc (ufs, local)
/dev/md1 on /mnt (ufs, local)
/dev/md2 on /var (ufs, local)
/dev/ufs/FreeNASs4 on /data (ufs, local, noatime, soft-updates)
fourslot-data on /mnt/fourslot-data (zfs, local, noatime, nfsv4acls)
fourslot-data/Jail on /mnt/fourslot-data/Jail (zfs, local, noatime, nfsv4acls)
fourslot-data/Jail/.warden-template-9.1-RELEASE-amd64-pluginjail on /mnt/fourslot-data/Jail/.warden-template-9.1-RELEASE-amd64-pluginjail (zfs, local, noatime, nfsv4acls)
fourslot-data/Plugins on /mnt/fourslot-data/Plugins (zfs, local, noatime, nfsv4acls)
fourslot-data/benc on /mnt/fourslot-data/benc (zfs, local, noatime, nfsv4acls)
fourslot-data/benc/transmission on /mnt/fourslot-data/benc/transmission (zfs, local, noatime, nfsv4acls)
fourslot-data/jails on /mnt/fourslot-data/jails (zfs, local, noatime, nfsv4acls)
fourslot-data/jails/.warden-template-9.1-RELEASE-amd64-pluginjail on /mnt/fourslot-data/jails/.warden-template-9.1-RELEASE-amd64-pluginjail (zfs, local, noatime, nfsv4acls)
fourslot-data/jails/bit_1 on /mnt/fourslot-data/jails/bit_1 (zfs, local, noatime, nfsv4acls)

so there are nested mounts at /mnt/fourslot-data and /mnt/fourslot-data/benc. You are brilliant! After umounting the nested directories:

Code:

[benc@fourslot /mnt/fourslot-data]$ su
Password:
[benc@fourslot] /mnt/fourslot-data# umount /mnt/fourslot-data/benc
umount: unmount of /mnt/fourslot-data/benc failed: Device busy
[benc@fourslot] /mnt/fourslot-data# umount /mnt/fourslot-data/benc/transmission/
[benc@fourslot] /mnt/fourslot-data# umount /mnt/fourslot-data/benc/

I get access to my data!

Code:

[benc@fourslot] /mnt/fourslot-data/benc# ls
./                  .login              201303-House_Damage/
../                  .login_conf          Cape Pics/
.DS_Store            .mail_aliases        Documents/
.bash_history        .mailrc              bin/
.bash_profile        .profile            data/
.bashrc              .rhosts              gallery/
.cshrc              .shrc                transmission/
.lesshst            .ssh/

Now to not touch it until I get a full backup of everything on an external drive.

After the backup what do people recommend? Don't get me wrong I am elated to have my data, but still not sure how to fix the actual problem. I am concerned that the jail setup is still wrong, and that while I no longer have the jails or plugin, the nested mountpoints are still being mounted. Is there a better option than going nuclear and backing everything up, paving the config, starting from scratch, and rsyncing the data back onto a fresh install?

Again, thanks for the help guys! I am giddy to see the data intact!