jasonbuechler
Dabbler
- Joined
- Mar 28, 2018
- Messages
- 20
Updated:
I figured out most of the issue. And as predicted it's kind of a forehead slap: although I had checked my snapshots for the jail in question... I went searching for a *single* very large snapshot reflecting the sudden "deletions" last night. But I foolishly forgot that snapshots are progressive, and, of course, so were my regular archivals of security videos... thus, they nickel and dimed their way up to the "missing" space.
I'm still not fully clear on why there was no single huge snapshot covering all-at-once deletions, but I'm close enough to not care to look into it anymore. If anyone has an easily digestible explanation, please do share!
---
Ok... Sigh... I just know this is going to be a dead simple thing but after hours, I'm not even an inch closer to understanding what's happening. Please help? (And keep in mind my brain is fried with frustration at the moment!)
For months, I've been running a jail ("jail1") whose main purpose (of 2 or 3) was to ftp recordings off my IP-cameras. The recordings were saved to the jail's internal filesystem. Last night I finally got around to creating a jail for this exclusive purpose, and simultaneously move the archive to a dataset created for security recordings. I sudo'd rsync with
I tried to triage the situation using the obvious tools, but different tools give me different totals, and whether or not a jail is running (aka its external mount point is mounted) will also change the totals, so I don't know what to believe, and more importantly, don't know what to do about it. Halp!
There's at least 2 confusing things going on here. I'm prepared to accept one, but then the other still makes no sense.
With the original "archival" jail ("jail1") down,
(I don't think it matters to the core problem, but both jails mount my htpc dataset. fstab is below, too.)
The du utility should be smart enough to tabulate only once on links, so maybe it makes a difference running it from / ... and I accept that, but it still doesn't explain the "jail1" issue. Just for completeness, you can see du's totals change after stopping the plex plugin:
Beginning to grasp at straws, I removed the external mount from jail1, but that changed nothing in either du or iocage df. And now I'm out of ideas...
Someone please tell me I'm missing some crucial part of understanding how to read some of these outputs and there's a quick and obvious way to fix up what I'm seeing? Thaaaaanks.
--jason
I figured out most of the issue. And as predicted it's kind of a forehead slap: although I had checked my snapshots for the jail in question... I went searching for a *single* very large snapshot reflecting the sudden "deletions" last night. But I foolishly forgot that snapshots are progressive, and, of course, so were my regular archivals of security videos... thus, they nickel and dimed their way up to the "missing" space.
I'm still not fully clear on why there was no single huge snapshot covering all-at-once deletions, but I'm close enough to not care to look into it anymore. If anyone has an easily digestible explanation, please do share!
---
Ok... Sigh... I just know this is going to be a dead simple thing but after hours, I'm not even an inch closer to understanding what's happening. Please help? (And keep in mind my brain is fried with frustration at the moment!)
For months, I've been running a jail ("jail1") whose main purpose (of 2 or 3) was to ftp recordings off my IP-cameras. The recordings were saved to the jail's internal filesystem. Last night I finally got around to creating a jail for this exclusive purpose, and simultaneously move the archive to a dataset created for security recordings. I sudo'd rsync with
--remove-source-files
and watched the output for a while to verify things looked like they were running smoothly (i.e. that it accurately reported it was deleting original files) and went to bed with a little less than 3TB free space... and woke up with 0.2GB free space!? (And restarting the machine hasn't had an effect.)I tried to triage the situation using the obvious tools, but different tools give me different totals, and whether or not a jail is running (aka its external mount point is mounted) will also change the totals, so I don't know what to believe, and more importantly, don't know what to do about it. Halp!
- Stock Dell PowerEdge T30
- Xeon E3-1225 v5 3.3GHz
- 8GB ram
- 4x 4TB WD Reds + 1x 8TB WD "white-red" -- RaidZ2/double redundant
- IO Crest 4 port SATA III PCI-e 2.0 x1 controller (https://www.amazon.com/dp/B00AZ9T3OU)
- Onboard ethernet
There's at least 2 confusing things going on here. I'm prepared to accept one, but then the other still makes no sense.
With the original "archival" jail ("jail1") down,
iocage df
reports jail1 using almost 2.91TB, but du
reports that same jail using 480GB. My plex plugin should be totally unrelated to the jail1 issue, but since it's also kinda weird, I'll mention this too: At the same time, with the "plex_2" jail up, iocage df
reports plex_2 using only 4.69GB, but du
reports it's using 3.0TB. ...!??!(I don't think it matters to the core problem, but both jails mount my htpc dataset. fstab is below, too.)
Code:
[fox@tree ~]$ iocage list +-----+---------------+-------+--------------+------+ | JID | NAME | STATE | RELEASE | IP4 | +=====+===============+=======+==============+======+ | - | IPC_archive | down | 11.2-RELEASE | DHCP | +-----+---------------+-------+--------------+------+ | - | jail1 | down | 11.2-RELEASE | DHCP | +-----+---------------+-------+--------------+------+ | - | media_sharing | down | 11.2-RELEASE | DHCP | +-----+---------------+-------+--------------+------+ | 2 | plex_2 | up | 11.2-RELEASE | DHCP | +-----+---------------+-------+--------------+------+ | 1 | transmission | up | 11.2-RELEASE | DHCP | +-----+---------------+-------+--------------+------+ [fox@tree ~]$ iocage df --long +---------------+-------+------+------+-------+------+ | NAME | CRT | RES | QTA | USE | AVA | +===============+=======+======+======+=======+======+ | IPC_archive | 1.69x | none | none | 68.0M | 176G | +---------------+-------+------+------+-------+------+ | jail1 | 1.00x | none | none | 2.91T | 176G | <===== !? most was rsync'd externally +---------------+-------+------+------+-------+------+ | media_sharing | 2.10x | none | none | 88.8M | 176G | +---------------+-------+------+------+-------+------+ | plex_2 | 1.63x | none | none | 4.69G | 176G | <===== iocage df shows <5G +---------------+-------+------+------+-------+------+ | transmission | 1.02x | none | none | 2.74G | 176G | +---------------+-------+------+------+-------+------+ [fox@tree /tree]$ sudo du -h -d 1 /tree/iocage/jails/* 2.2T /tree/iocage/jails/IPC_archive/root 2.2T /tree/iocage/jails/IPC_archive 480G /tree/iocage/jails/jail1/root 480G /tree/iocage/jails/jail1 1.4G /tree/iocage/jails/media_sharing/root 1.4G /tree/iocage/jails/media_sharing 126K /tree/iocage/jails/plex_2/plugin 3.0T /tree/iocage/jails/plex_2/root 3.0T /tree/iocage/jails/plex_2 <=== lies! counting external mount 118K /tree/iocage/jails/transmission/plugin 4.0G /tree/iocage/jails/transmission/root 4.0G /tree/iocage/jails/transmission [fox@tree ~/iocage/jails]$ sudo iocage fstab -l jail1 +-------+---------------------------------------------------------------------------------------------------+ | INDEX | FSTAB ENTRY | +=======+===================================================================================================+ | 0 | /mnt/tree_nas/htpc /mnt/tree_nas/iocage/jails/jail1/root/tree_htpc nullfs rw 0 0 | +-------+---------------------------------------------------------------------------------------------------+ [fox@tree ~/iocage/jails]$ sudo iocage fstab -l plex_2 +-------+---------------------------------------------------------------------------------------------------+ | INDEX | FSTAB ENTRY | +=======+===================================================================================================+ | 14 | /mnt/tree_nas/htpc /mnt/tree_nas/iocage/jails/plex_2/root/media nullfs rw 0 0 | +-------+---------------------------------------------------------------------------------------------------+
The du utility should be smart enough to tabulate only once on links, so maybe it makes a difference running it from / ... and I accept that, but it still doesn't explain the "jail1" issue. Just for completeness, you can see du's totals change after stopping the plex plugin:
Code:
[fox@tree ~]$ sudo du -s -h /mnt/tree_nas/iocage/jails/* Password: 1.4G /mnt/tree_nas/iocage/jails/IPC_archive 480G /mnt/tree_nas/iocage/jails/jail1 1.4G /mnt/tree_nas/iocage/jails/media_sharing 3.0T /mnt/tree_nas/iocage/jails/plex_2 <===== while jail is up: du shows 3T 4.0G /mnt/tree_nas/iocage/jails/transmission [fox@tree ~]$ sudo iocage stop plex_2 Password: * Stopping plex_2 + Running prestop OK + Stopping services OK + Tearing down VNET OK + Removing devfs_ruleset: 6 OK + Removing jail process OK + Running poststop OK [fox@tree ~]$ sudo du -s -h /mnt/tree_nas/iocage/jails/* 1.4G /mnt/tree_nas/iocage/jails/IPC_archive 480G /mnt/tree_nas/iocage/jails/jail1 1.4G /mnt/tree_nas/iocage/jails/media_sharing 4.3G /mnt/tree_nas/iocage/jails/plex_2 <===== jail down: du now "almost" 0 4.0G /mnt/tree_nas/iocage/jails/transmission
Beginning to grasp at straws, I removed the external mount from jail1, but that changed nothing in either du or iocage df. And now I'm out of ideas...
Code:
[fox@tree ~]$ sudo du -s -h /mnt/tree_nas/iocage/jails/* 1.4G /mnt/tree_nas/iocage/jails/IPC_archive 480G /mnt/tree_nas/iocage/jails/jail1 1.4G /mnt/tree_nas/iocage/jails/media_sharing 4.3G /mnt/tree_nas/iocage/jails/plex_2 4.0G /mnt/tree_nas/iocage/jails/transmission [fox@tree ~]$ iocage df --long +---------------+-------+------+------+-------+------+ | NAME | CRT | RES | QTA | USE | AVA | +===============+=======+======+======+=======+======+ | IPC_archive | 1.69x | none | none | 68.0M | 176G | +---------------+-------+------+------+-------+------+ | jail1 | 1.00x | none | none | 2.91T | 176G | +---------------+-------+------+------+-------+------+ | media_sharing | 2.10x | none | none | 88.8M | 176G | +---------------+-------+------+------+-------+------+ | plex_2 | 1.63x | none | none | 4.69G | 176G | +---------------+-------+------+------+-------+------+ | transmission | 1.02x | none | none | 2.74G | 176G | +---------------+-------+------+------+-------+------+ root@tree:/mnt/tree_nas/iocage/jails # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT freenas-boot 58G 3.87G 54.1G - - - 6% 1.00x ONLINE - tree_nas 18.1T 17.1T 1.02T - - 26% 94% 1.00x ONLINE /mnt
Someone please tell me I'm missing some crucial part of understanding how to read some of these outputs and there's a quick and obvious way to fix up what I'm seeing? Thaaaaanks.
--jason
Last edited: