Used space much larger than file size

[garachen]

Hello,

I recently inherited a large FreeNAS build. Being no expert, I read the manual and have poked around this forum and Google for a few days now.

When I inherited the pool, I noticed that it was at above 90% utilization so I deleted several million files to free up about 4TB of space. This is the current status:

This pool is 4 vdevs of 8 4TB drives each in RaidZ2.

My problem is that I don't see how it's possible that I'm using 74 TiB of disk space. The largest directory is 18.6TB and when I add all the other directory sizes together they sum to less than 30TB.

My method of calculation is using windows explorer on each directory at a time and taking the larger of "Size" and "Size on disk". While doing this I noticed that at times the "Size on disk" would be WAY larger than size. For example, on one folder the size was 42 GB but size on disk was 1.13 TB. The pattern I found is that "size on disk" uses a minimum size of 1MB per file so that folders with many small files had huge "size on disk". Not 1KB. 1MB. 1.13 TB was for 1M files.

Again, even with these inflated numbers, the sum of the directories is less than 30TB.

I'm not super familiar with FreeBSD / Unix, but I tried several variants of the "du" and "df" commands. However, these commands fail utterly when used in a large directories. My latest "du" has been running for over 48 hours.

I'm wondering if there is some issue with how the disks were set up that is causing these very high estimates.

using:
zdb -U /data/zfs/zpool.cache
I see ashift was set to 9. Maybe not great.

using:
zfs get all GroupA
I see recordsize is 128K. Also lz4 is on. Dedup is off.

I can't remember the command I ran to see that (free space?) disk fragmentation was 42%
I was running snapshots but deleted all of them after I deleted the 4TB of files. The amount of space released corresponded to the amount of disk usage that windows explorer had said was the "size" of the folder.

FreeNAS-9.3-STABLE-201506162331
262084MB memory
using NFS and CIFS

 

[HoneyBadger]

Can you post in code tags the full output of the zfs get all GroupA command? I have a feeling there's a UI bug here.

 

[garachen]

I'm not sure how to post in code tags.

Code:

$ zfs get all GroupA
NAME    PROPERTY              VALUE                  SOURCE
GroupA  type                  filesystem             -
GroupA  creation              Thu Apr 17 13:15 2014  -
GroupA  used                  74.6T                  -
GroupA  available             9.44T                  -
GroupA  referenced            74.6T                  -
GroupA  compressratio         1.37x                  -
GroupA  mounted               yes                    -
GroupA  quota                 none                   local
GroupA  reservation           none                   local
GroupA  recordsize            128K                   default
GroupA  mountpoint            /mnt/GroupA            default
GroupA  sharenfs              off                    default
GroupA  checksum              on                     default
GroupA  compression           lz4                    local
GroupA  atime                 off                    local
GroupA  devices               on                     default
GroupA  exec                  on                     default
GroupA  setuid                on                     default
GroupA  readonly              off                    default
GroupA  jailed                off                    default
GroupA  snapdir               hidden                 default
GroupA  aclmode               restricted             local
GroupA  aclinherit            passthrough            local
GroupA  canmount              on                     default
GroupA  xattr                 off                    temporary
GroupA  copies                1                      default
GroupA  version               5                      -
GroupA  utf8only              off                    -
GroupA  normalization         none                   -
GroupA  casesensitivity       sensitive              -
GroupA  vscan                 off                    default
GroupA  nbmand                off                    default
GroupA  sharesmb              off                    default
GroupA  refquota              none                   local
GroupA  refreservation        none                   local
GroupA  primarycache          all                    default
GroupA  secondarycache        all                    default
GroupA  usedbysnapshots       44.2G                  -
GroupA  usedbydataset         74.6T                  -
GroupA  usedbychildren        13.2G                  -
GroupA  usedbyrefreservation  0                      -
GroupA  logbias               latency                default
GroupA  dedup                 off                    default
GroupA  mlslabel                                     -
GroupA  sync                  standard               default
GroupA  refcompressratio      1.37x                  -
GroupA  written               304M                   -
GroupA  logicalused           102T                   -
GroupA  logicalreferenced     102T                   -
GroupA  volmode               default                default
GroupA  filesystem_limit      none                   default
GroupA  snapshot_limit        none                   default
GroupA  filesystem_count      none                   default
GroupA  snapshot_count        none                   default
GroupA  redundant_metadata    all                    default

 

[HoneyBadger]

Use square brackets [] with the word (code) and you get text formatted like this:

Code:

GroupA usedbysnapshots 44.2G

You've still got a few snapshots, but it's still measuring in GB and not TB - so they aren't the root cause.

You mentioned deleting "a few million files" - if that wasn't a figure of speech and you literally do have millions of files at play, you may simply be a victim of the files being smaller than the smallest block size ZFS can address. By default in 9.3, that's 4KB.

GroupA/BsdHome and GroupA/jails are also showing less than 100KB used each - so what exactly is eating the space? Do you have zvols or other datasets that aren't visible in the initial screenshot?

 

[garachen]

I deleted approx 7 million files. They were mostly small files. According to Windows there should be about 4 million files left on the disk.
After deleting the files I deleted all snapshots and then allowed the snapshots to keep running.

I have no other zvols or datasets.

Even at a minimum of 4KB per file the millions of file thing shouldn't cause a space problem. Even though windows reports 1MB per file minimum I'm still well under that the FreeNas GUI thinks. I'm currently trying to run a filecount in one of the directories using the shell command: find . | wc -l in order to see if perhaps there are actual files there that for some reason Windows doesn't see. Been running for about 3 hours now in a directory that should contain about 1 Million files. Seems like a long time.

I have not rebooted.... never rebooted this server. Not sure if that would help.

 

[HoneyBadger]

Can I get output of

zfs list -o space -r GroupA

Please use [/code] tags or a fixed-width font from the drop-down (eg: Courier New) to preserve formatting

 

[garachen]

Code:

zfs list -o space -r GroupA
NAME                                                     AVAIL   USED  USEDSNAP                            USEDDS  USEDREFRESERV  USEDCHILD
GroupA                                                   9.44T  74.6T     44.2G                             74.6T              0      13.2G
GroupA/.system                                           9.44T   370M         0                              328M              0      42.6M
GroupA/.system/configs-903a2a7d45924e86a448cccd86aa67c2  9.44T  49.4K         0                             49.4K              0          0
GroupA/.system/cores                                     9.44T  22.8M         0                             22.8M              0          0
GroupA/.system/rrd-903a2a7d45924e86a448cccd86aa67c2      9.44T  49.4K         0                             49.4K              0          0
GroupA/.system/samba4                                    9.44T  9.09M         0                             9.09M              0          0
GroupA/.system/syslog-903a2a7d45924e86a448cccd86aa67c2   9.44T  10.6M         0                             10.6M              0          0
GroupA/BsdHome                                           9.44T  72.9K         0                             72.9K              0          0
GroupA/jails                                             9.44T  44.9K         0

 

[HoneyBadger]

This one's a headscratcher. There's obviously something lurking around eating up that space in the dataset, since there's no easy answer in snap or refreserv.

@danb35 - you're online, and I'm about to not be. Thoughts?

 

[garachen]

Thanks for looking at it. I'm new enough at this that I can't tell if the problem is obvious or elusive. I'm almost at the point where $20K gets me an extra 256TB and a case to put them in. Kick the can down the road 2-3 years. Because, other than the crazy space issue it works fine.

 

[wblock]

Wait... deleting files on Windows? Maybe Windows is just helpfully squirreling them away in a recycle location?

 

[garachen]

I used rsync to delete. Copied empty directory onto offending directory.

 

[HoneyBadger]

Wait... deleting files on Windows? Maybe Windows is just helpfully squirreling them away in a recycle location?

This is plausible. I recall there being an experimental setting or tunable to expose a Windows recycle bin. Make sure there isn't some hidden/system file.

 

[danb35]

@danb35 - you're online, and I'm about to not be. Thoughts?

Not in any kind of detail, I'm afraid. It rather sounds to me like it's a record size issue, but the output of zfs get all looks normal.

 

[depasseg]

It's quite possible you have a bunch of folders under the "GroupA" dataset. What's the output of "ls /mnt/GroupA"?

 

[garachen]

Yes, there are a bunch of folders there, but shouldn't that to be expected? It's these individual folders I've been measuring the size of (again, just via windows as du seems non-functional). There are about 30 folders. Are there not supposed to be folders there?

Maybe something interesting. I've succeeded in getting: find . | wc -l to return (supposedly a file+directory count) on some of the subfolders of GroupA.

On the large (size wise) folder.
Windows explorer measures 18.6 TB over 2.6M files
Du -Ahs returns 57TB and find . | wc -l returns 11M files/folders

Been checking another folder (call this folder B).
Windows explorer measures 42GB (size), 1.13 TB (size on disk) and 1M files
DU - Ahs been running about 20 hours.. nothing yet. find . | wc -l returned 87M files/folders!!

Maybe this find command isn't doing what I think. Folder B is pretty high touch from windows. People are adding and deleting files all the time on that via windows. Possibly the "deletes" aren't releasing disk space completely in freeBSD?

 

[CPLeyden1282]

Hello all,,

New to the forum, built my first NAS about a month ago and am loving FreeNAS so far. Apologies for reviving this super old thread but this seems to be the most relevant situation to what I'm experiencing currently. I seem to have a phantom 700+GB being held up somehwere but I'm unable to figure out exactly where. I am hoping someone here can help me hunt this one down.

I will start with my specs:
X10SL7-F motherboard
Xeon E3-1231 v3 processor
Fractal Design Node 804 case
5 - 4TB WD Red drives configured in a single RAID-Z2 volume
Running Freenas 11.2 (volumes created with 11.1, upgraded when the new release was out)

Some info on the problem...
volume_1/media/Downloads/Complete is a dataset that is showing 658G worth of data in use
This dataset is mounted to sabnzbd, sonarr and radarr jails.
The problem is, the directories under this dataset have little or no data. I have already considered (and subsequently deleted) snapshots.

Code:

root@freenas:/mnt/volume_1/media/Downloads # du -sh *
 17G    Complete
512B    Incomplete

Code:

root@freenas:/mnt/volume_1/media/Downloads # zfs list | grep volume_1
volume_1                                                                              8.91T  1.48T   868G  /mnt/volume_1
volume_1/Movies                                                                       4.98T  1.48T  4.91T  /mnt/volume_1/Movies
volume_1/TV                                                                           2.35T  1.48T  2.34T  /mnt/volume_1/TV
volume_1/appdata                                                                       391K  1.48T   156K  /mnt/volume_1/appdata
volume_1/appdata/sabnzbd                                                               234K  1.48T   234K  /mnt/volume_1/appdata/sabnzbd
volume_1/homedirs                                                                      284K  1.48T   284K  /mnt/volume_1/homedirs
volume_1/iocage                                                                       25.7G  1.48T  3.74M  /mnt/iocage
volume_1/iocage/appdata                                                                469K  1.48T   156K  /mnt/iocage/appdata
volume_1/iocage/appdata/database                                                       312K  1.48T   156K  /mnt/iocage/appdata/database
volume_1/iocage/appdata/database/sabnzbd                                               156K  1.48T   156K  /mnt/iocage/appdata/database/sabnzbd
volume_1/iocage/download                                                               532M  1.48T   156K  /mnt/iocage/download
volume_1/iocage/download/11.1-RELEASE                                                  260M  1.48T   260M  /mnt/iocage/download/11.1-RELEASE
volume_1/iocage/download/11.2-RELEASE                                                  272M  1.48T   272M  /mnt/iocage/download/11.2-RELEASE
volume_1/iocage/images                                                                 156K  1.48T   156K  /mnt/iocage/images
volume_1/iocage/jails                                                                 22.5G  1.48T   156K  /mnt/iocage/jails
volume_1/iocage/jails/plex                                                            4.30G  1.48T   312K  /mnt/iocage/jails/plex
volume_1/iocage/jails/plex/root                                                       4.30G  1.48T  5.25G  /mnt/iocage/jails/plex/root
volume_1/iocage/jails/radarr                                                          1.34G  1.48T   327K  /mnt/iocage/jails/radarr
volume_1/iocage/jails/radarr/root                                                     1.34G  1.48T  2.28G  /mnt/iocage/jails/radarr/root
volume_1/iocage/jails/sabnzbd-2                                                       16.3G  1.48T   170K  /mnt/iocage/jails/sabnzbd-2
volume_1/iocage/jails/sabnzbd-2/root                                                  16.3G  1.48T  17.6G  /mnt/iocage/jails/sabnzbd-2/root
volume_1/iocage/jails/sonarr                                                           558M  1.48T   312K  /mnt/iocage/jails/sonarr
volume_1/iocage/jails/sonarr/root                                                      557M  1.48T  1.49G  /mnt/iocage/jails/sonarr/root
volume_1/iocage/log                                                                    199K  1.48T   199K  /mnt/iocage/log
volume_1/iocage/releases                                                              2.61G  1.48T   156K  /mnt/iocage/releases
volume_1/iocage/releases/11.1-RELEASE                                                 1.29G  1.48T   156K  /mnt/iocage/releases/11.1-RELEASE
volume_1/iocage/releases/11.1-RELEASE/root                                            1.29G  1.48T  1.29G  /mnt/iocage/releases/11.1-RELEASE/root
volume_1/iocage/releases/11.2-RELEASE                                                 1.32G  1.48T   156K  /mnt/iocage/releases/11.2-RELEASE
volume_1/iocage/releases/11.2-RELEASE/root                                            1.32G  1.48T  1.32G  /mnt/iocage/releases/11.2-RELEASE/root
volume_1/iocage/templates                                                              156K  1.48T   156K  /mnt/iocage/templates
volume_1/media                                                                         730G  1.48T  5.10G  /mnt/volume_1/media
volume_1/media/Downloads                                                               725G  1.48T   156K  /mnt/volume_1/media/Downloads
volume_1/media/Downloads/Complete                                                      725G  1.48T   725G  /mnt/volume_1/media/Downloads/Complete
volume_1/media/Downloads/Incomplete                                                    156K  1.48T   156K  /mnt/volume_1/media/Downloads/Incomplete

Code:

root@freenas:/mnt/volume_1/media/Downloads # zfs list -o space -r volume_1
NAME                                        AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
volume_1                                    1.49T  8.91T         0    868G              0      8.06T
volume_1/Movies                             1.49T  4.99T         0   4.91T              0      75.5G
volume_1/TV                                 1.49T  2.35T         0   2.34T              0      1.25G
volume_1/appdata                            1.49T   391K         0    156K              0       234K
volume_1/appdata/sabnzbd                    1.49T   234K         0    234K              0          0
volume_1/homedirs                           1.49T   284K         0    284K              0          0
volume_1/iocage                             1.49T  10.3G         0   3.74M              0      10.2G
volume_1/iocage/appdata                     1.49T   469K         0    156K              0       312K
volume_1/iocage/appdata/database            1.49T   312K         0    156K              0       156K
volume_1/iocage/appdata/database/sabnzbd    1.49T   156K         0    156K              0          0
volume_1/iocage/download                    1.49T   532M         0    156K              0       532M
volume_1/iocage/download/11.1-RELEASE       1.49T   260M         0    260M              0          0
volume_1/iocage/download/11.2-RELEASE       1.49T   272M         0    272M              0          0
volume_1/iocage/images                      1.49T   156K         0    156K              0          0
volume_1/iocage/jails                       1.49T  7.11G         0    156K              0      7.11G
volume_1/iocage/jails/plex                  1.49T  4.30G         0    312K              0      4.30G
volume_1/iocage/jails/plex/root             1.49T  4.30G         0   4.30G              0          0
volume_1/iocage/jails/radarr                1.49T  1.34G         0    327K              0      1.34G
volume_1/iocage/jails/radarr/root           1.49T  1.34G         0   1.34G              0          0
volume_1/iocage/jails/sabnzbd-2             1.49T   949M         0    170K              0       949M
volume_1/iocage/jails/sabnzbd-2/root        1.49T   949M         0    949M              0          0
volume_1/iocage/jails/sonarr                1.49T   558M         0    312K              0       557M
volume_1/iocage/jails/sonarr/root           1.49T   557M         0    557M              0          0
volume_1/iocage/log                         1.49T   199K         0    199K              0          0
volume_1/iocage/releases                    1.49T  2.61G         0    156K              0      2.61G
volume_1/iocage/releases/11.1-RELEASE       1.49T  1.29G         0    156K              0      1.29G
volume_1/iocage/releases/11.1-RELEASE/root  1.49T  1.29G      185K   1.29G              0          0
volume_1/iocage/releases/11.2-RELEASE       1.49T  1.32G         0    156K              0      1.32G
volume_1/iocage/releases/11.2-RELEASE/root  1.49T  1.32G     4.13M   1.32G              0          0
volume_1/iocage/templates                   1.49T   156K         0    156K              0          0
volume_1/media                              1.49T   730G         0   5.10G              0       725G
volume_1/media/Downloads                    1.49T   725G         0    156K              0       725G
volume_1/media/Downloads/Complete           1.49T   725G         0    725G              0          0
volume_1/media/Downloads/Incomplete         1.49T   156K         0    156K              0          0

 

[CPLeyden1282]

Looks like a system reboot solved this problem. If anyone has any ideas what may have caused this, I'm still interested in hearing some theories. If not, thank you all anyway!