Discrepancy between zfs list Used and du?

[victorhooi]

I have a ZFS dataset - if I use `zfs list`, it reports as only 2.85 TB used:

NAME USED AVAIL REFER MOUNTPOINT
datastore/ablage 2.85T 14.4T 2.85T /mnt/datastore/ablage

However, I use do a du on it - it reports as 10TB used (which is what I expected):

freenas-naulty-place# cd /mnt/datastore
freenas-naulty-place# du -chs ablage
10T ablage
10T total

Is there an explanation for the discrepancy? Why is zfs list reporting less space used than the files?

(I do have lz4 compression - however, I checked the compression ratio and it's 1.0x:


freenas-naulty-place# zfs get all datastore/ablage
NAME PROPERTY VALUE SOURCE
datastore/ablage type filesystem -
datastore/ablage creation Mon May 7 6:07 2018 -
datastore/ablage used 2.85T -
datastore/ablage available 14.3T -
datastore/ablage referenced 2.85T -
datastore/ablage compressratio 1.00x -
datastore/ablage mounted no -
datastore/ablage quota none local
datastore/ablage reservation none local
datastore/ablage recordsize 128K default
datastore/ablage mountpoint /mnt/datastore/ablage default
datastore/ablage sharenfs off default
datastore/ablage checksum on default
datastore/ablage compression lz4 local
datastore/ablage atime off local
datastore/ablage devices on default
datastore/ablage exec on default
datastore/ablage setuid on default
datastore/ablage readonly off default
datastore/ablage jailed off default
datastore/ablage snapdir hidden default
datastore/ablage aclmode passthrough local
datastore/ablage aclinherit passthrough inherited from datastore
datastore/ablage canmount on default
datastore/ablage xattr on default
datastore/ablage copies 1 local
datastore/ablage version 5 -
datastore/ablage utf8only off -
datastore/ablage normalization none -
datastore/ablage casesensitivity sensitive -
datastore/ablage vscan off default
datastore/ablage nbmand off default
datastore/ablage sharesmb off default
datastore/ablage refquota none local
datastore/ablage refreservation none local
datastore/ablage primarycache all default
datastore/ablage secondarycache all default
datastore/ablage usedbysnapshots 0 -
datastore/ablage usedbydataset 2.85T -
datastore/ablage usedbychildren 0 -
datastore/ablage usedbyrefreservation 0 -
datastore/ablage logbias latency default
datastore/ablage dedup off local


 

[MrToddsFriends]

However, I use do a du on it - it reports as 10TB used (which is what I expected):

freenas-naulty-place# cd /mnt/datastore
freenas-naulty-place# du -chs ablage
10T ablage
10T total

What's the output of du -chsx /mnt/datastore/ablage? -x for "File system mount points are not traversed".

What makes you think that the output of du is correct while the output of the zfs command is not? With ZFS file systems it's usually the other way around.

 

[victorhooi]

Output of du -chsx is here:

freenas-naulty-place# du -chsx /mnt/datastore/ablage
10T /mnt/datastore/ablage
10T total


 

[Ericloewe]

it reports as 10TB used (which is what I expected)

How do you figure? ZFS doesn't lie.

 

[DaveY]

What does df -h give you for size?

 

[victorhooi]

Ok - it gets weirder.

I rebooted the machine - and now "du" was reporting around 3 TB, not 4TB for that particular directory.

I know I did try a few methods of syncing data between the two FreeNAS machines - e.g. zfs send/recv, then some rsync - but I don't know if that would explain data somehow been caught in some in-between stage?

Anyhow, I'm continuing to sync data across - but ZFS list shows me as having used up 28 TB - yet if I add up the columns it's less than that:

Have I somehow got data caught in some purgatory or something? Any idea

Code:

freenas-naulty-place# zfs list
NAME														 USED  AVAIL  REFER  MOUNTPOINT
datastore												   28.0T  5.68T  10.2T  /mnt/datastore
datastore/.system											566M  5.68T   538M  legacy
datastore/.system/configs-e5e9010fa67d4405afd69696284bdd17  1.22M  5.68T  1.22M  legacy
datastore/.system/cores									  671K  5.68T   671K  legacy
datastore/.system/rrd-e5e9010fa67d4405afd69696284bdd17	  24.5M  5.68T  24.5M  legacy
datastore/.system/samba4									 390K  5.68T   390K  legacy
datastore/.system/syslog-e5e9010fa67d4405afd69696284bdd17	639K  5.68T   639K  legacy
datastore/.system/webui									  141K  5.68T   141K  legacy
datastore/ablage											7.32T  5.68T  7.32T  /mnt/datastore/ablage
datastore/comics											4.39G  5.68T  4.39G  /mnt/datastore/comics
datastore/documents										 5.43T  5.68T  5.43T  /mnt/datastore/documents
datastore/ebooks											 141K  5.68T   141K  /mnt/datastore/ebooks
datastore/jails											 61.8G  5.68T  61.8G  /mnt/datastore/jails
datastore/music											 6.69G  5.68T  6.69G  /mnt/datastore/music
datastore/photos											 310G  5.68T   310G  /mnt/datastore/photos
datastore/software										  70.8G  5.68T  70.8G  /mnt/datastore/software
datastore/synology										  3.67T  5.68T  3.67T  /mnt/datastore/synology
datastore/transmission-downloads							 494G  5.68T   494G  /mnt/datastore/transmission-downloads
datastore/videos											 419G  5.68T   419G  /mnt/datastore/videos
freenas-boot												 875M  54.4G	64K  none
freenas-boot/ROOT											872M  54.4G	29K  none
freenas-boot/ROOT/Initial-Install							  1K  54.4G   870M  legacy
freenas-boot/ROOT/default									872M  54.4G   870M  legacy

Thanks,
Victor

 

[Ericloewe]

Do you have reservations on any of those sub-datasets? The ZFS man page suggests that reservations on child datasets are accounted for in the parent dataset's used property

 

[victorhooi]

Nope, I don't believe I've ever setup a ZFS reservation on them.

Is there any way an aborted rsync or zfs send/recv could have left half done, or temporary files around?

 

[Ericloewe]

Is there any way an aborted rsync or zfs send/recv could have left half done, or temporary files around?

No. A ZFS pool is always consistent.

Let's dig deeper. Please post the output of zfs list -r -o space datastore.

 

[victorhooi]

Of course - here is the output:

Code:

freenas-naulty-place# zfs list -r -o space datastore
NAME														AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
datastore												   4.97T  28.7T		 0   10.2T			  0	  18.5T
datastore/.system										   4.97T   566M		 0	538M			  0	  27.6M
datastore/.system/configs-e5e9010fa67d4405afd69696284bdd17  4.97T  1.22M		 0   1.22M			  0		  0
datastore/.system/cores									 4.97T   671K		 0	671K			  0		  0
datastore/.system/rrd-e5e9010fa67d4405afd69696284bdd17	  4.97T  24.6M		 0   24.6M			  0		  0
datastore/.system/samba4									4.97T   403K		 0	403K			  0		  0
datastore/.system/syslog-e5e9010fa67d4405afd69696284bdd17   4.97T   639K		 0	639K			  0		  0
datastore/.system/webui									 4.97T   141K		 0	141K			  0		  0
datastore/ablage											4.97T  7.71T		 0   7.71T			  0		  0
datastore/comics											4.97T  4.39G		 0   4.39G			  0		  0
datastore/documents										 4.97T  5.43T		 0   5.43T			  0		  0
datastore/ebooks											4.97T   141K		 0	141K			  0		  0
datastore/jails											 4.97T  61.8G		 0   61.8G			  0		  0
datastore/music											 4.97T  6.69G		 0   6.69G			  0		  0
datastore/photos											4.97T   310G		 0	310G			  0		  0
datastore/software										  4.97T  70.8G		 0   70.8G			  0		  0
datastore/synology										  4.97T  4.00T		 0   4.00T			  0		  0
datastore/transmission-downloads							4.97T   494G		 0	494G			  0		  0
datastore/videos											4.97T   419G		 0	419G			  0		  0

This is currently a 6 x 8 TB RAID-Z1 zpool.

 

[Ericloewe]

There's your mystery data, 10.2TB in the top-level dataset.

 

[victorhooi]

Oh, that's weird - there shouldn't be anything in the top-level dataset (datastore) itself - almost everything should be in the child ones.

I'll check it out myself tonight, and see if something went wrong or it copied something to the wrong location.

The problem now is my rsync has pushed my ZFS used space above 80% =(. I've been told that this is bad.

Even after I clear the space - is there any way to fix things, or defragment them, to restore performance?

 

[Ericloewe]

Even after I clear the space - is there any way to fix things, or defragment them, to restore performance?

No but it's probably not needed.

 

[victorhooi]

Hmm - the rsync is still running though and I'm down to around 2TB free - that is not good (I presume).

I've checked the contents of /mnt/datastore - each folder should match up with a dataset (as in, I don't have anything directly under the root:

Code:

freenas-naulty-place# ls -lah
total 239
drwxr-xr-x   13 victorhooi  wheel	15B May 13 23:36 .
drwxr-xr-x	3 root		wheel   128B May 13 22:40 ..
-rwxr-xr-x	1 victorhooi  wheel	 0B May  7 06:08 .apple
-rwxr-xr-x	1 victorhooi  wheel	 0B May  7 06:12 .mac
drwxr-xr-x   17 victorhooi  wheel	21B Nov 20  2016 ablage
drwxr-xr-x   13 victorhooi  wheel	23B Jun 25  2017 comics
drwxr-xr-x   62 1001		wheel	71B Feb  6 21:25 documents
drwxr-xr-x	2 victorhooi  wheel	 3B May  7 06:09 ebooks
drwxr-xr-x	7 root		wheel	 8B May 11 19:46 jails
drwxr-xr-x   26 1001		wheel	38B Dec 25 09:40 music
drwxr-xr-x   26 1001		wheel	31B Dec  2 16:26 photos
drwxr-xr-x   20 1001		wheel	23B Jan 18 22:00 software
drwxrwxrwx  138 root		wheel   145B May 12 05:47 synology
drwxr-xr-x	3 root		wheel	 4B May 11 16:49 transmission-downloads
drwxr-xr-x   17 1001		wheel	19B Oct 21  2016 videos

I also ran du directly from the root - and it reports 21 TB used?

Code:

freenas-naulty-place# du -chs *
8.9T	ablage
4.4G	comics
5.4T	documents
1.0K	ebooks
 62G	jails
6.7G	music
310G	photos
 71G	software
5.0T	synology
494G	transmission-downloads
419G	videos
 21T	total

This is the latest zfs list:

Code:

freenas-naulty-place# zfs list -r -o space datastore
NAME														AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
datastore												   2.82T  30.8T		 0   10.2T			  0	  20.6T
datastore/.system										   2.82T   565M		 0	538M			  0	  26.8M
datastore/.system/configs-e5e9010fa67d4405afd69696284bdd17  2.82T  1.34M		 0   1.34M			  0		  0
datastore/.system/cores									 2.82T   671K		 0	671K			  0		  0
datastore/.system/rrd-e5e9010fa67d4405afd69696284bdd17	  2.82T  23.6M		 0   23.6M			  0		  0
datastore/.system/samba4									2.82T   403K		 0	403K			  0		  0
datastore/.system/syslog-e5e9010fa67d4405afd69696284bdd17   2.82T   690K		 0	690K			  0		  0
datastore/.system/webui									 2.82T   141K		 0	141K			  0		  0
datastore/ablage											2.82T  8.88T		 0   8.88T			  0		  0
datastore/comics											2.82T  4.39G		 0   4.39G			  0		  0
datastore/documents										 2.82T  5.43T		 0   5.43T			  0		  0
datastore/ebooks											2.82T   141K		 0	141K			  0		  0
datastore/jails											 2.82T  61.8G		 0   61.8G			  0		  0
datastore/music											 2.82T  6.69G		 0   6.69G			  0		  0
datastore/photos											2.82T   310G		 0	310G			  0		  0
datastore/software										  2.82T  70.8G		 0   70.8G			  0		  0
datastore/synology										  2.82T  4.97T		 0   4.97T			  0		  0
datastore/transmission-downloads							2.82T   494G		 0	494G			  0		  0
datastore/videos											2.82T   419G		 0	419G			  0		  0

 

[Ericloewe]

Sounds like you might have directories with the same name as some of the datasets, if you can't see your mystery data. You'll want to look into that and ensure you're using the datasets and not the folders.

 

[victorhooi]

Aha, I hadn't thought of that?

But how is that possible? And how exactly do you check for it?

As in, I ran "du" from /mnt/datastore - I thought that would add up all the space under that. Everything else checks out - as in, if you compare the "USEDDS" column from zfs list against the du output, it all matches up, except for the mystery 10TB in datastore itself?

 

[victorhooi]

Mystery solved (I think).

Eric may be right - I must have had a directory called ablage under /mnt/dataset - that the child dataset ablage then mounted on top of.

I ran:

Code:

zfs unmount datastore/ablage

And found I still had a /mnt/datastore/ablage directory. I moved that out of the way and remounted the child dataset, and ran du again:

Code:

freenas-naulty-place# mv ablage ablage_old
freenas-naulty-place# zfs mount datastore/ablage
freenas-naulty-place# ls -lah
total 252
drwxr-xr-x   14 victorhooi  wheel	16B May 17 08:02 .
drwxr-xr-x	3 root		wheel   128B May 17 06:59 ..
-rwxr-xr-x	1 victorhooi  wheel	 0B May  7 06:08 .apple
-rwxr-xr-x	1 victorhooi  wheel	 0B May  7 06:12 .mac
drwxr-xr-x   17 victorhooi  wheel	21B Nov 20  2016 ablage
drwxr-xr-x   17 1001		wheel	20B Nov 20  2016 ablage_old
drwxr-xr-x   13 victorhooi  wheel	23B Jun 25  2017 comics
drwxr-xr-x   62 1001		wheel	71B Feb  6 21:25 documents
drwxr-xr-x	2 victorhooi  wheel	 3B May  7 06:09 ebooks
drwxr-xr-x	7 root		wheel	 8B May 11 19:46 jails
drwxr-xr-x   26 1001		wheel	38B Dec 25 09:40 music
drwxr-xr-x   26 1001		wheel	31B Dec  2 16:26 photos
drwxr-xr-x   20 1001		wheel	23B Jan 18 22:00 software
drwxrwxrwx  138 root		wheel   145B May 12 05:47 synology
drwxr-xr-x	3 root		wheel	 4B May 11 16:49 transmission-downloads
drwxr-xr-x   17 1001		wheel	19B Oct 21  2016 videos
freenas-naulty-place# du -chs ablage*
8.9T	ablage
 10T	ablage_old
 19T	total

So I can delete /mnt/datastore/ablage_old.

My next question is - because of this I got the ZFS pool up to 88% usage =( with only a few TB free.

I know Eric you mentioned it should be ok (but that was back when it was 80% usage) - is it going to be an issue now that it's nearly 90%? As in, after deleting the excess data will it be back to before, or is there permanent degradation. I'm happy to blow it away and do a fresh rsync if that's better.

I'm really glad you helped solve the mystery =).

 

[Ericloewe]

Don't worry about it unless you actually measure a problem. Just get rid of what is undesired and worry about fixing performance later if it's revealed to be necessary.