SOLVED Why does a replicated zvol take up more space than the source?

[Patrick M. Hausen]

Hi all,

Code:

zfs list -t all
[...]
vm/windows											27.9G   158G  27.9G  -
vm/windows@auto-20180205.0130-1w					  35.9M	  -  27.9G  -
[...]
zfs/vmbackup/windows								  40.3G  5.13T  40.3G  -
zfs/vmbackup/windows@auto-20180205.0130-1w				0	  -  40.3G  -

If this is counting the raw disk space including redundancy within the vdev, the numbers still don't add up. vm is a partition==vdev stripe without redundancy, zfs is a four disk RAIDZ2 pool. So according to what I know the numbers should be identical or the copy should be twice the size of the source. What am I missing?

Thanks,
Patrick

 

[MrToddsFriends]

What am I missing?

AFAIK redundancy isn't counted in into these numbers. Some other details that might have an impact:

• different compression settings across datasets
• existence of many small files combined with different minimum allocation sizes across pools
• different deduplication settings across pools

 

[rs225]

Or in this case, also the block size.

 

[Patrick M. Hausen]

OK, let's check ... ;)

Code:

root@freenas-pmh:~ # zfs get compression vm/windows zfs/vmbackup/windows
NAME				  PROPERTY	 VALUE	 SOURCE
vm/windows			compression  lz4	   inherited from vm
zfs/vmbackup/windows  compression  lz4	   inherited from zfs/vmbackup
root@freenas-pmh:~ # zfs get dedup vm/windows zfs/vmbackup/windows
NAME				  PROPERTY  VALUE		  SOURCE
vm/windows			dedup	 off			default
zfs/vmbackup/windows  dedup	 off			inherited from zfs
root@freenas-pmh:~ # zfs get volblocksize vm/windows zfs/vmbackup/windows
NAME				  PROPERTY	  VALUE	 SOURCE
vm/windows			volblocksize  4K		-
zfs/vmbackup/windows  volblocksize  4K		-

How do I get the minimum allocation size of the respective pools?

Edit - some more numbers:

vm/windows:

Code:

NAME		PROPERTY			  VALUE				  SOURCE
vm/windows  used				  28.3G				  -
vm/windows  referenced			27.9G				  -
vm/windows  compressratio		 1.21x				  -
vm/windows  reservation		   none				   default
vm/windows  volsize			   60G					local
vm/windows  volblocksize		  4K					 -
vm/windows  refreservation		none				   default
vm/windows  usedbysnapshots	   412M				   -
vm/windows  usedbydataset		 27.9G				  -
vm/windows  refcompressratio	  1.21x				  -
vm/windows  written			   39.3M				  -
vm/windows  logicalused		   28.1G				  -
vm/windows  logicalreferenced	 27.7G				  -
vm/windows  volmode			   default				default

zfs/vmbackup/windows:

Code:

NAME				  PROPERTY					  VALUE						 SOURCE
zfs/vmbackup/windows  used						  40.8G						 -
zfs/vmbackup/windows  referenced					40.3G						 -
zfs/vmbackup/windows  compressratio				 1.21x						 -
zfs/vmbackup/windows  reservation				   none						  default
zfs/vmbackup/windows  volsize					   60G						   local
zfs/vmbackup/windows  volblocksize				  4K							-
zfs/vmbackup/windows  refreservation				none						  default
zfs/vmbackup/windows  usedbysnapshots			   520M						  -
zfs/vmbackup/windows  usedbydataset				 40.3G						 -
zfs/vmbackup/windows  refcompressratio			  1.21x						 -
zfs/vmbackup/windows  written					   0							 -
zfs/vmbackup/windows  logicalused				   28.1G						 -
zfs/vmbackup/windows  logicalreferenced			 27.7G						 -
zfs/vmbackup/windows  volmode					   default					   default

So the logicalused and logicalreferenced numbers at least match.

Kind regards,
Patrick

 

[MrToddsFriends]

How do I get the minimum allocation size of the respective pools?

Sorry for being vague here. I overlooked the "zvol" part of your question and did not name the ashift property I had in mind. I don't think that the value of ashift is relevant in your case, but anyway:

Code:

~ # zdb  | grep -i ashift
			ashift: 9
~ # zdb -U /data/zfs/zpool.cache | grep -i ashift
			ashift: 12
			ashift: 12
			ashift: 12

The first of these two commands shows the value of ashift for freenas-boot and the second shows it for all other pools in the system (handled by cachefile /data/zfs/zpool.cache, which is the default for FreeNAS data pools). Some more info here:
http://open-zfs.org/wiki/Performance_tuning#Alignment_Shift_.28ashift.29

 

[Patrick M. Hausen]

Yeah, I know ashift ;)

freenas-boot: 9
vm: 12
zfs: 12

Kind regards,
Patrick

 

[droeders]

I had the exact same questions a while ago. The best reference I could find was:

https://forums.freebsd.org/threads/62593/#post-361615

TL;DR: The pool types and volblocksize (as @rs225 alluded to) are the biggest factors here.

 

[MrToddsFriends]

@jgreco gave an explanation in this posting, with the abstract in the first sentence: "You shouldn't be using RAIDZ with block storage at all, because RAIDZ uses a variable amount of parity to protect your data depending on the block size, and this can cause excessive waste."

 

[Patrick M. Hausen]

*patsch*

That's German onomatopoetic for my hand hitting my forehead ;)

I completely forgot about the redundancy of the RAIDZ2 target pool. And when in doubt ZFS numbers are always raw capacity ;)

So for backup purposes it would probably be better to send/receive the snapshots to regular files on some dataset on the target pool ... I'll try and report back.

Patrick

 

[Patrick M. Hausen]

OK, now the numbers make more sense:

Source:

Code:

vm/windows@auto-20180208.0130-1w					  16.8G	  -  28.3G  -

Destination: (regular file within that dataset)

Code:

zfs/testbackup										22.3G  5.02T  22.3G  /mnt/zfs/testbackup

Which immediately leads to the next question:

The destination is actually smaller than the source. The source compression ratio is 1.07, the destination 1.34. I used sdelete -z c: to fill the free space of the Windows "hard disk" with zeroes. I expected ZFS compression to kick in and the original zvol to actually shrink. Which it did not. Why is this the case? All those zero-filled blocks should compress quite nicely?

The source volblocksize is 4k to meet the NTFS blocksize. The destination blocksize is the regular 128k.

Is the assumption that you should create a zvol for a VM with the same blocksize as the filesystem you are going to put on top inside the VM wrong? I used to always lookup what the filesystems used and pick e.g. 16k for FreeBSD/UFS and 4k for Linux/Ext4 or Windows/NTFS ...

Kind regards,
Patrick

 

[Bidule0hm]

Here's your answer:

The source volblocksize is 4k to meet the NTFS blocksize. The destination blocksize is the regular 128k.

See this thread https://forums.freenas.org/index.php?threads/misaligned-pools-and-lost-space.40288/ for more details :)

 

[Patrick M. Hausen]

Hmmm ... I did know quite a bit of your posts in that thread. Thanks a lot for refreshing my memory. I'm still missing a similar explanation for zvols. Or did I overlook anything?

The OpenZFS wiki has this to say about the volblocksize:
http://www.open-zfs.org/wiki/Performance_tuning#zvol_volblocksize

Which supports my assumption that for a virtual hard disk with e.g. 4k writes, a matching volblocksize would be best. Or doesn't it? ;) Isn't a virtual disk similar to swap in that regard?

Thanks,
Patrick

 

[Bidule0hm]

Maybe I'm wrong but I assume the zvols blocksize works more or less the same way as the dataset recordsize and so 4k is the best for performance but not for space.

 

[Patrick M. Hausen]

So for backup purposes it would probably be better to send/receive the snapshots to regular files on some dataset on the target pool ... I'll try and report back.

Quoting myself in case somebody stumbles on this through search ...

Indeed for archiving purposes a file created by zfs receive has got the lowest overhead and can be most easily moved around.

For recurring backup that does not work because you lose the ability to sync incremental snapshots ...

Patrick