Shrink zvol of VM

[zierbeek]

Hi all

Question:
Currently, according to Ubuntu, it's size is 74.7% of 43.43GB which would fit on my ssd to improve performance. Truenas on the other hand reports 101.57 GiB. Where would this discrepancy come from? The snapshots are deleted through the ui. Does this mean I can reduce the size of my Zvol? How should I do that?

according to a post of Patrick, I should follow these guidlines

• Make sure compression is on for the zvol
• Make sure refreservation is none/off/disabled for the zvol - I don't know the precise value from the top of my head
• Fill the disk with zeros from inside the guest OS

So yes, compression is turned on. Refreservation gave 102G so is definitely turned on. This is where I'm stuck at the moment. I can't clone the vm to this spot since it's to big. I can move it to another drive. would that be the zfs send and receive command?

 

[zierbeek]

changed question and added more info

 

[Patrick M. Hausen]

Try zfs inherit -S refreservation pool/path/to/zvol to clear the reservation and turn the zvol into a "sparse volume". Moving the zvol would mean copying with zfs send | zfs receive, yes.

 

[zierbeek]

Try zfs inherit -S refreservation pool/path/to/zvol to clear the reservation and turn the zvol into a "sparse volume". Moving the zvol would mean copying with zfs send | zfs receive, yes.

Hi Patrick, thank you for your help.

I tried zfs inherit -S refreservation jail/mediavm-0b6la and pushed enter but can't spot a difference atm.

On how to create a sparse volume, would that be creating a new zvol and zfs send and receive? How would that go since I don't have a lot of spare room on that drive?

 

[Patrick M. Hausen]

If refreservation is not set to a fixed size but to "none", the volume *is* sparse. It only takes up space for blocks the guest has written to. If you create a new volume, the initial refreservation is equivalent to the volume size, so it takes up all the assigned space.

Of course you should not use a copy-on-write filesystem like ZFS inside the VM for the guest but UFS or Ext4 for FreeBSD or Linux, respectively. Otherwise the guest OS will eventually write to every single block of the virtual disk expanding the zvol to the maximum size.

If you have some manuvering space, you can fill the volume with a file containing zeroes in the guest with dd, then remove that again. If you then copy the volume with zfs send | zfs receive, the compression should work its magic.

 

[zierbeek]

Allright, I have searched in the ui and saw that the size is set to 100G in the UI. The filesystem of the VM is 45G. The available space is 18G.
The VM is Ubuntu so I guess that I would fall in last scenario of creating a file?

 

[Patrick M. Hausen]

Sorry, I don't understand. How large is the zvol on disk? The "REFER" column in the ouput of zfs list. Should be smaller than the 100G size if refreservation is not set. What does zfs get refreservation pool/path/to/zvol show?

 

[zierbeek]

Sorry, I don't understand. How large is the zvol on disk? The "REFER" column in the ouput of zfs list. Should be smaller than the 100G size if refreservation is not set. What does zfs get refreservation pool/path/to/zvol show?

The zfs list shows 123G Used, 113G available and 21,2G Refer

Refreservation says 102G. The 'partition' as far as I understand of the vm is only set to 45G That is what the vm says as in use

 

[Patrick M. Hausen]

So refreservation is set to 102G? You did not clear it with the zfs inherit command? Sorry, really confused now. And what do you mean by partition? What is the volsize of that virtual disk?

See for example:

Code:

root@freenas[~]# zfs get volsize ssd/vms/wiki-pmh-disk0
NAME                    PROPERTY  VALUE    SOURCE
ssd/vms/wiki-pmh-disk0  volsize   100G     local
root@freenas[~]# zfs get refreservation ssd/vms/wiki-pmh-disk0
NAME                    PROPERTY        VALUE      SOURCE
ssd/vms/wiki-pmh-disk0  refreservation  none       default
root@freenas[~]# zfs get refer ssd/vms/wiki-pmh-disk0
NAME                    PROPERTY    VALUE     SOURCE
ssd/vms/wiki-pmh-disk0  referenced  18.4G     -

So the disk that Linux sees is 100G in size. refreservation is set to "none". And the "disk" is occupying 18.4G in my ZFS.

The reported "used" size is larger, because that sums up the zvol as well as all snapshots:

Code:

root@freenas[~]# zfs get used  ssd/vms/wiki-pmh-disk0
NAME                    PROPERTY  VALUE  SOURCE
ssd/vms/wiki-pmh-disk0  used      66.0G  -

But it's still way smaller than the nominal size of the virtual disk of 100G - "sparse volume". The only "knob" dividing sparse/"thing" from "thick" is the refreservation setting.

So the volsize is probably something we will need to re-check. Second, if that zvol is on a RAID vdev, of course the reported space can be larger taking redundancy into account. And you might suffer from write amplification if the volblocksize of the zvol and the block size of the filesystem inside the VM don't match.

I create virtual disks this way - both Ext4 and NTFS have a native blocksize of 4k:

Code:

zfs create -V 100g -o volblocksize=4k ssd/vms/some-vm-disk0
zfs inherit -S refreservation ssd/vms/some-vm-disk0

and then I make sure to use Ext4 for Linux guests. For Windows guests there isn't a choice, but NTFS is fine.

 

[zierbeek]

Allright, I have read through the first part of your helpful answers.
my code output looks like this.

Code:

root@truenas[~]# zfs get volsize jail/mediavm-0b6la
NAME                PROPERTY  VALUE    SOURCE
jail/mediavm-0b6la  volsize   100G     local
root@truenas[~]# zfs get volsize jail/mediavm-0b6la
NAME                PROPERTY          VALUE    SOURCE
jail/mediavm-0b6la    refreservation    102G     received
root@truenas[~]# zfs get refer jail/mediavm-0b6la
NAME                PROPERTY  VALUE    SOURCE
jail/mediavm-0b6la  referenced 21.2G    -    

This zvol is not on raid. My important documents are nonetheless.

When I talked about the vm, when I get into the vm itself, it reports:
Usage of /: 73.9% of 43.43GB

 

[Patrick M. Hausen]

That's only the root filesystem, possibly there are more. Try df inside the VM to see where that space is.

As for the "sparse" feature - once you clear the refreservation you should see way less space taken up. Please post the output of zfs get -r refreservation jail.

 

[zierbeek]

That's only the root filesystem, possibly there are more. Try df inside the VM to see where that space is.

As for the "sparse" feature - once you clear the refreservation you should see way less space taken up. Please post the output of zfs get -r refreservation jail.

The aim is to have my vm take 60G, I need to expand it a bit due to the space need of certain apps.
this gets me:

Code:

matthias@mediavm:~$ df -h
Filesystem                              Size  Used Avail Use% Mounted on
udev                                    3.9G     0  3.9G   0% /dev
tmpfs                                   796M  2.3M  794M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv        44G   33G  9.4G  78% /
tmpfs                                   3.9G     0  3.9G   0% /dev/shm
tmpfs                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                   3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/sda2                               976M  208M  702M  23% /boot
/dev/sda1                               511M  5.3M  506M   2% /boot/efi
/dev/loop2                               68M   68M     0 100% /snap/lxd/22526
/dev/loop1                               56M   56M     0 100% /snap/core18/2284
/dev/loop4                               62M   62M     0 100% /snap/core20/1376
/dev/loop3                               68M   68M     0 100% /snap/lxd/22753
/dev/loop0                               56M   56M     0 100% /snap/core18/2344
/dev/loop5                               44M   44M     0 100% /snap/snapd/15177
/dev/loop6                               45M   45M     0 100% /snap/snapd/15314
/dev/loop7                               62M   62M     0 100% /snap/core20/1405
//192.168.7.16/media                    477G   96K  477G   1% /media
//192.168.7.16/tank1/Photoprism/Photos  5.2T  112K  5.2T   1% /photos
//192.168.7.16/jail/ssdPhotoprism        20G  1.2G   19G   6% /ssdstorage
//192.168.7.16/tank1/Photoprism/Import  5.2T  112K  5.2T   1% /import
tmpfs                                   796M     0  796M   0% /run/user/1000

the second command gives this:

Code:

truenas% zfs get -r refreservation jail
NAME                                                     PROPERTY        VALUE      SOURCE
jail                                                     refreservation  none       default
jail/.system                                             refreservation  none       default
jail/.system/configs-7bdf26d378f44463a02df9c5bc4d0ea3    refreservation  none       default
jail/.system/cores                                       refreservation  none       default
jail/.system/rrd-7bdf26d378f44463a02df9c5bc4d0ea3        refreservation  none       default
jail/.system/samba4                                      refreservation  none       default
jail/.system/samba4@update--2021-08-21-07-20--12.0-U5    refreservation  -          -
jail/.system/samba4@update--2021-10-08-08-54--12.0-U5.1  refreservation  -          -
jail/.system/samba4@wbc-1633683477                       refreservation  -          -
jail/.system/samba4@update--2021-11-29-16-33--12.0-U6    refreservation  -          -
jail/.system/samba4@wbc-1638203741                       refreservation  -          -
jail/.system/samba4@update--2021-12-09-07-52--12.0-U6.1  refreservation  -          -
jail/.system/samba4@wbc-1639043581                       refreservation  -          -
jail/.system/samba4@wbc-1639671859                       refreservation  -          -
jail/.system/samba4@wbc-1640590508                       refreservation  -          -
jail/.system/samba4@wbc-1642578917                       refreservation  -          -
jail/.system/samba4@update--2022-02-20-10-27--12.0-U7    refreservation  -          -
jail/.system/samba4@wbc-1645353076                       refreservation  -          -
jail/.system/samba4@wbc-1650017673                       refreservation  -          -
jail/.system/samba4@wbc-1650349323                       refreservation  -          -
jail/.system/services                                    refreservation  none       default
jail/.system/syslog-7bdf26d378f44463a02df9c5bc4d0ea3     refreservation  none       default
jail/.system/webui                                       refreservation  none       default
jail/iocage                                              refreservation  none       default
jail/iocage/download                                     refreservation  none       default
jail/iocage/download/12.1-RELEASE                        refreservation  none       default
jail/iocage/download/12.2-RELEASE                        refreservation  none       default
jail/iocage/images                                       refreservation  none       default
jail/iocage/jails                                        refreservation  none       default
jail/iocage/jails/MQTT                                   refreservation  none       default
jail/iocage/jails/MQTT/root                              refreservation  none       default
jail/iocage/jails/Speedtest                              refreservation  none       default
jail/iocage/jails/Speedtest/root                         refreservation  none       default
jail/iocage/log                                          refreservation  none       default
jail/iocage/releases                                     refreservation  none       default
jail/iocage/releases/11.3-RELEASE                        refreservation  none       default
jail/iocage/releases/11.3-RELEASE/root                   refreservation  none       default
jail/iocage/releases/12.1-RELEASE                        refreservation  none       default
jail/iocage/releases/12.1-RELEASE/root                   refreservation  none       default
jail/iocage/releases/12.2-RELEASE                        refreservation  none       default
jail/iocage/releases/12.2-RELEASE/root                   refreservation  none       default
jail/iocage/templates                                    refreservation  none       default
jail/mediavm-0b6la                                       refreservation  102G       received
jail/mediavm-0b6la@VM                                    refreservation  -          -
jail/mediavm-0b6la@manual-2022-04-14_11-55               refreservation  -          -
jail/ssdPhotoprism                                       refreservation  none       local
jail/ssdPhotoprism/Storage                               refreservation  none       local

 

[Patrick M. Hausen]

zfs set refreservation=none jail/mediavm-0b6la - I did not know this is already a cloned volume, so my initial "inherit" command to clear that did not work. You also might want to remove the snapshots:

Code:

zfs destroy jail/mediavm-0b6la@VM
zfs destroy jail/mediavm-0b6la@manual-2022-04-14_11-55

As for the 44G vs. 60/100G size discrepancy inside the guest - you will need to fiddle with Linux tools so see the partition table (fdisk? parted? whatever ...). I always end up using a search engine. I really have not memorized how Linux does this stuff today.

The "disk drive" Linux sees should be 100G in size. You should be able to use that by increasing the volume group. Repeating myself: no idea how to do that with LVM but I know it can be done and the Ubuntu docs or Stackoverflow will probably be of help.

 

[zierbeek]

zfs set refreservation=none jail/mediavm-0b6la - I did not know this is already a cloned volume, so my initial "inherit" command to clear that did not work. You also might want to remove the snapshots:

Code:

zfs destroy jail/mediavm-0b6la@VM
zfs destroy jail/mediavm-0b6la@manual-2022-04-14_11-55

As for the 44G vs. 60/100G size discrepancy inside the guest - you will need to fiddle with Linux tools so see the partition table (fdisk? parted? whatever ...). I always end up using a search engine. I really have not memorized how Linux does this stuff today.

The "disk drive" Linux sees should be 100G in size. You should be able to use that by increasing the volume group. Repeating myself: no idea how to do that with LVM but I know it can be done and the Ubuntu docs or Stackoverflow will probably be of help.

Ah yeah, that was cloned since I moved my vm from hdd to ssd.

I would follow this guide https://www.truenas.com/community/r...-a-linux-vms-llvm-virtual-disk-on-a-zvol.174/

But the question is still unanswered I guess, on how to change the 100G to a lower amount.

 

[Patrick M. Hausen]

You cannot reduce the size without data loss unless you can make 100% sure that Linux uses only the first, say, 50G. Again - partition table in the guest, LVM, Linux tools ... no idea.
You can at any time set the volume size to a smaller value on the command line. That will essentially "cut" the virtual disk. If the partitions and LVM only use the first part of the disk cutting it to 60g will probably work. If the guest OS uses all of the disk, the data will be gone.
Even when successful you will probably need to at least repair the partition table. GPT has a table at the start and at the end of the disk and the one at the end will need to be recreated when you cut the disk short.

I thought you wanted to reduce the space taken up in ZFS while keeping the nominal size at 100G. Doesn't hurt, does it?

So could you successfully clear the refreservation? Does it take up less space now?

 

[zierbeek]

Yes, I get the first part!

Basically my goal is that the zvol will take up less space on the drive.
the command could not work with the inherit, as you said, no?
zfs inherit -S refreservation pool/path/to/zvol

 

[Patrick M. Hausen]

So go ahead and clear the refreservation.

 

[Patrick M. Hausen]

zfs inherit -S does not work in your case, because the value is "received". But I already wrote: zfs set refreservation=none jail/mediavm-0b6la. If that does not work, try zfs inherit refreservation jail/mediavm-0b6la (without the "-S").

 

[zierbeek]

zfs inherit does not work in your case, because the value is "received". But I already wrote: zfs set refreservation=none jail/mediavm-0b6la

apologies, found that in the docs as well.
So now, truenas ui shows:

name	type	used	available	compression
mediavm-0b6la	VOLUME	28.46 GiB	113.13 GiB	lz4	1.83

while ubuntu shows:
Usage of /: 73.9% of 43.43GB

this needs to be solved or doesn't it since it's?

 

[Patrick M. Hausen]

Sorry - without the headers how should I tell which field is which?

You can use zfs get refreservation|refer|used (three separate commands) to check the three relevant values. Also zpool list should show less space taken, now.