Used space up to 100% single zvol in the pool

[bojan]

Hey everyone,
it seems like I'm having some troubles which I can't sort out myself.

Hardware:

• Truenas Mini X+
• 5x 6TB in RAID-Z2
• Single VDEV with all 5 drives

Setup:

• single pool with this one VDEV
• one ZVOL used as iSCSI to a Windows server
• snapshots enabled

When I logged in to increase the size of the ZVOL (currently 6TB) to 7TB I was presented with 99% usage of the pool. This pool has only this one ZVOL and nothing else. There were few hundred snapshots (I've deleted some of them to get some breathing room) but the size of the snapshots are up to 4GB. Most of them are in MBs and only 2 are 1.7 and 4.something GB.
I can't seem to be able to wrap my head around the "missing space". Drives in Z2 are 15.2 TB where the referenced value for the snapshot is 7.1 TB. Where's the remaining 60% of the space, 8.2TB?!

Please help:) Thanks in advance

 

[Arwen]

Could you supply the following for your zVol?

zfs get volblocksize HDD_POOL/SHARED_FOLDER

And this for your top level pool dataset?

zfs get recordsize HDD_POOL

Others may want some other information.

 

[jgreco]

RAIDZ uses a variable amount of parity, you are not guaranteed a particular amount of space.

See these two articles

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files. ZFS will allocate long, contiguous stretches of disk...

www.truenas.com

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

 

[bojan]

Could you supply the following for your zVol?

zfs get volblocksize HDD_POOL/SHARED_FOLDER

And this for your top level pool dataset?

zfs get recordsize HDD_POOL

Others may want some other information.

Code:

 zfs get volblocksize HDD_POOL/SHARED_FOLDER
NAME                    PROPERTY      VALUE     SOURCE
HDD_POOL/SHARED_FOLDER  volblocksize  16K       -

Code:

zfs get recordsize HDD_POOL
NAME      PROPERTY    VALUE    SOURCE
HDD_POOL  recordsize  128K     default

 

[Arwen]

Part of the problem, and I am no expert in this aspect of ZFS, is that a zVol using 16K blocks wants to write to a stripe of 3 disks, (5 disks in a RAID-Z2, means up to 3 data disks). This is somewhat an un-even number. Assuming 4KByte disk block sizes, 16KByte zVol blocks wants to write to FOUR / 4 disks. But, that won't work. Thus, probably 3 x 8Kbyte blocks from the disks, (24KByte), which wastes 16KBytes per block. That works out to 33% waste.

Or perhaps ZFS is writing 8KBytes to just 2 data disks, (besides the 2 parity disks).

This is one reason why Mirrored vDevs are recommended for VM storage.

Now of course, my numbers could be wildly off, and ZFS could be much smarter, (the ZFS developers have added new features recently).

So don't take my word, read what @jgreco linked.