decreasing space due to snapshots

[TheShellshock67]

Hello everyone,


I came across these errors last week.

But first my system's specs:

Code:

Build number : FreeNAS-9.10.2-U5 (561f0d7a1)

Motherboard make and model
-AsRock z170m extreme 4

CPU make and model
-Intel Celeron G3900

RAM quantity
-64GB ECC Kingston

Hard drives, quantity, model numbers, and RAID configuration
-8x Toshiba DT01ACA300 in RAIDZ2 & 1x Crucial MX300 525GB as a L2ARC & 2x Samsung 750 Evo 120GB as SLOG

Hard disk controllers
-IBM M1015 or the LSI equivalent one in IT mode

network cards
-The one built-in to the motherboard (Intel L219V) and a Dell 0yt674 (Intel Gigabit VT Quad port)

The Error :
##TIMEADNDATE## manage.py: [middleware.exceptions:37] [MiddlewareError: Snapshot could not be taken: cannot create snapshot '#POOLNAME#/#ZVOLNAME#@#Periodicsnapshotname#-#DATE#': out of space
no snapshots were created

I was surprised when I read this in a mail delivered to my phone.
As I was sure my server had at least 2 TiB of free space.
Well, at least that's what I thought I saw the evening before.

As soon as I got home I sat down next to the server.
opened the WebGUI and went to storage...... 101GiB Free on the parent dataset ... more on the Volume but I forgot the exact size.
okay.. maybe the disk is actually full.

As this is my personal storage machine and since there isn't much data on it that is not backed up already I went ahead and deleted all the snapshots.
(I know this is not to be done on production machines, but trust me there is nothing critical on this machine)
Okay, promising 2.7TiB of free space on the parent dataset and 9.4TiB on the Volume

I went ahead and stopped ALL access to the storage server.
There shouldn't be any activity on the pool now.
So... There shouldn't be any blocks (at least not much) being written to the pool right?
Created a manual snapshot and again watched the storage go nearly full....

This didn't compute with me and I stared at the screen for a couple of minutes....thinking ... snapshots shouldn't take that much space when there is almost no change happening on the pool.

Am I right? or am I a complete Noob that doesn't know anything about this level of rocket science?
(might be the latter :-( )
Did I miss anything?
Where should you search?
How to troubleshoot this any further?

 

[styno]

You are right, ZFS is a COW (copy on write) filesystem. This basically means that a snapshot only starts using space when data on the filesystem is altered.
In your case I would guess that something is adding and deleting data in a loop to explain a non-snapshotted filesystem not to fill up but a snapshotted one does. (maybe a native speaker can explain that one more clearly ;) )

 

[rs225]

Somebody reported ESXi making a lot of writes to iSCSI recently.

 

[TheShellshock67]

Somebody reported ESXi making a lot of writes to iSCSI recently.

I do not use ESXi, i use Hyper-v via iSCSI.

But the weird thing is that i made sure everything that could write to the pool was off.
There shouldn't be any activity (especially not 2 TiB) between the moment i press the snapshot button and the refresh of the page.
It is not even possible to write 2 TiB in a couple of seconds, my server is not that fast.

 

[rs225]

It sounds like a UI or accounting glitch. It is pretty unlikely you could write that much.

Could there be a space reservation that is somehow causing this?