iSCSI zvol Free Space

[CompMaster]

Hello All,
I fear I may be having a space issue with my FreeNAS Server. I will try to give as much detailed information as possible, if I am missing anything please feel free to know and I will get that posted to the thread as quickly as possible.


24 Disks @ 4TB
4 vdevs in Raid-2z
Total Usable should be 64TB

Using a zvol iSCSI Extent with 21TB assigned to it.
df –h shows:
vol/0 with 18T of free space left.

There are some other things that show under vol/0 such as Jails and other system things that appeared from FreeNAS 9.3, but none of them hold any consider amount of data in it. If you wish to see the whole print out I can post it.

I have a Windows 2012 R2 Server that attaches to this iSCSI zvol and shows 20.9TB allocated to it.
Can I really have 25TB of block space being chewed up? I knew it would grow, but I think it being that big after 1.5 years is a little strange. I know the recommended is allocate only 50-70% of the pools overale usable storage to the zvol, which I believe I am well below.

Is my math off? Am I even doing the math correctly? I certainly am not going to pretend I am a FreeNAS expert but I do try to do my homework and research.

Is iSCSI writing 0’s back for data being deleted from the drive in Windows? (UNMAP or sdelete?)

Appreciate any feedback from anyone. Again, if you need more info I would be more than happy to provide it.

 

[mav@]

Your math looks fine, you should get something about 64TB of available space gross. But remember that there is a difference between GB and GiB, which at those sizes becomes quite significant.

`df -h` is a bad tool for ZFS. You should use `zpool list` and `zfs list` instead. Or the FreeNAS Web UI, which shows compilation of both.

What's about UNMAP -- Windows 2012 supports UNMAP and should be able free space when files are deleted. But I would test it to be sure before deployment, just to know what to expect, that is about Microsoft. :) UNMAP is definitely useful if you want to go higher on the extent size.

 

[CompMaster]

Your math looks fine, you should get something about 64TB of available space gross. But remember that there is a difference between GB and GiB, which at those sizes becomes quite significant.

`df -h` is a bad tool for ZFS. You should use `zpool list` and `zfs list` instead. Or the FreeNAS Web UI, which shows compilation of both.

What's about UNMAP -- Windows 2012 supports UNMAP and should be able free space when files are deleted. But I would test it to be sure before deployment, just to know what to expect, that is about Microsoft. :) UNMAP is definitely useful if you want to go higher on the extent size.

Thanks for the reply. I have attached screen shots of both those commands. Numbers still seem off to me. Additionally is UNMAP defaultly enabled? What would be the best way to test/confirm.

 

[mav@]

Numbers still seem off to me.

Pool spaces are indeed higher then expected, but that is fine, since pool measures physical space, including checksums. What's about zfs spaces -- the only question I see is why zvol takes 37TB instead of mentioned 25. Don't you have some snapshots there that hold the space? Or haven't you tuned ZVOL block size from recommended? For RAIDZ2 of 6 disks you should have block of at least 16KB (or better even 32-64KB) to be space efficient.

Additionally is UNMAP defaultly enabled? What would be the best way to test/confirm.

UNMAP is enabled on FreeNAS side for ZVOL extents and disabled for file extent. But Windows has own roaches -- for example, if file system was first created on older FreeNAS version without UNMAP support, then Windows won't detect its addition.

 

[CompMaster]

Pool spaces are indeed higher then expected, but that is fine, since pool measures physical space, including checksums. What's about zfs spaces -- the only question I see is why zvol takes 37TB instead of mentioned 25. Don't you have some snapshots there that hold the space? Or haven't you tuned ZVOL block size from recommended? For RAIDZ2 of 6 disks you should have block of at least 16KB (or better even 32-64KB) to be space efficient.


UNMAP is enabled on FreeNAS side for ZVOL extents and disabled for file extent. But Windows has own roaches -- for example, if file system was first created on older FreeNAS version without UNMAP support, then Windows won't detect its addition.

Sorry for not posting better formatted quotes. I am on a mobile.

Snapshots -- I thought that could be it as well. I confirmed I have no snapshots.

Correction on the space. I have 21T allocated to the zvol. I think I was misunderstood in earlier post.

zvol Block Size -- Not 100% sure on this. Will try to lookup myself right now. Could you advise best way to tell?

File System Creation -- FreeNAS first built on 9.2 (possibly 9.1 but I don't think so). Windows attached to zvol during that time and hasn't moved to another Windows server since. So it's the original Windows Server.

 

[mav@]

Snapshots -- I thought that could be it as well. I confirmed I have no snapshots.
Correction on the space. I have 21T allocated to the zvol. I think I was misunderstood in earlier post.
zvol Block Size -- Not 100% sure on this. Will try to lookup myself right now. Could you advise best way to tell?

Please show output of `zfs get all vol0/iscsi`. It should include everything.

File System Creation -- FreeNAS first built on 9.2 (possibly 9.1 but I don't think so). Windows attached to zvol during that time and hasn't moved to another Windows server since. So it's the original Windows Server.

If this specific iSCSI extent was first formatted on FreeNAS 9.2 or 9.1, then Windows may not use UNMAP there. I don't know how to fix that other then create new extent, format it and move the data.

 

[CompMaster]

Please show output of `zfs get all vol0/iscsi`. It should include everything.


If this specific iSCSI extent was first formatted on FreeNAS 9.2 or 9.1, then Windows may not use UNMAP there. I don't know how to fix that other then create new extent, format it and move the data.

See attached for the output. Looks like 8k block size.

 

[mav@]

So block size is indeed 8K. Probably your zvol was created before autotuning logic for block size was added to UI. The only more thing you may check to confirm my theory is that your pool formatted with 4K sectors (ashift=12) with `zdb vol0 | grep ashift`. If it will show "ashift: 12", that will explain excessive space usage, since in that case space efficiency of this ZVOL is below 50% (8KB of parity per each 8KB of data). If your zvol would have block of 32-64K, as automatically proposes modern UI, space efficiency would be about 66% or better (8KB of parity per each 16KB of data, plus better compression).

Using bigger blocks you may loose some performance on short and random I/O, while improving performance on large sequential I/O. But, considering terrible space efficiency otherwise, there is no other choice for RAIDZ2.

 

[CompMaster]

So block size is indeed 8K. Probably your zvol was created before autotuning logic for block size was added to UI. The only more thing you may check to confirm my theory is that your pool formatted with 4K sectors (ashift=12) with `zdb vol0 | grep ashift`. If it will show "ashift: 12", that will explain excessive space usage, since in that case space efficiency of this ZVOL is below 50% (8KB of parity per each 8KB of data). If your zvol would have block of 32-64K, as automatically proposes modern UI, space efficiency would be about 66% or better (8KB of parity per each 16KB of data, plus better compression).

Using bigger blocks you may loose some performance on short and random I/O, while improving performance on large sequential I/O. But, considering terrible space efficiency otherwise, there is no other choice for RAIDZ2.

Your suspicions are correct. See the attached.

So from here this is what I'm thinking my plan is. I have 18 TB of free space remaining. I should make a new zvol. Attach it to the Windoes server and start migrating data. There is about 15TB of data I have to migrate from the existing zvol so I am barley going to make it.

How should I setup my new zvol? If I didn't mention it before this zvol holds backup data images that are part of a continuous chain. Software manages the chain to consolidate images down into daily, weekly, and monthly images.

Also am I moving the data in the most efficient way?

 

[mav@]

If you are running modern version of FreeNAS, setting up new zvol should not require any specifics. UI should automatically set appropriate block size for it, respecting present pool configuration, but you may check it to be sure by clicking "Advanced" button on zvol creation page.

Your migration plan with filling pool completely may work (though consider that there is also some metadata overhead, so some more space will be allocated), but I would still try to avoid, if possible. Writing pool up to its full capacity will increase data fragmentation, and when you later delete old zvol, the new one will still remain fragmented. That is not fatal, but I would prefer my data could be read faster, is possible. So if you have some other storage, I would temporary offloaded some data there to not max out the pool.