ESXi block storage zvol reports full size in TrueNAS

[bstev]

I recently moved to a new build and I am guessing I made a mistake on a configuration somewhere, but I can not find what it is. My issue is that my current TrueNAS install is reporting the full size of my datastore as being used space, where as my old install would have TrueNAS reporting slightly less than what ESXi was seeing as used since I am using compression.

On top of that, my snapshots appear to take up the full referenced size....

Here it is without any snapeshots.

Now after a snapshot I get this.

Here is how I have it configured.

 

[ChrisRJ]

Please describe your hardware configuration in more detail. Esp. how the storage is set up.

 

[bstev]

Dell R720 running ESXi 6.7U3
TrueNAS13 core running as a VM on the same host above
two NVMe drives plugged into PCIe and passed through to TrueNAS
the two NVMe are setup as a mirror with a slog and shared iSCSI multipath back to ESXi
ESXi created datastore VMFS6

 

[artlessknave]

note that using truenas virtualisation on the same host to host your VM storage is rarely a good idea, because you have to always start truenas before any other host

you mention a SLOG but not where it is? if your SLOG is not on reliable disks, your entire pool is at risk. loss of the SLOG disk means loss of the pool.
if you are using NVME drives for your VM storage, SLOG is most likely irrelevant, beyond adding another point of failure for nebulous benefit. the NVME drives themselves will likely outperform all but the most expensive SLOG devices.

next, usually "missing" space is caused by snapshots. the fact that a snapshot tapes up 160GB gerenally tells us that ZFS considers 160GB to have changed since the last snapshot. maybe ESX is using thick storage ?

 

[Patrick M. Hausen]

You have a pool slightly below a Terabyte in size. You created a 700 G (roughly) volume as an iSCSI target. That volume is going to take up all the space assigned to it. Feature, not bug.

If you really insist to overprovision you need to create a sparse volume - the effect being that if it ever fills up your VMFS datastore will probably crash and burn. You can use zfs inherit -S refreservation <pool><path-to-volume> to achieve that after creation.

Use at your own risk. Also read @jgreco's guide about block storage and the capacity you can expect to use. A pair of NVMe drives is definitely not a bad start for a small setup but don't expect to be able to fill that space.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

 

[bstev]

Thank you Patrick M Hausen, that sounds like the difference from my old install to my new install. The old install was originally created all the way back in FreeNAS 9 which I see in old documenation had a check box for "sparse volume" which I probably did that long ago.

I imagine this checkbox was removed on the following versions because it is not a good practice in general. I do also recall the negative side of using a sparse volume, which you mentioned.

That answered my issue perfectly. Thank you!

 

[artlessknave]

I just looked, and sparse is still an option when creating a zvol.

 

[bstev]

I just looked, and sparse is still an option when creating a zvol.

Thank you, I am glad you mentioned it. I just double checked too.

 

[artlessknave]

you still probably should be really considering your design. at the very least, look into SLOG devices risks and where they are useful.

 

[bstev]

you still probably should be really considering your design. at the very least, look into SLOG devices risks and where they are useful

Thank you for the tips. I am sure some things could be better or more reliable, but I feel it is an okay balance and I do make improvements as budget allows.

the NVMe I have are just Samsung EVO's and the slog is an optane drive. I am giving 128GB RAM to TrueNAS, so between the ARC size and using SLOG to reduce double writes to the EVO's that a ZIL would have, it is not too bad for the VM's I am running from it. I also have TrueNAS SSH to ESXi after it finishes booting so it does a storage rescan and finds the iSCSI datastore so the other VM's still boot fine during the auto start sequence.

 

[Patrick M. Hausen]

As far as I know there is no double write without the SLOG. @jgreco Care to confirm or deny?

 

[bstev]

Yes, please let me know if I misunderstand that part. I am under the impression that sync writes will first hit a write cache "so to speak" that is the ZIL and located on the pool too if no SLOG is being used. If it was not sync writes then it would just hit the RAM and only be written once to the pool.

SLOG Devices

Provides general information on ZFS intent logs (ZIL) and separate intent logs (SLOG), their uses cases and implementation in TrueNAS.

www.truenas.com

TrueNAS Block Storage & ESXi (Part 1)

In this blog post I cover some basic ZFS theory and setting up a high performance ZFS pool for ESXi VM storage on TrueNAS core.

www.breznet.com

 

[jgreco]

As far as I know there is no double write without the SLOG. @jgreco Care to confirm or deny?

I interpreted the statement to mean the extra write to the in-pool ZIL. If you do not have a SLOG device, then there are two writes to the pool, one to commit to the ZIL through the fast write mechanism that is intended for ZIL writes, and then one to flush the entire TXG to the pool.

There was some talk at one time about ways to integrate these two writes to reduce the wear on SSD pool devices, but my takeaway was that this was impractical for RAIDZ and potentially possible for mirrors, but still problematic from a design point because the ZIL region of the pool has to be a known factor. If it isn't, then, when the pool is being imported, discovering what blocks are "part" of the ZIL would become very complicated. It is much simpler to have a FIFO queue region, replay the block transactions, and guarantee that they've been written to the pool. I do not believe that anyone has put serious effort into fixing the in-pool ZIL to perform better, especially with the advent of Optane. On the other hand, I would say that this could be considered a design error for at least mirror pools in the ZFS model.

 

[jgreco]

Yes, please let me know if I misunderstand that part. I am under the impression that sync writes will first hit a write cache "so to speak" that is the ZIL and located on the pool too if no SLOG is being used. If it was not sync writes then it would just hit the RAM and only be written once to the pool.

That sentence gives me a bit of a headache, sorry.

As pictured by your picture, which I dislike because it incorrectly implies that there are separate write streams for async and sync writes going to the ZFS write cache (main memory); ALL writes proceed to main memory and are placed in a TXG, which is the name for a group of ZFS blocks that are going to be written to the pool.

TXG's are then flushed to disk as indicated by your picture.

The part where I think your picture does a poor job as an explainer is that all sync writes follow an additional parallel flow out to SLOG/ZIL. Neither the SLOG nor the ZIL are any form of write cache. They are an intent log, which differs because there is no read activity AT ALL during normal operations, unlike a cache which might be read on a FIFO, LIFO, hash, etc. basis.

Ffffffffine. I'll break out the ol' paint program.


This clarifies that there is really only one dataflow to the TXG aggregation going on in main memory, and that what is going on with sync writes is actually a second copy data stream.

 

[bstev]

Thank you all for keeping the info correct. Having a knowledgable and active community for help here has been great.