Memory utilization and performance problem.

[joeschmuck]

ZFS in FreeNAS knows how much it sees and that is what counts.

I have no idea how you will replace your drives because I'm not familiar with your setup. Before you do anything, back up your data! You have no idea how many times we warn about this but sure enough every once in a while someone thinks it will be fine and it isn't. Data loss sucks. Thankfully it looks like you could backup all your data to a 2TB drive. If you have the capability you could attach the 2TB drive as a UFS drive and do a quick copy of the data, then remove the drive from your system. 2TB over a NIC will suck!

 

[datnus]

I will move data over to another server then.

 

[joeschmuck]

Good luck.

 

[datnus]

BTW, I deleted a few hundred GB from iscsi (vmfs), zpool list is unchanged.
The ESXi VMFS shows Free capacity go up.

I really doubt how ZFS reflects the Free size in VMFS.

For example: If FreeNas exports an iSCSI share and I format the share as NTFS / VMFS, I believe only NTFS / VMFS know what the free capacity is.

 

[paleoN]

For example: If FreeNas exports an iSCSI share and I format the share as NTFS / VMFS, I believe only NTFS / VMFS know what the free capacity is.

I believe you have this half-assed backwards. ;) Perhaps this will help:

-ESX / VMFS Filesystem
1. VMFS Filesystem can be considered to be a storage pool containing other filesystems . When it is created on a device presented from FreeNAS sitting on ZFS, it technically ends up being a "pool within a pool". The VMWare Guest OS filesystem then sits within this, creating many layers of storage allocation..
ZFS Pool
->ZFS Block Volume
-->VMFS Pool (via iSCSI)
--->Guest OS filesystem (VMDK) - NTFS/ext3/etc

One last thing, do everyone a favor by stop quoting everything and start using [code][/code] tags for CLI output.

Type in a shell, 'zpool list' (let us know what it says) and it will give you the Size, Allocated (Used), and Free.

With zpool list we can just look at CAP, i.e. capacity.

 

[joeschmuck]

BTW, I deleted a few hundred GB from iscsi (vmfs), zpool list is unchanged.

I don't know at what point the ZFS file system starts to work better, maybe you need to reboot it? paleoN said it switches to a different way to store data at under 10% so maybe a reboot will shift it back? Of course I am assuming the slowdown is due to the used amount of space on the drives.

 

[datnus]

I don't know at what point the ZFS file system starts to work better, maybe you need to reboot it? paleoN said it switches to a different way to store data at under 10% so maybe a reboot will shift it back? Of course I am assuming the slowdown is due to the used amount of space on the drives.

Reboot is not an easy option :D
I search around "ZFS shows wrong capacity" and find some queries without a final fix / explanation.
May be there is something mismatch between ZFS and VMFS.

BTW, the VMFS block size is 1MB by default.
I'm looking around to understand the "magic" logically, hehe.

 

[datnus]

Yet, ZFS is unable to read inside the iSCSI LUN, VMFS controls the free capacity.

The reason * says you've used 2.01TB is because you effectively HAVE. Even though VMware is releasing freed FILESYSTEM blocks from deleted files or changes in filesystem file size (swap files etc) without iSCSI UNMAP (ESXi5 supports this at 'higher' license levels) with * whats going to happen is:

You write the files to disk, * sees that as added used space (written blocks). So eventually your LUN according to * is going to be at "100% capacity" while the VMware is going to be able to read the actual filesystem capacity and available capacity. * can only track that blocks have been written for ZVOLs. With iSCSI UNMAP in 3.1.1 and a proper license from VMWARE you would then have the ability to 'free' those used blocks from the ZVOL and this 'problem' wouldn't exist any longer.

I'm pretty sure this has been discussed to ad nauseam on the web.

Also remember what you provision at (2.01TB) is raw disk space. Once VMWare gets ahold of it you're left with 1.76TB. The ZVOL itself can still take up 2.01TB (or more!) on the backing pool because of metadata/parity and refreservation.

Hitting 100% shouldn't matter for a ZVOL unless you're hitting 100% inside the FS itself or with thin provisioning of the ZVOL in conjunction to OVERprovisioning of your real capacity. 100% ZVOL capacity to Solaris/* usually just means at one point you allocated all of the available blocks to data.

Other product using ZFS also has this problem.
As it's competitor product, I don't paste link here, hehe.

 

[datnus]

If the free capacity is reported wrongly and the performance is related to the free capacity, I wonder if ZFS will decrease the performance / refuse to work when hitting 100% capacity.

 

[paleoN]

May be there is something mismatch between ZFS and VMFS.

Actually, you don't have a basic fundamental understanding of storage, VMFS datastores and VMDKs. I would suggest you go read the VMWare docs, i.e. the storage ones, as they are quite good. You also need to read up on ZFS some more as it also doesn't match your misconceptions. ZFS is correctly reporting free space in the pool.

Of course I am assuming the slowdown is due to the used amount of space on the drives.

It would be sufficient by itself to cause the slowdown.

I don't know at what point the ZFS file system starts to work better, maybe you need to reboot it?

Doesn't quite work that way. The last blocks to be written have been of necessity written fragmented. Best thing to do is zfs send, delete zvol or nuke pool, redundantly expand or recreate larger pool and zfs receive the data back. This will insure there is no leftover fragmentation. If you just expand the pool you would also see a performance improvement for any new writes.

 

[datnus]

http://kb.vmware.com/selfservice/mi...ype=kc&docTypeID=DT_KB_1_1&externalId=2014849

FreeNAS doesn't have iSCSI Unmap, so Vmware ESXi can't send unmap iSCSI to release the free capacity to FreeNAS.
If your storage vendor has tested this option with storage arrays, you may use it to reclaim deleted blocks resulting from deleting or moving files from VMFS datastores residing on thin-provisioned LUNs.

# esxcli storage core device vaai status get -d t10.bla bla
VAAI Plugin Name:
ATS Status: supported
Clone Status: unsupported
Zero Status: supported
Delete Status: unsupported

In iSCSI context:
- When Vmware adds new virtual machine, Vmware sends and iSCSI command to FreeNAS, fine, ZFS increases capacity.
- However, them Virtual machine moves to another iSCSI target / or is deleted, Vmware is unable to send iSCSI unmap to FreeNAS iSCSI. Hence, ZFS still believes the blocks are still in use.

The same problem happens with any iSCSI target not support VAAI (especially iSCSI unmap)

 

[datnus]

In this case, as FreeNAS may not have unmap iSCSI supported, is there any workaround for ZFS to "sync" back free capacity?

 

[datnus]

I have a test between 2 servers. Both servers have similar specification.
- 1 x CPU Xeon Quad Core
- 8 GB RAM
- All are 500 GB HDD SATA 7200 rpm
- RAID Z (No hardware RAID)

The differences are:
- Data server has 5 HDD (+ 1 spare). zpool shows 92% capacity.
- Backup server has 4 HDD. zpool shows 44% capacity.

Test procedure:
- To generate load: dd if=/dev/zero of=testfile bs=1024k count=250k
- Use zpool iostat 2 to monitor write bandwidth.
- User gstat to monitor %busy of HDD devices.

Test and result
- Test 1: test local write to backup server
Write rate up to 170MBps. The write rate is steady.
%busy < 50%

- Test 2: test iSCSI write from ESXi to backup server
Write rate up to ~160MBps. The write rate is up and down.
%busy < 90%
The performance local and remote iSCSI is quite similar.
The bottleneck is not at CPU or NIC

- Test 3: test iSCSI write from ESXi to data server
Write rate up to ~23MBps. The write rate is up and down.
%busy ~ 90%

The performance between 2 servers should be similar. But the test shows the differences.
I'm going to believe %Free capacity has an important impact on performance.


And in my previous post, FreeNAS doesn't support iSCSI Unmmap.
Hence, ESXi can't send iSCSI unmap to update ZFS of free capacity.

Haiz, real problem to Virtualization if VAAI (especially iSCSI unmap) is not supported by FreeNAS yet.

 

[datnus]

you still could issue a dd if=/dev/zero of=/vmfs/volumes/yourclutteredvolume/zerofile bs=8M; rm /vmfs/volumes/yourclutteredvolume/zerofile from the ESX(i) console. The zeroed blocks will be correctly recognized as "empty" by zfs.

http://serverfault.com/questions/31...-zfs-how-do-you-make-sure-you-dont-run-out-of

As syneticon-dj suggested, I run dd if=/dev/zero of=testfile bs=1024k count=50000 inside the iscsi vmfs from ESXi and rm textfile.

However, the zpool list doesn't decrease at all. In fact, the used storage increases when I do dd. :(

 

[datnus]

It seems that the only trick is to use another product which supports iSCSI Unmap.
Sigh.

 

[cyberjock]

I'm curious as to what the limitation is with iSCSi unmap. What doesn't have the support? Do we need to upgrade to FreeBSD 9?

 

[datnus]

I "believe" the iSCSI unmap is a feature of iSCSI daemon / service not ZFS.
And FreeNAS 8.3 is still based on FreeBSD 8.3, I think.

 

[datnus]

I think my only chance to export zvol and install other products which support iSCSI unmap.
No matter how big the zvol, the zvol will come to a slow performance at some points with virtualization.