NFS is now also blazing fast.
Does this mean I could loose all the data in Datastore 4 (DS04) or across the whole Dataset?
You shouldn't "lose" any data. However, you might scramble some data (see below).
Read this.
Having read that, now listen carefully to what I'm about to say. ESXi is excessively paranoid about writes out to NFS; it wants, very much, to ensure that it is not the party responsible for mangling your data.
ZFS is also excessively paranoid and does not want to be that party either.
So in an odd way, ZFS/NFS/ESXi are actually awesome together because they're all working together to ensure data integrity. But it tanks performance to provide those safety guarantees.
What you are doing by setting sync=disabled is telling ZFS to disregard the sync requests from ESXi. ZFS will still take reasonable care with your data. However, if, for any reason, you were to lose the NAS during the window where a VM has written something (and ESXi has written it to the filer) and before ZFS has flushed it out to disk, then when the NAS comes back, those changes won't be there. This could be a problem for your VM.
Why is the QNAP much faster? We have had power outages with the QNAP and have not lost a single byte?
The QNAP might have a battery backed write cache. Or you might have configured the QNAP to enable write caching without a BBU. Or it might even enable write caching without a BBU without telling you. In any case, you MIGHT not have lost a single byte. However, it is much more likely that you've lost data and just not realized it. Basically, if your guest VM writes data and your storage system confirms that it is written, and it isn't actually committed to nonvolatile storage of some sort (flash, spinny rust, etc), then you lose that data if the NAS crashes/loses power/etc. This is very basic.
So here's the thing. If your VM was largely idle and maybe only adding entries to log files, and the VM crashed at the same time as the NAS filer because the VM host lost power too, then when everything reboots, any "lost data" gets chalked up to the power loss because the reboot process will do file system consistency checks and everything comes back up dandy.
But if your VM is busy writing database file updates to your critical business database, and suddenly your NAS crashes, and a gig of write data is in memory but not flushed to disk (and your VM's been told that it's been stored), and the NAS comes back up, and your VM is still running, well, now, you have a Real Problem. Your on-disk database and its indexes are no longer consistent with what the VM reasonably expects to be out there, and so now as further writes happen, the on-disk database gets corrupted (remember a gig worth of updates "happened" but also didn't happen!) and as reads happen, your database app freaks out and takes a dump and you're left with data goulash.
Ok, so here's the thing. You need to understand that turning sync off shouldn't cause ZFS to lose the files in your datastore. However, those files represent virtual disks for your virtual machines. Turning off sync will increase performance, yes. But turning off sync also reduces the resiliency of the storage system to failures/crashes/power-outages/etc. Given what you are storing in the files, it is possible that a crash could cause havoc with your virtual machines.
