NFS sync and async puzzle

[feyi1986]

Hi, I decide to make this post after a couple of hours searching on google. Maybe my google-fu is not good, please forgive me if my question is dumb.
So I started to build my freenas server with two expensive nvme SSD which are put in lz4 compressed mirror without SLOG. I export them as NFS via 10GE nic to anther server and trying to copy 2GB file from the local disk of the second server to my freenas.
For different configurations, I get table below.

So my first question is that why ZFS with sync standard performs worst in both cases ? Isn't it supposed to be at least comparable to ZFS sync always ?
The second question is that how do I understand difference in the second row? My understanding is that when ZFS is set to sync always it returns write committed only if it finishes actually write in the ZIL, no matter if NFS has explicitly requested sync or not. So why it still performs better when NFS is set to async? What kind of role the NFS sync option plays here ?

 

[dlavigne]

Did you find the answer to your questions?

 

[Heracles]

Hi Feyi,

Unfortunately, we can not answer that one... The reason is simply because your test procedure is not completely defined and what is defined is not proper testing.

In which order you did your tests ? If you started with Standard, may be the source was the slowest part of it because it did not had any data in cache before sending them. The source was slow to read, so slow to send. For the other tests, the data was still in cache, at least partially, so the source was faster.

Another example can be an anti-virus on the source size. It scanned the file on the first shot, so slowed down the transfer. Because it was scan recently and not modified since, no re-scan for the other test.

In all cases, when doing tests like this, always do it from RAM and ensure your source is itself constant before monitoring the result.

So first test is RAM to RAM :
From a Unix box, you use DD with something like /dev/zero as a source and on the destination part, you write it to /dev/null. You go for X blocks of Y size so you are happy with the total volume.

For statistical validity, you need at least 10 measures. So you re-do it at lest 10 times and confirm that your result is constant before going next step.

Because here you go after NFS speed and you do compression, I would suggest again to use /dev/zero as a source. Because all blocks will be the same, compression should be as constant and regular as possible, so should reduce the noise in your measurement.

Time-based performance testing is very difficult to achieve in a way that will produce valid and objective result...

 

[jgreco]

https://www.ixsystems.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

I think you might have made the mistake of assuming that SLOG has no role in a situation like this. In a SLOG-free pool, ZFS needs to commit sync writes through the in-pool ZIL mechanism, which is a lot more heavyweight and a lot less optimized than the SLOG data path.

In general, remember that requesting sync is going to involve changes to the data flow that require a full round trip per block, and this is generally going to be a slow thing. The bottleneck is likely to be the network, so I would expect that toggling NFS from sync to async would have more of an effect, which is exactly what your chart shows. It isn't clear that the remaining differences in your chart are statistically significant or were performed with sufficiently rigorous testing methodology to be meaningful. I might disagree somewhat with using /dev/zero as @Heracles suggests for testing purposes, but lots of other good thoughts in that post.