NFS Slow Write Speed With Sync ON.

[SeanHayes]

I have Proxmox on one host and TrueNAS on my other host.
Both have 10GB fibre links over the network to talk to each other.
I have setup an NFS share on truenas that proxmox connects to, I have 5 vms and was getting annoyed with how slow it was to install windows.
I started investigating and stumbled on people saying disabling sync gets them faster nfs performance, at this point my vms were getting about 500MB/s read and 10MB/s Write, so I disabled sync on nfs and boom i suddenly get 500MB/s read and 1100MB/s write.

My Truenas has 6x3TB HDD, and 3x 120gb SSDs, hard drives are in a z2 raid and ssds are used as a Mirrored SLOG.

Any ideas on why this may be happening? Or any input as to how I can have sync on but with better performance as I dont want to have any loss of data Integrity.

 

[HoneyBadger]

Welcome!

Disabling sync will indeed get you faster write performance, but at the rather significant cost of "the safety of your pending writes."

Please check on the resource below - while it is written with the perspective of an ESXi/VMware hypervisor, the same rule still applies for other hypervisors. Async is fast, but unsafe. Sync is slow, but safe - adding a fast SLOG device helps regain some of the speed you lost in becoming safe.

Resource - Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster?

This post is not specific to ESXi, however, ESXi users typically experience more trouble with this topic due to the way VMware works. When an application on a client machine wishes to write something to storage, it issues a write request. On a...

www.truenas.com

The qualities that make an SSD a good SLOG are low write latency, consistent write latency, and high write endurance. Generally speaking, these aren't found in "consumer" or even "prosumer" SSDs - you need to dip your toes into the water of "enterprise" or "very high-end consumer" such as Optane to reach those marks. Have a look at this resource for some discussion and benchmarks.

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you need for a "great" SLOG: 1. Fast throughput 2...

www.truenas.com

The final piece is the vdev configuration. RAIDZ with parity works well with larger, single-stream workloads - but VMs generally work with smaller pieces of files, and in a more random manner. Mirrors are highly preferred for this type of workload - but even with mirrors, six spinning disks are unlikely to be able to handle line-rate speed of 10Gbps. You can see it for very short bursts, but it won't be able to sustain that.

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.truenas.com