Why is my NFS write performance this bad?

Status
Not open for further replies.

SirHaxalot

Dabbler
Joined
Aug 27, 2012
Messages
10
I mentioned having 90MB/s synchronous write in Samba/CIFS file copy (copying an ~15GB compressed file to a network share in Windows), thus the problem seems to be isolated to NFS.

The workload I'm planning to use this for is Visualization with VMWare ESXi, from where I connect with NFS. VMWare will be opening files with the O_SYNC flag and there's nothing I can do about that. The guests will also initiate NFS connections for /home and the likes but performance isn't as large of an issue there. After all, random read/write are the most important aspect and I get better performance there at ~500-1000 IOPS for write and between 1 000-10 000 IOPS for read (I'm guessing whether it's cached or not plays a huge role here).

Additionally, the SSDs I'm using are Agility 3 and are specified for 475MB/s sequential write and 50 000 - 80 000 4k random-write IOPS. I think they should be able to do it, but I guess I could try to disable the read cache and see if performance increases.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I don't know how you're going to easily get better performance. Just following this thread it really looks like ZILs and L2ARCs aren't going to help you. It just looks like NFS and ZFS aren't good solutions for your situation.
 

bollar

Patron
Joined
Oct 28, 2012
Messages
411
There are some tools to help you see the impact of changes to your ZIL and ARC Cache strategy.

Zilstat
Arcstat

This entry has lots of advice for tuning for NFS, but in particular, they focus on write caches as the key culprit in slow NFS performance:
ZFS Evil Tuning Guide - Write Caches
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
NFS/ZFS and ESXi do appear to play particularly poorly by default.

This little N36L server we have (16GB, 4xWD20EARS in RAIDZ2, 4k), I've been torturing it with ESXi backups. The NFS dataset it had on it, soaking up ESXi dumps, was performing awfully (10-12 megabits maybe?), and we didn't have the right stuff in inventory here so we just set sync=disabled on the afflicted dataset and it flies (around 200 megabits now). Of course, if something crashes, there's a substantial chance of a corrupted backup. Not For Production Use.

I would expect that a ZIL would be helpful, but I haven't yet tried one for this box (and yay I get to decide whether or not to requisition resources for this in the next two days).
 

SirHaxalot

Dabbler
Joined
Aug 27, 2012
Messages
10
I have finally managed to bump the performance up to acceptable levels! There seems to have been a few parts that played a big role in this.

First of all, block size on the ZFS datastore. Apparently ZFS only decide the block size based upon file size, which resulted in all files becoming split up in huge blocks, which isn't exactly good for write performance of 4kB blocks with Copy-on-Write (has to read the whole first stripe then rewrite it), although I think this might have actually gotten better because of the ZIL.

Second part to improve the performance, which I'm actually surprised made such a huge difference, was to move the L2ARC off the ZIL devices. I managed to dig up an older Intel 2nd generation SSD which now has to role as dedicated L2ARC. So I still will be able to enjoy fast IOPS on Read.

The performance as of now is ~70MB/s read, 40MB/s write sequential and 150 IOPS read, 1000-2000 IOPS write at 4kB random. Not too bad at all. I expect the random read to increase as the L2ARC gets populated.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
It's probably just the cynical curmudgeon in me, but it has always seemed to me that the reality of SSD is somewhat less stellar than the promise of SSD. FreeNAS doesn't support mixing ZIL and L2ARC, and my reading in-between-the-lines is that it is because many others have run into sort-of-questionable performance issues as well. SSD performance is dependent upon so many complicated factors, and your setup is fairly nonoptimal in other ways. With only 8GB of system RAM, your ARC is going to be stressed - heavily! - providing 2 x 54GB L2ARC worth of space; you essentially have far too much L2ARC. With an 8KB record size, estimate around 32MB of ARC consumed per GB of L2ARC; you have 108GB of L2ARC, so that's about 3.5GB of your RAM used just to support the L2ARC. Yikes. ARC on an 8GB system is probably only around 4GB to begin with! You really don't have space for L2ARC.

But then there's the other problem. SSD writes will start to perform poorly over time, due to garbage collection, if you're using the full thing and there's no way for an OS to coordinate with the SSD through things like TRIM. Some people have advocated using Host Protected Area on MLC SSD's as a way to help the controller maintain a large free pool, and on a MLC ZIL SSD that might work pretty well. But what I would point out to you is that you've created a catastrophic situation for the controller from a performance point of view, you're basically demanding that the SSD "use it all" AND hoping that it'll be fast about it. Unlikely. If you haven't seen this, even if you think you understand the technology, it's good to go over this and catch up on the latest details of the SSD technology.
 
Status
Not open for further replies.
Top