Slow NFS Transfers with Write Sync

[ryanw]

Hi,
Testing out writing from my Manjaro system to TrueNas 3 x old WD drives [for testing before I get shiny new ones] in RaidZ1 (other specs in signature).
I mount the NFS share and use DD to copy a file to the NAS

Code:

sudo mount -t nfs4 192.168.10.134:/mnt/Mirror/TestMirror /mnt/nfs
dd if=/mnt/nfs/ubuntu.iso of=Documents/testcopy/ubuntu.iso 
  

I'm getting only about 70MB/s for this 1GB file.
If I turn of write sync on the NFS share then I get up to about 180 MB/s.
(Incidentally this is a dual boot system and botting into windows I get around 250MB/s write to the NAS!)

Surely I shouldn't have to turn off write sync to take advantage of the 2.5Gbe?
How can I improve the speed and get it closer to Windows SMB performance?

I've read a lot on these usual forums about ZIL and SLOG etc.
Do I really need an SSD to try to offload the additional writes for the sync from the main pool or am I missing something obvious?

 

[jgreco]

It seems like the obvious question here is, do you understand what sync writes are?

Please do go and read

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

70 megabytes per second seems pretty good.

 

[ryanw]

It seems like the obvious question here is, do you understand what sync writes are?

Please do go and read

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

70 megabytes per second seems pretty good.

I thought I understood it and read through that earlier. Maybe I am getting terminology mixed up - apologies I am new to ZFS in general.

The underlying question is really how do I get my Manjaro machine performing similarly to the Windows boot on the same hardware. Something doesn't seem correct to me (I know I'm comparing NFS with SMB) but really I expected Linux to be faster at simple file copy to the NAS than Windows.

 

[jgreco]

What size are you using for read and write buffers?

 

[ryanw]

Everything is running at default settings TrueNAS CORE 12.
Where would I find those settings if you still require them please?

 

[jgreco]

The "mount" command from the client will probably tell you. Look for "rsize" and "wsize".

 

[ryanw]

The "mount" command from the client will probably tell you. Look for "rsize" and "wsize".

Thanks @jgreco - once again apologies for being a noob.
I'm not supplying any rsize or wsize params on the mount when I do it, but looking at the value from nfsstat I see: rsize=131072,wsize=131072

 

[jgreco]

once again apologies for being a noob.

No worries. I'm tossing off pithy responses in between doing real work.

rsize=131072,wsize=131072

I think that may be as big as FreeBSD allows. We went thru this a year or two ago in another thread IIRC. Unfortunately this may end up being the limiting factor, along with CPU and network speeds. NFS is just a little sucky in some ways.

I'm trying to think whether larger TCP buffers at the system level makes this better or not. I'm at the wrong end of the day to be thinking about this...

 

[ryanw]

No worries. I'm tossing off pithy responses in between doing real work.



I think that may be as big as FreeBSD allows. We went thru this a year or two ago in another thread IIRC. Unfortunately this may end up being the limiting factor, along with CPU and network speeds. NFS is just a little sucky in some ways.

I'm trying to think whether larger TCP buffers at the system level makes this better or not. I'm at the wrong end of the day to be thinking about this...

Would you suggest anything instead of NFS from my Manjaro (I'm just using NAS as file storage at this point but hope to use some block storage for game library at another point - more learning than anything and trying to centralise storage)?

Definitely not the time to be taxing yourself especially if you have real work to do. My worries can wait!

 

[jgreco]

Well, the thing is, normally where I'm uncertain of an answer, I'd go and do a little testing to see what's what. I'm usually always right because I've made certain of it.

In this case, though, I'm not likely to get to this for awhile, and there's really no harm in you just TRYING. It also happens to be helpful generally for SMB as well (known fact).

There are various sysctl tunables that you can tweak to increase system level buffer sizes. I am not certain this will help with NFS. I am not telling you it will help with NFS. You get to discover that on your own.

You can see a list of tunables I've used in the past in this post;

https://www.truenas.com/community/threads/intel-x540t2-wont-run-past-300mbps.71108/post-491858

and my suggestion is to just pick one or two of those and search the forum for recent versions of such a list, since they change slightly as time goes on.

 

[ryanw]

Well, the thing is, normally where I'm uncertain of an answer, I'd go and do a little testing to see what's what. I'm usually always right because I've made certain of it.

In this case, though, I'm not likely to get to this for awhile, and there's really no harm in you just TRYING. It also happens to be helpful generally for SMB as well (known fact).

There are various sysctl tunables that you can tweak to increase system level buffer sizes. I am not certain this will help with NFS. I am not telling you it will help with NFS. You get to discover that on your own.

You can see a list of tunables I've used in the past in this post;

https://www.truenas.com/community/threads/intel-x540t2-wont-run-past-300mbps.71108/post-491858

and my suggestion is to just pick one or two of those and search the forum for recent versions of such a list, since they change slightly as time goes on.

Thanks for the pointers. NFS may be a lost cause so I may start with SMB and look through the various tunables to get that running at a decent lick first.