NFS read speed half of write speed

[CJRoss]

I've searched around and it doesn't appear that my scenario matches others situations.

Transferring a 40gb file from my client to my TrueNAS I get 7-800MiBps. Transferring a 40gb file from TrueNAS to my client, I get 350MiBps. iperf3 shows a bit over 7Gbps each way.

I'm running TrueNAS Core 13.0-U3.1 on a Xeon D-1540 with 64G of ram and an 11 disk z3.
My client is Ubuntu 22.10 R5 3600 with 32G of ram and a 980 PRO.

Everything is using the default settings. No tuneables. The pool is primarily for bulk storage. My jails are all on a separate SSD pool.

Any suggestions?

 

[NugentS]

Thanks for posting your hardware - but you only posted some of it - can you post the rest please. In particular.

1. Hard Disks & Pool Makeup
2. Network Card
3. How the machines are connected?

 

[CJRoss]

Thanks for posting your hardware - but you only posted some of it - can you post the rest please. In particular.

1. Hard Disks & Pool Makeup
2. Network Card
3. How the machines are connected?

Pool makeup is already posted. 11 disk z3. It's all shucked WD 14tb.
Xeon D is the built in 10g. Ubuntu is Aquantia 10g.
Connected via Cat 6 to a 10g switch.

 

[NugentS]

Sorry - I missed that.
Does that transfer (writes) start fast and then slow down (with a big file)? Its what I am expecting

 

[ChrisRJ]

That difference may come from the size of your RAM acting as write cache. Does it change if you do a 200 GB file?

Please use correct terminology. You are inventing abbreviations. And while there is a good chance that we decipher them correctly, it is an unneccessary risk for misunderstandings.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to translate terms on the reader's side when reading posts...

www.truenas.com

 

[NugentS]

@ChrisRJ - thats where I was going. My initial suspicion is that the OP's pool is too slow for the network card. He's filling up the RAM faster than he can flush to disk. But thats just an initial thought

 

[CJRoss]

I forgot to include my NFS settings.

type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=NAS,mountvers=3,mountport=743,mountproto=udp,local_lock=none,addr=NAS)

Sorry - I missed that.
Does that transfer (writes) start fast and then slow down (with a big file)? Its what I am expecting

No. The transfer speed is relatively consistent.

That difference may come from the size of your RAM acting as write cache. Does it change if you do a 200 GB file?

What write cache? Are you talking about the ZIL or a SLOG? I don't have a SLOG. I have not tested with a 200GB file.

Please use correct terminology. You are inventing abbreviations. And while there is a good chance that we decipher them correctly, it is an unneccessary risk for misunderstandings.

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to translate terms on the reader's side when reading posts...

www.truenas.com

What terminology am I using incorrectly?

 

[CJRoss]

@ChrisRJ - thats where I was going. My initial suspicion is that the OP's pool is too slow for the network card. He's filling up the RAM faster than he can flush to disk. But thats just an initial thought

I think you may have misunderstood my original post. Writing TO the pool via NFS is twice as fast as reading FROM the pool.

My 980 PRO is rated for 7GB reads and 5GB writes. Using fio I only see 350MB write speeds when using 4k block sizes. But as I posted, my NFS block size is set to 128k so I don't think that's the issue.

 

[NugentS]

I think you may have misunderstood my original post. Writing TO the pool via NFS is twice as fast as reading FROM the pool.

Oops - guilty (I am doing well here aren't I!)
Most of my ideas have just vanished I am afraid. Thats a somewhat unusual problem.

 

[winnielinnie]

Why are you using NFS version 3? Why not 4.2? TrueNAS Core 13.0-U3.1 supports NFS 4.2 now.

Also add nconnect=16 to your mount options (on the client) for the NFS share.

Then see if it helps with the performance.

 

[CJRoss]

Oops - guilty (I am doing well here aren't I!)
Most of my ideas have just vanished I am afraid. Thats a somewhat unusual problem.

Hence my comment that my situation seemed different from all of the other posts I found. :)

Why are you using NFS version 3? Why not 4.2? TrueNAS Core 13.0-U3.1 supports NFS 4.2 now.

Because I was using the defaults and getting 700MiBps writes was acceptable performance. Doing a search on the differences implies that it's not a simple change to move from v3 to v4.

Also add nconnect=16 to your mount options (on the client) for the NFS share.

Then see if it helps with the performance.

I'll give that a try, but is there a reason you think it will help considering that I can already get fast writes?

 

[winnielinnie]

I'm running TrueNAS Core 13.0-U3.1 on a Xeon D-1540 with 64G of ram and an 11 disk z3.
My client is Ubuntu 22.10 R5 3600 with 32G of ram and a 980 PRO.

Both support NFS v4.2.

Doing a search on the differences implies that it's not a simple change to move from v3 to v4.

It's trivial. Just check the box under TrueNAS -> Services -> NFS -> Enable NFSv4

Then make sure you don't specify the version number in your mount parameters (as it should default to vers=4.2), and if not, manually specify vers=4.2

I'll give that a try, but is there a reason you think it will help considering that I can already get fast writes?

Why not just try it? It doesn't hurt. Use NFS v4.2 + nconnect=16 and see if it yields better read speeds.


If it doesn't work, at least you can say you tried.



You can also set the recordsize to 1 MiB on the dataset, which may also help large sequential read speeds.


EDIT: Did you change your dataset's Sync property to "Disabled" (i.e, "always async") by any chance?

 

[CJRoss]

Both support NFS v4.2.

But TrueNAS defaults to v3 when you set up a share.

It's trivial. Just check the box under TrueNAS -> Services -> NFS -> Enable NFSv4

Then make sure you don't specify the version number in your mount parameters (as it should default to vers=4.2), and if not, manually specify vers=4.2

According to this article, it's not as trivial as just checking a box. https://www.linux.com/news/benchmarking-nfsv3-vs-nfsv4-file-operation-performance/

Additionally, the test results in the article show that v3 and v4 perform similarly.

Why not just try it? It doesn't hurt. Use NFS v4.2 + nconnect=16 and see if it yields better read speeds.


If it doesn't work, at least you can say you tried.

Because I like to understand why things are the way they are. If I just tried everything I found I'd have a stack of tuneables, etc.

You can also set the recordsize to 1 MiB on the dataset, which may also help large sequential read speeds.

It's already set to 1M.

EDIT: Did you change your dataset's Sync property to "Disabled" (i.e, "always async") by any chance?

Why would the sync property have anything to do with read speeds?

 

[winnielinnie]

Why would the sync property have anything to do with read speeds?

Because you're claiming slow reads speeds relative to your write speeds. (It's in your thread's title.) If you had been using "async" on the dataset + the NFS parameters, then your write speeds wouldn't reflect data being written to disk, but rather simply being sent to the server.

Rather than respond with the dataset parameters, it feels like you're playing a game of "little-by-little revelation." You could just reply with "Nope, dataset Sync property is set to Standard."

It's already set to 1M.

I can only guess, since you didn't post the dataset properties that you're running these tests on.

According to this article, it's not as trivial as just checking a box. https://www.linux.com/news/benchmarking-nfsv3-vs-nfsv4-file-operation-performance/

Quick and easy process of elimination. Both your client and server support the latest version and standards, including nconnect. Take advantage of it, with nconnect=16 so that you can at least uncork any TCP/network bottleneck.




EDIT: I'm not trying to come off as rude or "do it this way!!!"

If you try it out, and it doesn't make a difference, then you can put it behind you. If it does make a difference, even a small amount, it might further clue you in on what else to look for.


Another test you can try is to read from the NFS mount to a ramdisk on your client. Like a 4GB file to a 5GB ramdisk, to rule out the 980 PRO.

 

[ChrisRJ]

What write cache?

As I wrote, the RAM. It is acting as write cache.

I have not tested with a 200GB file.

I must admit that I did not fully read all the messages in between. But it is generally considered good practice to use file sizes that are so big that caching cannot influence the results.

What terminology am I using incorrectly?

See below:

11 disk z3

Is that RAIDZ3?

Xeon D is the built in 10g. Ubuntu is Aquantia 10g.

Does that mean that the server and client both have NICs with 10 Gbps?

Connected via Cat 6 to a 10g switch.

Same here.

You may think "why should I bother, these guys understand what I mean". Well, not necessarily. Plus it makes it harder to read and reduces your chances to get the best possible response.

 

[CJRoss]

Because you're claiming slow reads speeds relative to your write speeds. (It's in your thread's title.) If you had been using "async" on the dataset + the NFS parameters, then your write speeds wouldn't reflect data being written to disk, but rather simply being sent to the server.

My understanding of async is that they're acknowledged when added to the transaction group, not necessarily the disk. If the disk isn't fast enough to flush the transaction groups, it could still hang the transfer. I could be wrong, though.

Rather than respond with the dataset parameters, it feels like you're playing a game of "little-by-little revelation." You could just reply with "Nope, dataset Sync property is set to Standard."

Because I'm trying to understand why people think someone might be an issue and everyone immediately through I was having slow write speeds.

The dataset sync is set to disabled. NFS is default.

Quick and easy process of elimination. Both your client and server support the latest version and standards, including nconnect. Take advantage of it, with nconnect=16 so that you can at least uncork any TCP/network bottleneck.

Again, my understanding of moving to v4 is that it's more than just a quick change. In regards to nconnect, I haven't had a chance to test it yet. I'm just trying to understand why it would make a speed different in reads but not writes.

EDIT: I'm not trying to come off as rude or "do it this way!!!"

If you try it out, and it doesn't make a difference, then you can put it behind you. If it does make a difference, even a small amount, it might further clue you in on what else to look for.

Same. But I have a limited time to make changes and test things and want to understand how everything works.

Another test you can try is to read from the NFS mount to a ramdisk on your client. Like a 4GB file to a 5GB ramdisk, to rule out the 980 PRO.

I'll give that a try.

As I wrote, the RAM. It is acting as write cache.

There are multiple sets of RAM. The client, the NAS, and the SSD all have RAM in them so just saying that RAM is a write cache doesn't tell me anything. Hence why I asked if you were referring to the ZIL or SLOG.

I must admit that I did not fully read all the messages in between. But it is generally considered good practice to use file sizes that are so big that caching cannot influence the results.

I have 32G of ram on the client and 64G of ram on the NAS. I'm not sure why I would be concerned about the NAS caching the file but regardless, a 40G file is definitely more than the 32G of ram on my client.

Is that RAIDZ3?
Does that mean that the server and client both have NICs with 10 Gbps?
Same here.

Yes, yes, and yes.

 

[winnielinnie]

The dataset sync is set to disabled. NFS is default.

Since your dataset is set to sync=disabled, then it means all writes are enforced as "async", even though NFS by default would use "sync" (for example, if the property was set to "sync=standard".)

Set your dataset property to sync=standard, then see if your write speeds still maintain 800 MiB/s.

 

[CJRoss]

Since your dataset is set to sync=disabled, then it means all writes are enforced as "async", even though NFS by default would use "sync" (for example, if the property was set to "sync=standard".)

Set your dataset property to sync=standard, then see if your write speeds still maintain 800 MiB/s.

I'm still struggling to see how testing sync writes which go to the ZIL and the disk will give me insight into my reads. What are you trying to test? That writes to disk are slower than reads from them?

 

[CJRoss]

I just tested nconnect=16 and there was no change in read speeds.

 

[CJRoss]

Copying the same two 6G files to a ramdisk ran about 450MiBps. Better than the SSD but not as good as the writes.

This is with all of the same settings as I had originally. Not using nconnect or any other suggested change.