Losing performance.

[IrY100Fan]

Hi Everyone,

A few months ago I installed TrueNAS SCALE on a home-built server for storing mostly audio and video files with a scattering of documents. At first the NFS network performance was pretty good (about 850MB/s read or write) but it started falling off and now fell to about 150MB/s read/write. I rebooted the server but that made no difference. After doing some searching, I found a few suggestions...
1) Change the number of NFS threads - I tried 8 and 16 threads. That makes absolutely no difference.
2) Increase MTU size - Changed it to 9000 (which I think is the maximum). No difference.
3) Disable encryption - It was originally set to LZ4 (default for installation), changed it to None. No difference.
4) Add rsize/wsize to NFS mount options - This increased the throughput to about 450MB/s but still about half of what it used to be.
5) Use SMB instead of NFS - That gets me to the same 450MB/s as above.

I also checked SMART status of drives, no problems reported. Did a SCRUB of the pool and that returned no errors found. Ran "iperf3" on the network and got a result of 9.41Gbps. I am running one virtual machine on this server and have a 2TB dataset attached to it. Ran a disk benchmark in the VM and it reported about 1100MB/s read/write, which sounds about right for my setup. Not sure what to try now and am hoping someone can suggest the next course of action. Below are details of my system.

TrueNAS SCALE (v22.12.3.3)

AMD EPYC 7252 8-Core Processor
ASRockRack ROMED6U-2L2T Motherboard (built-in dual 10Gb Networking)
32GB ECC 3200MHz RAM (4x8G)
(7x) Seagate Exos X20 SAS Hard Drives (as a single RAID-Z2)
Supermicro 9300-8i HBA

Thanks.
-Brian

 

[IrY100Fan]

...bump...

 

[Arwen]

...
3) Disable encryption - It was originally set to LZ4 (default for installation), changed it to None. No difference.
...

Actually, you mean compression. While compression can affect write speed by slowing things down slightly, reads can be faster. Less hard disk head movement, and disk sector transfers. Thus, reads can be faster with more compressible data. However, some pre-compressed data, like video and audio files don't benefit from ZFS transparent compression. (But, raw video and audio files might...) The general rule of thumb is to leave the default LZ4 compression enabled unless you are certain you don't need it.


What block size are you using for your various datasets?

Video and audio file datasets can benefit from larger than the default 128KByte dataset block size. But, smaller files might find something smaller more suited. However, larger block sizes can cause eventual higher levels of fragmentation. Thus, the rule about keeping a pool below 80%.


You might review the various 10Gbps network tuning Resources. Here is one;

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

Next, NFS writes are synchronous and can benefit from a SLOG. Since you are using SATA disks in a RAID-Z2, even a SATA SSD SLOG might help. You can even disable "sync" temporarily and see if that makes a difference. If it does, perhaps a SLOG will help. I am no expert on ZFS SLOGs for NFS writes... so this is more of a subject for you to research.


Last, you don't mention your pool fullness level. Use this command and post the results here in code tags;
zpool list -v
You can remove the boot-pool info from the output.

 

[IrY100Fan]

Hello Arwen,
Thanks for the reply.

I did mean compression not encryption.

Left the block size (record size?) at the default 128K. 90% of the data that will be put on the machine will be large files of audio, video, drive images and zip archives. I shouldn't be anywhere near 80% yet.

I will look through the network tuning resources you linked to.

NFS Sync was already disabled as I know that can make a large improvement in write performance. I enabled it just to see what difference it made. It drops my write speed to about 30MB/s.

I read about SLOG drives but my interpretation of what I read is they don't make that much difference in a single user environment. Maybe I'm wrong. Also, I believe that you need to add at least two drives as a RAID1 for SLOG or you risk losing data integrity, or am I wrong with that?

Output from "zpool list -v"

Code:

NAME                                       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
main                                       127T  1.96T   125T        -         -     0%     1%  1.00x    ONLINE  /mnt
  raidz2-0                                 127T  1.96T   125T        -         -     0%  1.53%      -    ONLINE
    035fcd92-5e41-4685-b2c7-8e6f8eb837ed  18.2T      -      -        -         -      -      -      -    ONLINE
    391bbf2a-5bb3-4af4-83c6-9e85ed94f387  18.2T      -      -        -         -      -      -      -    ONLINE
    c0079fe6-ec2f-46dd-a1c3-dc4a78c5dc42  18.2T      -      -        -         -      -      -      -    ONLINE
    993fdc26-5985-47d3-8410-c144ee844348  18.2T      -      -        -         -      -      -      -    ONLINE
    4c671116-ac29-4829-80dd-a14d06485e43  18.2T      -      -        -         -      -      -      -    ONLINE
    fcb07fb9-2c02-4555-953a-8076ff94094b  18.2T      -      -        -         -      -      -      -    ONLINE
    e5770260-300d-4191-b904-c33b788ba61c  18.2T      -      -        -         -      -      -      -    ONLINE

What confuses me is that for a couple of months the performance was excellent and then it just dropped off. All of these configuration changes and additional hardware might be nice but the system used to perform fine without those. That is what has me baffled.

Again, thanks for the reply and suggestions.

 

[Arwen]

...
I read about SLOG drives but my interpretation of what I read is they don't make that much difference in a single user environment. Maybe I'm wrong. Also, I believe that you need to add at least two drives as a RAID1 for SLOG or you risk losing data integrity, or am I wrong with that?
...

Using a Mirrored, (similar to RAID-1), SLOG can help for extreme reliability. The thing is, a SLOG is only read during a graceless shutdown & boot cycle, (aka crash or forced power off). As long as the SLOG is readable during that SPECIFIC case, no data is lost. Thus, some people risk using an un-Mirrored SLOG.

If an un-Mirrored SLOG fails during normal operation, ZFS simply defaults to in-pool LOG. Slower yes, but still meets the sync write requirements.

Now if a SLOG fails, (un-Mirrored, or both Mirrors fail) during a graceless shutdown & boot cycle, you loose the transactions that were pending. Again some people are willing to risk this. However, for iSCSI zVols, this can be bad... but perhaps not for your NFS use case.


Will a SLOG make a difference in your case?
I don't know, and can't guess.

...
What confuses me is that for a couple of months the performance was excellent and then it just dropped off. All of these configuration changes and additional hardware might be nice but the system used to perform fine without those. That is what has me baffled.
...

Sorry, I can't help with why performance was good early on.

 

[winnielinnie]

However, some pre-compressed data, like video and audio files don't benefit from ZFS transparent compression.

EEEEEEEEEEEEEEEEEEEEEEEEEEEEVERYTHING benefits from inline compression.

Sorry for yelling. Just watched that infamous clip from The Professional.

The reason to always use inline compression (LZ4 or ZLE at minimum) is that it will always "disappear" the extraneous zeros padded to the end of a record. This becomes more evident when using a large recordsize, such as 1MiB.

On top of that, LZ4 employs "early abort", and supposedly so will ZSTD in the future.

 

[winnielinnie]

Left the block size (record size?) at the default 128K. 90% of the data that will be put on the machine will be large files of audio, video, drive images and zip archives. I shouldn't be anywhere near 80% yet.

If these files will not be often edited / modified in-place, you'll have better results with a recordsize of 1MiB.

 

[IrY100Fan]

Just for fun I added a ZFS LOG drive (P1600X 118G Optane) and my write speeds dropped even further. Hmm, that was unexpected. I have used these drives in other systems and I know it can handle about 1000MB/s writes.

Tried some of the suggestions in the High Speed Network Tuning Resources guide. Nothing seemd to make any difference either good or bad.

And I don't believe processing power is a problem. My CPU graph never goes above 7% when trying to move data to and from the NAS.