Help to improve my Truenas Scale NFS/write performance performance

[zeusthabomb]

Hello everyone,

I have been researching this topic for a while now, there's way too much topics about it, I tried loads of configurations without sucess.

1. Turning sync off and on
2. Turning compression off and on
3. Playing around with Kernel parameters
4. Playing around with blocksizes

The issue is my write performance is very slow, a transfer on NFS starts at 600+ Mb/s and dips into Kb/s

My setup:
Running TrueNAS-SCALE-22.02.4 on ProLiant MicroServer Gen8 with 16 GB ECC ram, Intel(R) Xeon(R) CPU E31260L @ 2.40GHz, 4 Ironwolfs of 4TB each and Gigabit connection. Running 1vDEV on raidz2

My use case:
Host all my media files, mainly movies in 4k and some backups.

I have a server running on proxmox that will transfer the files via NFS to the truenas server. I have tried a couple of NFS mout options without any sucess of improvements. Right now I'm using ``
I have also tried to rsync from the server to the truenas with also very slow performance, with async and sync.

What could be causing the problem? I can't find the reason or even understand the issue here. Would it improve if I had a ZLOG device doing cache for the writes in the ZFS?

Iperf3 from Server to Truenas


/etc/fstab mount permissions:

NAS:/mnt/whatever /mountpoint nfs noauto,vers=3,nolock,tcp,fg,intr,rsize=16384,wsize=16384,noatime,nodiratime 0 0

Pulling via RSYNC from Server to truenas ( Not even through NFS )


I'm just transfering a big single file not many.

Any advice is welcome, is there is any missing information, sorry.

Thanks for reading.

 

[Davvo]

A SLOG won't help you, it's used as a protection against data loss while using Sync=Always.
You can theoretically improve your vdev write speed adding more disks.

 

[zeusthabomb]

bump

 

[sretalla]

I have a server running on proxmox that will transfer the files via NFS to the truenas server.

NFS will demand sync writes, so with RAIDZ2, it's normal that those will be SLOW. see: https://www.truenas.com/community/threads/the-path-to-success-for-block-storage.81165/

Either don't use NFS (rather SMB) or set sync=disabled on that dataset.

Would it improve if I had a ZLOG device doing cache for the writes in the ZFS?

See the link I posted above, SLOG may help, but not much as your pool design sucks for sync writes, so the slog will quickly be exhausted and you'll be back to pool disk speeds.

Pulling via RSYNC from Server to truenas ( Not even through NFS )

RSYNC over SSH isn't necessarily the best either... consider SMB.

With your pool as it is, you may still see issues.

Perhaps try some testing with fio to see what can be expected.

fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500g --io_size=1500g --blocksize=1m --iodepth=1 --direct=1 --numjobs=1 --runtime=3600 --group_reporting

you can play with the numjobs and iodepth to really test harder and reduce the size of the size and io_size to force more IOPS.

Playing around with blocksizes

My use case:
Host all my media files, mainly movies in 4k and some backups.

By all means, you should set the recordsize for that dataset to 1m