Scale performance issues, filetransfer stops and continues random

[z3us]

Running Scale on new custom hardware with 10Gbe adapter.
SMART enabled on all drives, all without errors.

I have randomly issues with transfers over SMB. Transfers suddenly run on 0% speed. Then after a few seconds it continues copying.
Very weird.
Only SSH and SMB services active.

Hardware:
i5-8600 @3.1Ghz
64GB memory
4x 8TB in 1 pool
2x 5TB in 1 pool
2x 2TB in 1 pool (system pool)
2x 1TB in 1 pool
OS on NVMe drive onboard
H200 in IT mode controller
Intel 540 10Gbe fiber
Onboard 1Gbe ethernet

Version:
TrueNAS-SCALE-22.02.0

Maybe this log entry has something to do with this issue?
syslog shows every x seconds:

FileNotFoundError: [Errno 2] No such file or directory: '/proc/net/rpc/nfsd'
Feb 25 15:24:35 scale collectd[9365]: Traceback (most recent call last):
File "/usr/local/lib/collectd_pyplugins/nfsstat.py", line 23, in read
with open("/proc/net/rpc/nfsd", "r") as f:

Anyone know what above means, maybe this causes the issue?
There is no nfsd at this fil location...

Thanks in advance!

 

[HoneyBadger]

Please post the following information:

- Drive makes/models (to rule out SMR)
- Amount of used vs free space in each pool (impacts fragmentation/write speed)
- How are the drives connected (which are using HBA vs onboard SATA or if they are going through an expander)

If you are trying to throw 10Gbps connectivity at spinning disks you may at some point outstrip your drive's write speeds.

 

[z3us]

Thanks for your reply, @HoneyBadger

Fixed my issue removing the 1TB and 2TB pool, they were degraded although SMART tests result in zero issue with the drives.
After removing the datasets and pools, shutdown the server, removed the disks and booted.
New filecopies without issues now.

The FileNotFound error went away after enabling NFS service, it immediately stopped because I haven't configured it.
Strange enough, the error stopped in the syslog....

Solved, may be closed, thanks again!

 

[anodos]

Thanks for your reply, @HoneyBadger

Fixed my issue removing the 1TB and 2TB pool, they were degraded although SMART tests result in zero issue with the drives.
After removing the datasets and pools, shutdown the server, removed the disks and booted.
New filecopies without issues now.

The FileNotFound error went away after enabling NFS service, it immediately stopped because I haven't configured it.
Strange enough, the error stopped in the syslog....

Solved, may be closed, thanks again!

nfsstat issue is fixed for 22.02.1 - https://github.com/truenas/middleware/pull/8349/files