Slow read performance

[mrXHEE2XTbF6]

I am experiencing slow read performance on my TrueNAS R20. I have one pool using 3.5 TB of storage with 16.29 available. 9 drives 3.64 TB drives configured using RAIDZ3. I have one drive for Caching and one drive for logs. Copying files to the TrueNAS seem to be ok, not great, but useable( 30MBps over 1 Gbps). Copying files from TrueNAS hover between 1.5 and 2.0 MBps.

Looking at the reports for the drives, it does not appear to be using the cache from memory or from the drive.

latency on the 9 data drives in the pool seem to have extremely high read latency.

I bought this system as a turn key solution, but when contacting support, I found out the have separate contracts for hardware and software. I've searched through these forums and have not been able to find anything to solve my issue. It seems like memory is caching some things, but the SSD is not caching anything at all.


Any help is appreciated!

 

[HoneyBadger]

What's the model of the drives you've got?

Does arcstat show a bunch of cache misses when you're doing these slow reads?

 

[morganL]

Even with no caching, its very slow.... there is something wrong. My guess is the disk's don't appear to be very busy.. can you confirm.

Are you copying files from one NAS folder to another folder? or back to your PC?

Copying is often slower than reading a file... the copy software does one I/O at a time.

Have you checked the network speeds (iperf) ?

 

[mrXHEE2XTbF6]

What's the model of the drives you've got?

Does arcstat show a bunch of cache misses when you're doing these slow reads?

The Drives are Western Digital HUS726T4TAL4201 4TB

------------------------------------------------------------------------
ZFS Subsystem Report Mon May 09 10:56:34 2022
FreeBSD 12.2-RELEASE-p14 zpl version 5
Machine: spa version 5000

ARC status: HEALTHY
Memory throttle count: 0

ARC size (current): 81.2 % 23.0 GiB
Target size (adaptive): 81.2 % 23.0 GiB
Min size (hard limit): 3.5 % 1011.6 MiB
Max size (high water): 28:1 28.4 GiB
Most Frequently Used (MFU) cache size: 59.8 % 13.0 GiB
Most Recently Used (MRU) cache size: 40.2 % 8.7 GiB
Metadata cache size (hard limit): 75.0 % 21.3 GiB
Metadata cache size (current): 16.0 % 3.4 GiB
Dnode cache size (hard limit): 10.0 % 2.1 GiB
Dnode cache size (current): 22.8 % 496.8 MiB

ARC hash breakdown:
Elements max: 2.5M
Elements current: 99.8 % 2.4M
Collisions: 1.7M
Chain max: 7
Chains: 489.4k

ARC misc:
Deleted: 974.0k
Mutex misses: 43
Eviction skips: 5.5k

 

[mrXHEE2XTbF6]

Even with no caching, its very slow.... there is something wrong. My guess is the disk's don't appear to be very busy.. can you confirm.

Are you copying files from one NAS folder to another folder? or back to your PC?

Copying is often slower than reading a file... the copy software does one I/O at a time.

Have you checked the network speeds (iperf) ?

Copying from NAS folder to PC. This also occurs with our linux servers that mount to the truenas.

Iperf seems to have the expected speeds for a 1Gbps link.

[ 1] 0.00-10.03 sec 1.07 GBytes 917 Mbits/sec

 

[morganL]

Copying from NAS folder to PC. This also occurs with our linux servers that mount to the truenas.

Iperf seems to have the expected speeds for a 1Gbps link.

[ 1] 0.00-10.03 sec 1.07 GBytes 917 Mbits/sec

Can you report-a-bug or contact the iX support team if you have a support agreement.

I assume this must have been working with CORE before you changed to SCALE?

 

[mrXHEE2XTbF6]

Can you report-a-bug or contact the iX support team if you have a support agreement.

I assume this must have been working with CORE before you changed to SCALE?

This is CORE. This is my first and only deployment of this box. I have reached out to iX support team, but I only have a hardware contract. When I upload the debug to support, they tell me there doesn't appear to by anything wrong with the hardware.

 

[mrXHEE2XTbF6]

does anyone have any idea on what might be the cause? If you need more information I'd be happy to provide it.

 

[mav@]

I'd say it must be something with network to be THAT SLOW. You may try to run iperf in other direction just to be sure. You may also look on `systat -tcp 1` statistics while copying, looking for timeouts, retransmits, duplicate acks, that would speak for some network problems.

 

[johnbajer]

Hello, I have the same issue. I use Truenas 13.1
with strait connection using both ends intel chipset will not read more than 1/2 of the writing speed.
I have the same numbers when using Realtek 2.5 gigabit both ends.

 

[UnoSee]

I found a fix for my system. It appears that my PC was connected to a mesh router via an Ethernet cable, but the cable between the router I was connected to and the main router was broken. I ran a temporary Ethernet cable and it fixed the problem. Before, my PC was reading from my Truenas at around 50Mbps, but now it runs at full gigabit. I initially thought it was an issue with my drive, as it had some ZFS errors, but it turns out that it was my connection that was causing the problem.