File copy speed (read) slow

[Roaders]

Hi All
I am just setting up a new TrueNas scale system and I want to get as good performance as I can out of my network and hardware. WHne I first set this up I was getting consistent 280MBps read speeds from my box over my 2.5gbps network - so the network was saturated which is obviously all I can ask for. Write speeds were not as consistent and I think started at that sort of speed but then slowed down - presumably as the RAM got filled up.

I did a test this morning and read speeds were around 60MBps... not what I am expecting or hoping for. How can I go about starting to diagnose this???

My system runs on a 11th Gen Intel(R) Core(TM) i5-11400F @ 2.60GHz. I have 32 GB of memory

The pool in question is 5 Ironwolf spinning disks. There are 2 4TB and 2 6TB (I am replacing gradually) and one spare 4TB.
There is also a cache made up of 2 500GB nvme drives.

 

[Davvo]

There is also a cache made up of 2 500GB nvme drives.

Due to the way It works, L2ARC is harmful under 64GB of RAM; it also doesn't need to have redundancy.

I did a test this morning and read speeds were around 60MBps... not what I am expecting or hoping for. How can I go about starting to diagnose this??

Describe the test and the files size used for both this and the 280MBps one.

 

[Roaders]

Thanks for the reply. I tried removing the cache already and the read speed did not improve at all. So I should not use a cache at all then unless I have 64 GB of ram? Interesting.

I used the same file for both the 60MBps and 280Mbps test. It's a roungly 18GB video file. Both tests were simply copying the file from the SMB share to my windows machine.
Sometimes when copying from the spinning disk pool the read speed will be around 280 briefly but then dip down to 60. I can't see any significant changes in RAM usage when this happens.

 

[Roaders]

I just found this command on another thread. These numbers look useful but I don't know what they mean!