Read Performance (SMB/FTP/Any)

[paddy01]

Hi,

Super happy with Scale having been running TrueNas/FreeNas for some years now however I have a performance issue I just cannot fathom.

Issue is with read performance whether that's SMB/FTP or any other protocol is way underperforming (or so I believe based on hardware specs etc).

TrueNas server is on 10Gb links with a 10Gb switch and desktop is on 2.5Gb link.

Writing files from Desktop to NAS will saturate the 2.5Gb link of the desktop machine however copying files from Nas to Desktop (even multiple files) I cannot get more than 800Mbps so not evening hitting gigabit speed. I get the same results whether using a spinning disk or ssd based pool.

Iperf3 show 2.5Gb/s both directions so the network doesn't seem to be an issue.

dd tests (compression off) show write speed of 351MB/s and read of 417MB/s for the spinning disk pool so more than enough in both directions for the 2.5Gb link from desktop.

Spinning disks are a 5 disk RaidZ2 and SSD's are a pair in Mirror, all tests to/from desktop (Win10) used PCIe 3 NVME.

At this point I've run out of ideas, does anyone have any suggestions about what to look at next please?

Hardware:

Mobo - ASRockRack D1541D4U-28TR
RAM - 32GB ECC
HBA - LSI SAS3008 (IT Mode)
HDD - 5 x WD Green 5400RPM
Nic - Intel X540

Many thanks,

 

[morganL]

It's not clear what read performance you think is low?.... 417MB/s is quite high.
Only 3 drives of a 5WZ2 are used for reads.

 

[paddy01]

Apologies. Read performance of the pool in dd tests is absolutely fine.

when reading from the pool over an SMB /FTP or other protocol I.e copying a file from TrueNas to my desktop I’m getting less that gigabit speed (around 800Mb/s).

copying files to the Nas I’m getting 2.5Gb/s which is expected due to 2.5Gb nic in desktop.

many thanks

 

[marmoset]

@morganL I believe they are saying that 800Mbps (via ftp or smb) is the low performance part, and that the underlying raw performance (417MB/s or ~3.3Gbps) should be more than enough to do better than 800Mbs over the network.

 

[paddy01]

Exactly right @marmoset

 

[paddy01]

I'll be trying another client machine but in the mean time does anyone have any suggestions of things to look at or try next?

This instance of TrueNas isn't in active use yet so burn and build or maybe even try another version like TrueNAS Core?

Thanks,

 

[paddy01]

Well, after much swapping of components etc. I can only conclude at this point the culprit is most likely a 10G/5G/2.5G/1Gbps transceiver I'm using in a Unifi 16-XG switch to provide 2.5Gb/s to my desktop.

I've tried various other OS's on the (bare metal) server on which I intend to run scale and no change including different disk layouts.

I tried testing from a linux VM on another host with 10Gb connectivity and using a simple cp command from an SMB share was seeing well in excess of 1Gbps transfer and only limited by lower performance CPU on that host (was seeing single core peaking 100%).

I even tried ordering/installing an Intel based 2.5Gbe PCIe card on my desktop thinking the Realtek 2.5Gbe nic could be the issue as they don't have a very good reputation but same issue.

If I use any other 1G or 10G transceiver on the switch running at 1Gbps to desktop I see 1Gbps line rate read/write. Writing at 2.5Gbps is a loss but being able to read at 1Gbps is an improvement over what I was seeing.

I can only see the transceiver as the common component at this point.

I've got a dual X520 card I can drop into my desktop but unfortunately I can't do that and use the 2nd PCIE 3.0 M.2 NVME slot (PCIe x8 slot 3 is disabled if using 2nd M.2 NVme drive, and if I remove that NVMe drive and install the 10G card either the OS doesn't see it at all or I get no display output.

The desktop motherboard (MSI B550 Tomahawk) is proving a little limited in other ways and I'm now getting problems recognising the 2nd NVme drive so I might just throw in the towel and replace motherboard for something that at least disables SATA interfaces instead of M.2 NVMe drives if using multiple PCIe slots. That would then give me the option of straight to 10Gbps from desktop rather than messing about with 2.5Gbe any more and just adding to the parts bin!