New to TrueNAS, Abysmal SMB Read Speed, Write Speed Fine

[anthropogentric]

So, I built a TrueNAS box for the first time, and here's what I'm getting over SMB. This lines up with the general file transfer status I'm seeing in Windows. I'm trying to figure out what I've misconfigured or something

Hardware:

CPU: AMD 2400ge
MOBO: ASRock B450M Pro4 R2.0
RAM: 16GB Unbufferred 2400mhz ECC DDR4
HBA: LSI 9207-8 in it mode
NIC: Mellanox X3
HDDS: 6x 4TB WD Red Pro in Z1, lz4 compression
generic 120GB SSD boot drive,
16GB Intel M10 NVME as cache
TrueNAS version: 13.0-Release

Drive is connected with a 2m copper DAC to an Aruba S2500. Both TrueNAS and the switch are reporting 10gbe.

I've tested with and without the cache drive and it doesn't make a difference in any way. I only included it because I had one lying around, and I realize it's probably not doing what I had initially assumed it would do as the RAM is the actual ZFS cache.

While I have a 10gbe NIC in the box, the Windows computer I'm testing with is just using a standard 1gbe. I haven't thrown the X540 in yet, but I do have it. Obviously won't matter right now with the read speeds like that.

My next thought was network speed, and again, with 1gbe on the other end:

I did a quick read test across the array (I didn't turn compression off, but more looking to see if there was something obvious I was missing):

I've tried googling and I'm seeing some issues with the Mellanox NIC in 12, but that seemed to be fixed a few years ago. Even then, the read speeds were nowhere near as low as I'm seeing here. I'm trying think of a hardware issue that would be so asynchronous - why would writing work fine, yet reading be so slow?

Anybody have any idea?

 

[nap]

have you done an iperf3 with -R to test the other direction? could try iSCSI as a quick sanity check too, if iperf3 looks ok. have you checked to make sure MTUs are matching and there's no fragmentation going on?

 

[anthropogentric]

Definitely something with the network configuration. I'm still running 1500 mtu (at least on TrueNAS), so I'm going to try and figure out the network side. Thanks for the advice.

 

[anthropogentric]

View attachment 56184

Definitely something with the network configuration. I'm still running 1500 mtu (at least on TrueNAS), so I'm going to try and figure out the network side. Thanks for the advice.

Ok, so checking MTU's

Router: 1500
Switch: 1514
Nas: 1500

That has everything working great except the NAS. I bypassed the 10gbe card and just used the 1gbe RJ45 on the motherboard and got exactly the same results - lots of retransmitted packets, terrible read speed, write to the NAS is unaffected.

Checking for fragmentation, I need to lower the MTU from my windows 10 PC to ~1475 to get fragmentation to stop (using this methodology: https://kb.netgear.com/19863/Ping-Test-to-determine-Optimal-MTU-Size-on-Router) but the lowest value I can use in truenas is 1492 - which didn't affect anything.

Is there anything else network wise that I'm missing?

 

[Dice]

16GB Intel M10 NVME as cache

I had a look at the motherboard, I got the suspiscion that there might be some funky interaction with that m2 and the NIC and or the HBA.
I've ran into simlar fuckeries before, where some boards just didnt accept what I wanted to put on them and perform as expected.
I'd start by removing the "cache drive" L2ARC completely from the system, and try to move PCie cards around.

ASRock B450M Pro4 R2.0

Supports AMD AM4 Socket Ryzen™ 2000, 3000, 4000 G-Series, 5000 and 5000 G-Series Desktop Processors; Supports DDR4 3200<span style=color:red;>*</span>; 1 PCIe 3.0 x16, 1 PCIe 2.0 x16, 1 PCIe 2.0 x1; AMD Quad CrossFireX™; Graphics Output:HDMI, DVI-D, D-Sub; 7.1 CH HD Audio (Realtek ALC892/897...

www.asrock.com

 

[paulinventome]

I've just posted a similar question actually. I am seeing terrible read performance as well on a new install.

During the process of the install and various experiments before I had all my drives I did see 500MB up and down to a single SSD zpool, I am running a PCI 10gbe intel card. Since that test there are more drives and more pci cards in there.

At the moment I'm just trying to work out how to troubleshoot but I thought I would comment because I am seeing exactly the same as you.

cheers
Paul

 

[nap]

Odd, I haven't seen problems with reads in my testing.
On my home CORE NAS on 13.0 (1x4wZ1 10TB HDD; 1MiB recordsize) copying from the NAS to a Windows 10 box over 10GbE (MTU1500) I get ~455 MiB/s on first copy, then ~1.08 GiB/s on subsequent copies when the file is in ARC. This was a 4 GiB file, and I copied using Windows Explorer, which is usually a worst case scenario... (it performs equivalently to a synthetic test with iodepth=1)

I don't have the specs in my signature or anything so:
NAS:
Xeon E3-1275 v5 (4c8t @ 3.6GHz)
64 GiB ECC
Chelsio T520
Supermicro X11SSH-LN4F

PC:
i7 8086k
32 GiB RAM
Mellanox ConnectX-3

Switch: Mikrotik CRS305-1G-4S+