bridge causes slow smb read speed

[EtienneB]

Hi,
When I use a bridge between my 10Gb NIC and 1Gb NIC, the SMB read speeds are slow. Write is fine.

iperf3 -B <macbook> -c <server> -t 60 gives around 8-9Gbit for send/receive.
iperf3 -R -B <macbook> -c <server> -t 60 gives me only 900Mbit.......

Using blackmagic test, I get like 800mbyte/s write speed but only 110 mbyte read speed over SMB.

When I remove the bridge, blackmagic speed test writes and reads about 900Mbyte/sec to and from my pool.

Changing the hardware offloading doesn't really help. It makes things worse it seems.

The bridge is not used for jails/VMs, they have a separate port on a 1GB NIC.

I am running Core 13-u6.1. I am not sure if the speed changed by the latest TN updates or not as I upgraded my server last week. I also then did a clean install of 13u6 to SSDs and then upgrade to 6.1 last week (so no older legacy versions left on the boot pool; imported the config).

Any thoughts?
Thanks.

 

[EtienneB]

I troubleshooted some more.
igb1: 1Gb network adapter
ix1: 10Gb network adapter

when I remove igb1 from the bridge1, and thus only have ix1 on it, I get good read/write of 500-800 mbytes/sec.
when I then re-add igb1, the read speed drops to 110Mbytes/sec.

There is only 1 cable running between my mac and server, and that is directly to the ix1 port.

I also noticed this: hardware offloading is enabled (disabled is unchecked) and the speed is around 3.5Gbps write, then I disable the hardware offloading, it drops to around 2.1Gbs write. When I then enable (uncheck disable) hardware offloading again, the write goes up to 9.3Gbps.
So somehow enabling/disabling/enabling hardware offloading, resets the adapter and gives me better performance. And I have to do this after each reboot.
Read speeds are not influenced by these changes of hardware offloading.

Perhaps move over to Scale for better networking performance?

 

[EtienneB]

Some further testing and trail-and-error.

I connected/bridged ix0 and ix1, both ports being on the Intel 10Gb network card.
So the macbook is connected directly to ix1 giving 10Gb, and ix0 is connected to my switch at 1Gb.

(igb0 and igb1 are the 1Gbps ports on my Supermicro X10SL7-F mobo)

This does give me a better read performance, hitting around 500mbyte/s. Writing is about 800mbyte/s.

What I had to do after each boot is toggling the hardware offloading for both nics on and off again. Somehow it needs that reset/refresh. Else the bridge is stuck around 2-3-4 Gbps according to iperf3.

Without the bridge speeds read/write can hit 900+ mbyte/sec.

(Time for a physical switch I guess.)