How to debug this weird TrueNAS one way speed issue

[x130844]

Hi,

Show : See My TrueNas setup

• Motherboard: Gigabyte z87x- ud3h
• CPU: Intel i5 4570s
• RAM: 32GB
• Storage: 5 x 10TB WD (ATA WDC WD120EMFZ-11), RAID z1
• BootDrive: Samsung SSD eve 850 480GB
• LSI 9207-8i (HP H220 last with IT) SAS card.
• Chelsio T540-CR (10gb)
• Mikrotik switch for 10gb: CRS309-1G-8S+ (cables = optical (blue/teal). and SFP+ DAC twinax Cisco compatible on Finisar modules).
• Various other machines (Macs/PC/hackintosh) in network with Sonnet PCI SFP+ cards (Solo10G SFP+ PCIe Card)

Here's the strangest thing, I had the same type of issue months ago, but can't remember how I fixed it (or maybe it fixed itself?) but after the upgrade to TrueNas, the problem is back with a vengeance.

3 Machines in this scenario:

Machine A: windows/dual boot Mac.
Machine B: truenas
Machine C: macOS

All connected to 10gbe Mikrotik switch. (latest switchOS version , SFP twinax + optical mix)

iperf3 between A to B
from A to B = 3gbit/s only. (that's the real problem).
B to A: full 9gbit/s or so. (great)

iperf3 between A and C:
9gbit/s both ways. Perfect.

iperf3 between B and C:
9gbit/s. both ways. Perfect.


=
Machine A seems to be the problem, but what's strange is that it is not software because I can recreate the speed issue if I reboot in windows as well.
And what's weirder to me, if Machine A has an issue, why would it get full speed, both ways when connecting to Machine C?

What am I missing?

(I had disabled flow control on all machines when I had FreeNAS 11.x, once upgraded to TrueNas, I had to disable flow control again.
I use the standard MTU, and I updated the firmware of the NIC on Machine A, same results). (I don't use any tunable. didn't need to in freenas 11).

How would one debug this?
I spent a good 8 hours trying to debug/research this.

Thank you!

 

[Jeremy MItchell]

Check your network cables?

I had exactly the same problem about a week ago (albeit on a wimpy 1Gb network) and it turns out I had a couple of bad cables. Replaced them and back to full speed in both directions.

 

[x130844]

Thank you for replying.
I'd love it if it were only cables, but logically speaking, I don't get it.
Bad cables should prevent the same machine to be full speed if accessing another server. That's what's really puzzling me.
Also, it'd be weird that the cable would turn bad, exactly when the update to TrueNAS happened. (the server does not move, no cables were moved, nothing changed other than the software).
Regardless, one would think that if a cable was bad, or low quality, I'd be able to recreate the issue when connecting to another machine.

Am I missing something?

 

[x130844]

As in:
if the bad scenario is:
iperf3 between A to B where 3gb only in one direction, one would think that there could be a bad cable on either A, or B. since this is the only case where the speed is very bad.

But then, you can see that A is able to do full speed & both ways if connected to C.
so A is good.

And B is able to do full speed & both ways if connected to C too.
So, B is good.

 

[x130844]

ok. big news, weird discovery.

For whatever reasons. my quad port 10gb chelsio card, port 1 (clx0) shows that strange behavior.
but if I use any other port, 1, 2, 3, it works full speed both directions. no problem at all with the speed.
Now, I really don't think it is a hardware issue with that port, since if I use machine C (client) to B (freenas on "bad" NIC port)) it works full speed also both directions.
So I'm still puzzled.
Anyone can think of a setting in the NIC in Truenas which could effect 1 port only to behave strangely in 1 way depending on the client connected to it. (all machines go through the MikroTik managed switch).