Super high ping latency w/ Samba performance issues | TrueNAS SCALE

[Sawtaytoes]

I had a previous thread with similar problems on the same server, but these are different issues as I'm now using TrueNAS SCALE.

Previous thread: https://www.truenas.com/community/threads/random-ping-drops-samba-freezing-web-super-slow.106075/

• TrueNAS-SCALE-22.12.0
• ASRock Rack ROMED8-2T/BCM (Using 1 of the onboard Broadcom BCM57416 10Gb NICs connected to an 8-port UniFi Aggregation Switch with a MikroTik SFP+ adapter).
• Eypc 7323p
• 128GB ECC RAM
• 3 x LSI 9305 24i
• 1 x ASUS Hyper M.2 PCIe 3.0 V2
• 2 x 60GB SSDs (boot)
• 2 x 256GB SSDs (TrueNAS Apps)
• 8 x Intel Optane 905p (currently unused)
• 25 x 2TB SSDs (stripe of 2-drive mirrors)
• 7 x 4TB SSDs (stripe of 2-drive mirrors)

My issue:

Code:

Reply from 10.1.0.6: bytes=32 time=5ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=7ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=7ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=7ms TTL=64
Reply from 10.1.0.6: bytes=32 time=7ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=5ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=5ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=3ms TTL=64
Reply from 10.1.0.6: bytes=32 time=7ms TTL=64

• Ping is bad to and from the server.
• Pinging other machines from the server shows similar high-latency.
• Pinging other machines from my desktop shows `<1ms` like I'd expect including the IPMI NIC.
• Pinging localhost shows results in the `0.037-0.049ms` range.

This also affects the speed of file transfers and Samba as I get very inconsistent results and very slow transfer speeds around 25-30MB/s.

I have another TrueNAS SCALE install with 36 HDDs and 4 SSDs. The hardware is different, but it doesn't have these ping issues. They only occur on the two onboard Broadcom BCM57416 NICs.

TrueNAS SCALE isn't showing anything going over the wire:

Things I've tried:

• Restarting the server.
• Reinstalling the OS on new drives.
• Swapping the MikroTik 10Gb SFP+ adapter with another.
• Using a different Ethernet jack on a 1Gb UniFi switch rather than the 10Gb one.
• Using the other onboard Broadcom BCM57416.
• Disabling bonding in the ASRock Rack IPMI Management Interface.

 

[simonj]

Have you ever tried with another 10Gb ethernet adapter? As suggested in your other thread those Broadcom NICs might be the problem.
We are using the Intel T540s and they are very consistent and get us near full 10Gb speeds.

 

[Sawtaytoes]

I looks like 3 of the MikroTik adapters I had were faulty. I sent them back.

I had to physically power down the machine by unplugging both PSUs, and then the correct speed was back. I haven't had it happen since.

Another thing I learned was that MikroTik advises you keep the adapters more than 1 port apart! Ubiquiti doesn't advise that for their adapters, so I didn't think it mattered nor have I seen that from any documentation other than their own.

Either way, I bought a new switch and did space out my adapters just in case it happens with the old ones; although, I'm pretty sure this whole issue was limited to those 3 adapters.