Slow TrueNAS internet speed, other server on router factor 10 faster

[deniax]

Hi guys,

I'm migrating from Proxmox with LXC container with Plex to (baremetal) TrueNAS Scale , but I'm having internet speed issues, for some weird reason.

My server is in a datacenter, I'm testing from home.

Test scenario's:

• Speedtest app installed on Truenas, when I connect to it from my home, download speed is 35.5mbps and upload 197mbps . Hence the stutter I have with watching 4k streams
• Ruled out its my home internet by doing speedtest, getting 570mbit and 284mbit, so download is almost 20x faster
• Ruled out its the router that is connected to my server doing speedtest , getting 765mbit and 830mbit.
  • My Proxmox server has speedtest app installed on the LXC container, also there speeds of 645mbit and 743mbit

I also did iperf between Truenas and my home, got no higher then 31mbit :(

So as I ruled out home internet and the router in the datacenter, and Proxmox was getting the normal speeds on the same server, Im pretty sure it's "something" in Truenas
The MTU is set to 1500, other settings that can affect this, maybe some driver?

Hardware list:

• Asus Z790-PLUS
• Intel i7-13700
• 128GB (non ECC) 4x 32GB DDR4
• 7x Seagate exos x16 16TB in RaidZ1 , 1x SSD boot-pool, 1x SSD for cache vdev, 1x SSD for Truechart apps
• LSI SAS9211-8
• Intel Ethernet Controller I226-V - Intel Corporation Device 125c (rev 06)

 

[morganL]

If you do a speedtest to the TrueNAS but from the same LAN.... what do you get?

If you do a ping...from home to trueNAS, do you see any packet drops?

 

[deniax]

If you do a speedtest to the TrueNAS but from the same LAN.... what do you get?

If you do a ping...from home to trueNAS, do you see any packet drops?

Heh, point 1 was actually a good idea... but I removed Proxmox as I needed the space
Ping from home to Truenas, no packetloss

MTR from Truenas to home is interesting though, as from TrueNas, to the router (10.1.1.1) there is packetloss , I did a couple of mtr's all with same result

 

[morganL]

My guess is that is the problem.. poor link between NAS and router. Any packet loss drops bandwidth significantly due to TCP.

Judging by the numbers, its probably the link from TrueNAS to the router?

 

[deniax]

Yeah, the first hop (10.1.1.1) is the Ubiquiti Dreammachine, so it seems (maybe its not the main cause) something is wrong between truenas and the router.
But from another side, I ran on this machine Unraid before , it worked well, leading for me to think it maybe is caused by some driver/config issue as nothing changed hardware wise.

I did see that the Intel NIC I226-V has some very new problems in Windows, related to power management that should go away when you disable it, but it seems by default it's already disabled :(

Code:

root@truenas[~]# ethtool --show-eee eno1
EEE settings for eno1:
        EEE status: disabled
        Tx LPI: disabled
        Supported EEE link modes:  Not reported
        Advertised EEE link modes:  Not reported
        Link partner advertised EEE link modes:  Not reported

Another test that I did and made also very strange results is, that I set the port on the router that Truenas is attached to, to 100mbit, to see if that could stabilize the connection (as viewing anything above 30mbit is impossible) , and the results got actually even worse which is a bit baffling...

Code:

[  4] local 192.168.1.52 port 52940 connected to XXXXXXX port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.03 MBytes  8.63 Mbits/sec                 
[  4]   1.00-2.00   sec   228 KBytes  1.87 Mbits/sec                 
[  4]   2.00-3.00   sec   339 KBytes  2.79 Mbits/sec                 
[  4]   3.00-4.00   sec   780 KBytes  6.39 Mbits/sec                 
[  4]   4.00-5.00   sec  1.14 MBytes  9.51 Mbits/sec                 
[  4]   5.00-6.01   sec  1.41 MBytes  11.8 Mbits/sec                 
[  4]   6.01-7.00   sec  1.49 MBytes  12.5 Mbits/sec                 
[  4]   7.00-8.01   sec  1.59 MBytes  13.3 Mbits/sec                 
[  4]   8.01-9.01   sec   114 KBytes   931 Kbits/sec                 
[  4]   9.01-10.00  sec  1.08 MBytes  9.05 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  9.16 MBytes  7.68 Mbits/sec                  sender
[  4]   0.00-10.00  sec  9.07 MBytes  7.61 Mbits/sec                  receiver

 

[morganL]

Don't know...its not clear whether the I226-V NIC chip is having hardware or driver issues.

Intel is experiencing network issues: The I226-V controller is prone to connection loss

An issue with the Intel I226-V, a 2.5 Gb/s Ethernet controller found in Intel Z790, H770, and B760 motherboards, is causing an increase in network faults. These motherboards use the Intel Socket 1700 ...

www.guru3d.com

Anyone else running this chip?

 

[morganL]

This article says there is a new driver coming out. Work around is to disable energy Efficient Ethernet.

Intel Releases New Driver To Mitigate I226 & I225 Ethernet Controller Issues

Intel is releasing a new driver to mitigate connection drop issues faced by customers running boards with I226 & I225 ethernet controllers.

wccftech.com

 

[deniax]

This article says there is a new driver coming out. Work around is to disable energy Efficient Ethernet.

Intel Releases New Driver To Mitigate I226 & I225 Ethernet Controller Issues

Intel is releasing a new driver to mitigate connection drop issues faced by customers running boards with I226 & I225 ethernet controllers.

wccftech.com

It seems by default in Truenas its already disabled:

Code:

root@truenas[~]# ethtool --show-eee eno1
EEE settings for eno1:
        EEE status: disabled
        Tx LPI: disabled
        Supported EEE link modes:  Not reported
        Advertised EEE link modes:  Not reported
        Link partner advertised EEE link modes:  Not reported

 

[deniax]

What is strange though is that when using downloaders that do multithreaded downloads, there is full speed and also not a single drop in speed, constant 400mbit - 50MB/s. (Datacenter where the server is at is 1Gbit, I limited for now my speed to 400mbit for this test).

So it seems more in single-threaded (?) that there are problems:

- Constant Plex buffering for anything higher then 20-30mbit
- iperf3 showing dramatic speeds

I still think (hope...) this is a NIC/driver problem though. I will buy a intel i210 PCI-e card and try to see if that helps as from what I read, these cards are recommended for TrueNAS.
Unfortunately I'm abroad right now, so that needs to wait 3 weeks before I can go to the datacenter :(