Slow TrueNAS internet speed, other server on router factor 10 faster

deniax

Cadet
Joined
Mar 5, 2023
Messages
8
Hi guys,

I'm migrating from Proxmox with LXC container with Plex to (baremetal) TrueNAS Scale , but I'm having internet speed issues, for some weird reason.

My server is in a datacenter, I'm testing from home.

Test scenario's:
  • Speedtest app installed on Truenas, when I connect to it from my home, download speed is 35.5mbps and upload 197mbps . Hence the stutter I have with watching 4k streams
  • Ruled out its my home internet by doing speedtest, getting 570mbit and 284mbit, so download is almost 20x faster
  • Ruled out its the router that is connected to my server doing speedtest , getting 765mbit and 830mbit.
    • My Proxmox server has speedtest app installed on the LXC container, also there speeds of 645mbit and 743mbit
I also did iperf between Truenas and my home, got no higher then 31mbit :(

So as I ruled out home internet and the router in the datacenter, and Proxmox was getting the normal speeds on the same server, Im pretty sure it's "something" in Truenas
The MTU is set to 1500, other settings that can affect this, maybe some driver?

Hardware list:
  • Asus Z790-PLUS
  • Intel i7-13700
  • 128GB (non ECC) 4x 32GB DDR4
  • 7x Seagate exos x16 16TB in RaidZ1 , 1x SSD boot-pool, 1x SSD for cache vdev, 1x SSD for Truechart apps
  • LSI SAS9211-8
  • Intel Ethernet Controller I226-V - Intel Corporation Device 125c (rev 06)
 

Attachments

  • SCR-20230305-vbf.png
    SCR-20230305-vbf.png
    270.6 KB · Views: 124
  • SCR-20230305-tb0.png
    SCR-20230305-tb0.png
    133.9 KB · Views: 103
  • SCR-20230305-t9v.png
    SCR-20230305-t9v.png
    835.8 KB · Views: 113

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
If you do a speedtest to the TrueNAS but from the same LAN.... what do you get?

If you do a ping...from home to trueNAS, do you see any packet drops?
 

deniax

Cadet
Joined
Mar 5, 2023
Messages
8
If you do a speedtest to the TrueNAS but from the same LAN.... what do you get?

If you do a ping...from home to trueNAS, do you see any packet drops?
Heh, point 1 was actually a good idea... but I removed Proxmox as I needed the space
Ping from home to Truenas, no packetloss

MTR from Truenas to home is interesting though, as from TrueNas, to the router (10.1.1.1) there is packetloss , I did a couple of mtr's all with same result
 

Attachments

  • SCR-20230306-42.png
    SCR-20230306-42.png
    129.9 KB · Views: 126

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694
My guess is that is the problem.. poor link between NAS and router. Any packet loss drops bandwidth significantly due to TCP.

Judging by the numbers, its probably the link from TrueNAS to the router?
 

deniax

Cadet
Joined
Mar 5, 2023
Messages
8
Yeah, the first hop (10.1.1.1) is the Ubiquiti Dreammachine, so it seems (maybe its not the main cause) something is wrong between truenas and the router.
But from another side, I ran on this machine Unraid before , it worked well, leading for me to think it maybe is caused by some driver/config issue as nothing changed hardware wise.

I did see that the Intel NIC I226-V has some very new problems in Windows, related to power management that should go away when you disable it, but it seems by default it's already disabled :(

Code:
root@truenas[~]# ethtool --show-eee eno1
EEE settings for eno1:
        EEE status: disabled
        Tx LPI: disabled
        Supported EEE link modes:  Not reported
        Advertised EEE link modes:  Not reported
        Link partner advertised EEE link modes:  Not reported


Another test that I did and made also very strange results is, that I set the port on the router that Truenas is attached to, to 100mbit, to see if that could stabilize the connection (as viewing anything above 30mbit is impossible) , and the results got actually even worse which is a bit baffling...

Code:
[  4] local 192.168.1.52 port 52940 connected to XXXXXXX port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.03 MBytes  8.63 Mbits/sec                 
[  4]   1.00-2.00   sec   228 KBytes  1.87 Mbits/sec                 
[  4]   2.00-3.00   sec   339 KBytes  2.79 Mbits/sec                 
[  4]   3.00-4.00   sec   780 KBytes  6.39 Mbits/sec                 
[  4]   4.00-5.00   sec  1.14 MBytes  9.51 Mbits/sec                 
[  4]   5.00-6.01   sec  1.41 MBytes  11.8 Mbits/sec                 
[  4]   6.01-7.00   sec  1.49 MBytes  12.5 Mbits/sec                 
[  4]   7.00-8.01   sec  1.59 MBytes  13.3 Mbits/sec                 
[  4]   8.01-9.01   sec   114 KBytes   931 Kbits/sec                 
[  4]   9.01-10.00  sec  1.08 MBytes  9.05 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  9.16 MBytes  7.68 Mbits/sec                  sender
[  4]   0.00-10.00  sec  9.07 MBytes  7.61 Mbits/sec                  receiver
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
2,694

deniax

Cadet
Joined
Mar 5, 2023
Messages
8
This article says there is a new driver coming out. Work around is to disable energy Efficient Ethernet.

It seems by default in Truenas its already disabled:

Code:
root@truenas[~]# ethtool --show-eee eno1
EEE settings for eno1:
        EEE status: disabled
        Tx LPI: disabled
        Supported EEE link modes:  Not reported
        Advertised EEE link modes:  Not reported
        Link partner advertised EEE link modes:  Not reported
 

deniax

Cadet
Joined
Mar 5, 2023
Messages
8
What is strange though is that when using downloaders that do multithreaded downloads, there is full speed and also not a single drop in speed, constant 400mbit - 50MB/s. (Datacenter where the server is at is 1Gbit, I limited for now my speed to 400mbit for this test).

So it seems more in single-threaded (?) that there are problems:

- Constant Plex buffering for anything higher then 20-30mbit
- iperf3 showing dramatic speeds

I still think (hope...) this is a NIC/driver problem though. I will buy a intel i210 PCI-e card and try to see if that helps as from what I read, these cards are recommended for TrueNAS.
Unfortunately I'm abroad right now, so that needs to wait 3 weeks before I can go to the datacenter :(
 

Attachments

  • SCR-20230306-gkv.png
    SCR-20230306-gkv.png
    33.4 KB · Views: 130
Top