Network transfer speeds drop after some time regardless of protocol and disk usage

[codelett]

Hello,

my TrueNAS instance behaves weird for some months now and I have no idea why.

Network transfer speeds drop significantly after a lot of data has been transferred.
Please see this iperf3 log:

Code:

$ iperf3 -c 192.168.178.200 -p 5201 -w 64K -n 50G
Connecting to host 192.168.178.200, port 5201
[  5] local 172.28.249.71 port 48470 connected to 192.168.178.200 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   939 Mbits/sec    0    188 KBytes
[  5]   1.00-2.00   sec   112 MBytes   938 Mbits/sec    0    188 KBytes
[  5]   2.00-3.00   sec   112 MBytes   937 Mbits/sec    0    188 KBytes
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec    0    188 KBytes
[  5]   4.00-5.00   sec   112 MBytes   936 Mbits/sec    0    188 KBytes
[  5]   5.00-6.00   sec   112 MBytes   939 Mbits/sec    0    188 KBytes
[  5]   6.00-7.00   sec   112 MBytes   936 Mbits/sec    0    188 KBytes
[  5]   7.00-8.00   sec   112 MBytes   938 Mbits/sec    0    188 KBytes
[  5]   8.00-9.00   sec   112 MBytes   936 Mbits/sec    0    188 KBytes
[  5]   9.00-10.00  sec   112 MBytes   937 Mbits/sec    0    188 KBytes
[  5]  10.00-11.00  sec   112 MBytes   937 Mbits/sec    0    188 KBytes
[  5]  11.00-12.00  sec   112 MBytes   938 Mbits/sec    1    158 KBytes
[  5]  12.00-13.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  13.00-14.00  sec   112 MBytes   935 Mbits/sec    0    185 KBytes
[  5]  14.00-15.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  15.00-16.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  16.00-17.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  17.00-18.00  sec   111 MBytes   933 Mbits/sec    0    185 KBytes
[  5]  18.00-19.00  sec   111 MBytes   931 Mbits/sec    0    185 KBytes
[  5]  19.00-20.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  20.00-21.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  21.00-22.00  sec   112 MBytes   935 Mbits/sec    0    185 KBytes
[  5]  22.00-23.00  sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5]  23.00-24.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  24.00-25.00  sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5]  25.00-26.00  sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5]  26.00-27.00  sec   111 MBytes   932 Mbits/sec    0    185 KBytes
[  5]  27.00-28.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  28.00-29.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  29.00-30.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  30.00-31.00  sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5]  31.00-32.00  sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5]  32.00-33.00  sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5]  33.00-34.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  34.00-35.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  35.00-36.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  36.00-37.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  37.00-38.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  38.00-39.00  sec   111 MBytes   932 Mbits/sec    0    185 KBytes
[  5]  39.00-40.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  40.00-41.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  41.00-42.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  42.00-43.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  43.00-44.00  sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5]  44.00-45.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  45.00-46.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  46.00-47.00  sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5]  47.00-48.00  sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5]  48.00-49.00  sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5]  49.00-50.00  sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5]  50.00-51.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  51.00-52.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  52.00-53.00  sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5]  53.00-54.00  sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5]  54.00-55.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  55.00-56.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  56.00-57.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  57.00-58.00  sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5]  58.00-59.00  sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5]  59.00-60.00  sec   109 MBytes   917 Mbits/sec    0    284 KBytes
[  5]  60.00-61.00  sec   108 MBytes   909 Mbits/sec    0    284 KBytes
[  5]  61.00-62.00  sec   109 MBytes   919 Mbits/sec    0    284 KBytes
[  5]  62.00-63.00  sec   109 MBytes   916 Mbits/sec    0    284 KBytes
[  5]  63.00-64.00  sec   111 MBytes   932 Mbits/sec    0    284 KBytes
[  5]  64.00-65.00  sec   110 MBytes   924 Mbits/sec    0    284 KBytes
[  5]  65.00-66.00  sec   111 MBytes   931 Mbits/sec    0    284 KBytes
[  5]  66.00-67.00  sec   110 MBytes   920 Mbits/sec    0    284 KBytes
[  5]  67.00-68.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  68.00-69.00  sec   110 MBytes   919 Mbits/sec    0    284 KBytes
[  5]  69.00-70.00  sec   111 MBytes   930 Mbits/sec    0    284 KBytes
[  5]  70.00-71.00  sec   110 MBytes   926 Mbits/sec    0    284 KBytes
[  5]  71.00-72.00  sec   112 MBytes   940 Mbits/sec    0    284 KBytes
[  5]  72.00-73.00  sec   111 MBytes   933 Mbits/sec    0    284 KBytes
[  5]  73.00-74.00  sec   112 MBytes   938 Mbits/sec    0    284 KBytes
[  5]  74.00-75.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  75.00-76.00  sec   111 MBytes   934 Mbits/sec    0    284 KBytes
[  5]  76.00-77.00  sec   112 MBytes   938 Mbits/sec    0    284 KBytes
[  5]  77.00-78.00  sec   112 MBytes   938 Mbits/sec    0    284 KBytes
[  5]  78.00-79.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  79.00-80.00  sec   112 MBytes   936 Mbits/sec    0    284 KBytes
[  5]  80.00-81.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  81.00-82.00  sec   112 MBytes   938 Mbits/sec    0    284 KBytes
[  5]  82.00-83.00  sec   111 MBytes   934 Mbits/sec    0    284 KBytes
[  5]  83.00-84.00  sec   111 MBytes   931 Mbits/sec    0    284 KBytes
[  5]  84.00-85.00  sec   111 MBytes   934 Mbits/sec    0    284 KBytes
[  5]  85.00-86.00  sec   112 MBytes   936 Mbits/sec    0    284 KBytes
[  5]  86.00-87.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  87.00-88.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  88.00-89.00  sec   111 MBytes   930 Mbits/sec    0    284 KBytes
[  5]  89.00-90.00  sec   112 MBytes   935 Mbits/sec    0    284 KBytes
[  5]  90.00-91.00  sec   112 MBytes   937 Mbits/sec    0    284 KBytes
[  5]  91.00-92.00  sec   111 MBytes   935 Mbits/sec    0    284 KBytes
[  5]  92.00-93.00  sec   112 MBytes   938 Mbits/sec    0    284 KBytes
[  5]  93.00-94.00  sec   111 MBytes   931 Mbits/sec    1    198 KBytes
[  5]  94.00-95.00  sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5]  95.00-96.00  sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5]  96.00-97.00  sec   110 MBytes   924 Mbits/sec    0    198 KBytes
[  5]  97.00-98.00  sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5]  98.00-99.00  sec   111 MBytes   931 Mbits/sec    0    198 KBytes
[  5]  99.00-100.00 sec   111 MBytes   933 Mbits/sec    0    198 KBytes
[  5] 100.00-101.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 101.00-102.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 102.00-103.00 sec   111 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 103.00-104.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 104.00-105.00 sec   111 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 105.00-106.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 106.00-107.00 sec   112 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 107.00-108.00 sec   112 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 108.00-109.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 109.00-110.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 110.00-111.00 sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5] 111.00-112.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 112.00-113.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 113.00-114.00 sec   111 MBytes   930 Mbits/sec    0    198 KBytes
[  5] 114.00-115.00 sec   112 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 115.00-116.00 sec   111 MBytes   932 Mbits/sec    0    198 KBytes
[  5] 116.00-117.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 117.00-118.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 118.00-119.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 119.00-120.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 120.00-121.00 sec   111 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 121.00-122.00 sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5] 122.00-123.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 123.00-124.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 124.00-125.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 125.00-126.00 sec   112 MBytes   941 Mbits/sec    0    198 KBytes
[  5] 126.00-127.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 127.00-128.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 128.00-129.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 129.00-130.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 130.00-131.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 131.00-132.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 132.00-133.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 133.00-134.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 134.00-135.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 135.00-136.00 sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5] 136.00-137.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 137.00-138.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 138.00-139.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 139.00-140.00 sec   112 MBytes   940 Mbits/sec    0    198 KBytes
[  5] 140.00-141.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 141.00-142.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 142.00-143.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 143.00-144.00 sec   112 MBytes   940 Mbits/sec    0    198 KBytes
[  5] 144.00-145.00 sec   111 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 145.00-146.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 146.00-147.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 147.00-148.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 148.00-149.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 149.00-150.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 150.00-151.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 151.00-152.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 152.00-153.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 153.00-154.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 154.00-155.00 sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5] 155.00-156.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 156.00-157.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 157.00-158.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 158.00-159.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 159.00-160.00 sec   112 MBytes   939 Mbits/sec    0    198 KBytes
[  5] 160.00-161.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 161.00-162.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 162.00-163.00 sec   112 MBytes   936 Mbits/sec    0    198 KBytes
[  5] 163.00-164.00 sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5] 164.00-165.00 sec   112 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 165.00-166.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 166.00-167.00 sec   111 MBytes   935 Mbits/sec    0    198 KBytes
[  5] 167.00-168.00 sec   112 MBytes   937 Mbits/sec    0    198 KBytes
[  5] 168.00-169.00 sec   111 MBytes   932 Mbits/sec    0    198 KBytes
[  5] 169.00-170.00 sec   112 MBytes   938 Mbits/sec    0    198 KBytes
[  5] 170.00-171.00 sec   111 MBytes   934 Mbits/sec    0    198 KBytes
[  5] 171.00-172.00 sec   111 MBytes   933 Mbits/sec    1    154 KBytes
[  5] 172.00-173.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 173.00-174.00 sec   111 MBytes   932 Mbits/sec    0    185 KBytes
[  5] 174.00-175.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 175.00-176.00 sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 176.00-177.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 177.00-178.00 sec   111 MBytes   933 Mbits/sec    0    185 KBytes
[  5] 178.00-179.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 179.00-180.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 180.00-181.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 181.00-182.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 182.00-183.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 183.00-184.00 sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5] 184.00-185.00 sec   111 MBytes   933 Mbits/sec    0    185 KBytes
[  5] 185.00-186.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 186.00-187.00 sec   110 MBytes   926 Mbits/sec    0    185 KBytes
[  5] 187.00-188.00 sec   111 MBytes   929 Mbits/sec    0    185 KBytes
[  5] 188.00-189.00 sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5] 189.00-190.00 sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 190.00-191.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 191.00-192.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 192.00-193.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 193.00-194.00 sec   112 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 194.00-195.00 sec   111 MBytes   932 Mbits/sec    0    185 KBytes
[  5] 195.00-196.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 196.00-197.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 197.00-198.00 sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 198.00-199.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 199.00-200.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 200.00-201.00 sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 201.00-202.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 202.00-203.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 203.00-204.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 204.00-205.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 205.00-206.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 206.00-207.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 207.00-208.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 208.00-209.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 209.00-210.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 210.00-211.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 211.00-212.00 sec   112 MBytes   940 Mbits/sec    0    185 KBytes
[  5] 212.00-213.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 213.00-214.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 214.00-215.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 215.00-216.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 216.00-217.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 217.00-218.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 218.00-219.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 219.00-220.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 220.00-221.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 221.00-222.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 222.00-223.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 223.00-224.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 224.00-225.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 225.00-226.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 226.00-227.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 227.00-228.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 228.00-229.00 sec   112 MBytes   940 Mbits/sec    0    185 KBytes
[  5] 229.00-230.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 230.00-231.00 sec   112 MBytes   940 Mbits/sec    0    185 KBytes
[  5] 231.00-232.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 232.00-233.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 233.00-234.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 234.00-235.00 sec   112 MBytes   940 Mbits/sec    0    185 KBytes
[  5] 235.00-236.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 236.00-237.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 237.00-238.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 238.00-239.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 239.00-240.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 240.00-241.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 241.00-242.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 242.00-243.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 243.00-244.00 sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 244.00-245.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 245.00-246.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 246.00-247.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 247.00-248.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 248.00-249.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 249.00-250.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 250.00-251.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 251.00-252.00 sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5] 252.00-253.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 253.00-254.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 254.00-255.00 sec   112 MBytes   939 Mbits/sec    0    185 KBytes
[  5] 255.00-256.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 256.00-257.00 sec   111 MBytes   931 Mbits/sec    0    185 KBytes
[  5] 257.00-258.00 sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5] 258.00-259.00 sec   111 MBytes   934 Mbits/sec    0    185 KBytes
[  5] 259.00-260.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 260.00-261.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 261.00-262.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 262.00-263.00 sec   112 MBytes   937 Mbits/sec    0    185 KBytes
[  5] 263.00-264.00 sec   112 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 264.00-265.00 sec   111 MBytes   935 Mbits/sec    0    185 KBytes
[  5] 265.00-266.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 266.00-267.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 267.00-268.00 sec   112 MBytes   938 Mbits/sec    0    185 KBytes
[  5] 268.00-269.00 sec   112 MBytes   936 Mbits/sec    0    185 KBytes
[  5] 269.00-270.00 sec  2.19 MBytes  18.4 Mbits/sec  144   1.41 KBytes
[  5] 270.00-271.00 sec   327 KBytes  2.68 Mbits/sec  108   1.41 KBytes
[  5] 271.00-272.00 sec   428 KBytes  3.51 Mbits/sec   55   15.6 KBytes
[  5] 272.00-273.00 sec   243 KBytes  1.99 Mbits/sec   32   14.1 KBytes
[  5] 273.00-274.00 sec   365 KBytes  2.99 Mbits/sec   42   1.41 KBytes
[  5] 274.00-275.00 sec   304 KBytes  2.49 Mbits/sec   45   14.1 KBytes
[  5] 275.00-276.00 sec   365 KBytes  2.99 Mbits/sec   42   14.1 KBytes
[  5] 276.00-277.00 sec   243 KBytes  1.99 Mbits/sec   33   1.41 KBytes
[  5] 277.00-278.00 sec   430 KBytes  3.52 Mbits/sec   49   19.8 KBytes
[  5] 278.00-279.00 sec   304 KBytes  2.49 Mbits/sec   45   15.6 KBytes
[  5] 279.00-280.00 sec   365 KBytes  2.99 Mbits/sec   40   19.8 KBytes
[  5] 280.00-281.00 sec   243 KBytes  1.99 Mbits/sec   38   15.6 KBytes
[  5] 281.00-282.00 sec   426 KBytes  3.49 Mbits/sec   54   17.0 KBytes
[  5] 282.00-283.00 sec   243 KBytes  1.99 Mbits/sec   33   14.1 KBytes
[  5] 283.00-284.00 sec   245 KBytes  2.00 Mbits/sec   22   1.41 KBytes
[  5] 284.00-285.00 sec   426 KBytes  3.49 Mbits/sec   43   15.6 KBytes
[  5] 285.00-286.00 sec   365 KBytes  2.99 Mbits/sec   46   14.1 KBytes
[  5] 286.00-287.00 sec   304 KBytes  2.49 Mbits/sec   48   14.1 KBytes
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-287.03 sec  29.3 GBytes   877 Mbits/sec  922             sender
[  5]   0.00-287.03 sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated

During ftp transfers I did notice that transfer speeds would increase after some time (maybe 10 - 20 minutes), but until it does so, speeds are often around 500 KB/s.

Any ideas would be really welcome.

 

[Samuel Tai]

By any chance, would you happen to have a Realtek NIC?

 

[codelett]

Thanks for your response.

No, it is a Broadcom 332i integrated in my HP MicroServer Gen8.
It ran fine for years. :/

The system used as iperf3 client has got a Intel chip.

 

[mrpasc]

next guess: did you „play“ with fan speed settings of your Micro gen8? Or did you replace stock fans with any one of those „silence“ fans aka Noctua and co?
The Broadcom nic chip needs a really decent airflow to work properly, and so the the fans should ramp up after a while of heavy usage. If not something may be broken at your gen8 cooling.

 

[codelett]

No, hardware is pretty much stock except for the CPU and RAM.

I ran a test with bios set to "maximum cooling" and it still started to drop packets after some time.

Furthermore I successfully tested RAM with memtest.

Right now I'm about to test with a linux live cd to see if it's an issue with TrueNAS.

 

[codelett]

My system shows the same behaviour when tested with iperf3 on some linux distro called AntiX.

I've ordered a pci card with an Intel I225 NIC for further testing.

 

[codelett]

Soooo.. unfortunately the new network card shows the same exact issue :(

Further ideas are welcome.

edit: When I change network settings ("Disable hardware offloading" in this case), transfers are fast again until the same issue happens again.

 

[Volts]

When I change network settings ("Disable hardware offloading" in this case),

Do you have plugins, jails, or any other bridged interfaces?

 

[codelett]

Yes. Jails and VMs.

Enabling/disabling "Disable hardware offloading" doesn't change anything about this issue btw.

 

[Samuel Tai]

Have you tried testing your RAM? This smells like you may have a bank that's cutting out or overheating under load, and NIC DMA suffers when that happens.

Alternatively, you can try setting a sysctl tunable under System->Tunables for vfs.zfs.arc.meta_min, with value 536870912, which sets a minimum memory allocation for ZFS metadata of 512 MB, which is the correct floor for your system RAM size of 16 GB.

 

[Volts]

How do you restore it to full speed? Does the performance recover after "a while" of sitting idle? If you reboot, does it work again immediately?

Changing offload may renegotiate the link or even reinit the card. I wonder if that's why it helped.

FWIW it doesn't ever make sense for offload to be enabled when a NIC is a member of a bridge interface, so that should be disabled.

When plugins/jails/VMs are in use, TrueNAS does something a bit hinky with bridges. The performance slowdown doesn't seem like the same issue, but it still is worth changing that:

https://www.truenas.com/community/t...access-jail-from-my-network.99513/post-686408
https://www.truenas.com/community/threads/vlans-and-bridge-something-is-broken.100798/post-694321
https://www.truenas.com/community/threads/vlans-and-bridge-something-is-broken.100798/post-694264
https://www.truenas.com/community/threads/jail-networking-not-working.100430/post-691911

 

[codelett]

Does the performance recover after "a while" of sitting idle?

Yes, see first post. It even recovered under load.

If you reboot, does it work again immediately?

Yes

Changing offload may renegotiate the link or even reinit the card. I wonder if that's why it helped.

Same here.

I already swapped RAM banks and the system works as before.

So I don't think it's about temperatures or RAM?
Hardware offloading is disabled as well.

Are there any stats I could pull from the system? Any caches that could fill up so packets are dropped?

edit: ifconfig igc0 down && ifconfig igc0 up is enough to recover the performance.

 

[Volts]

I think iPerf rules out most cache-related issues.
Look at netstat -id for interface errors and drops.
I assume there's nothing in dmesg when this starts/ends?

When this happens, does a local iPerf still go fast? (iperf3 -s in one window, iperf3 -c 127.0.0.1 in another.)
When this happens, does iPerf from a jail still go fast?

Does this still happen without the bridge (and jails and VMs) active on the interface?
Disable autostart for the jails and VMs, reboot.
Verify that bridge0 wasn't created.
Try the iPerf test again with the bare interface.

It would also be interesting to test connected back-to-back with the other device to rule out intermediate networking gear.

 

[codelett]

I think iPerf rules out most cache-related issues.
Look at netstat -id for interface errors and drops.
I assume there's nothing in dmesg when this starts/ends?

No, neither netstat -id nor dmesg show any signs of problems.
netstat doesn't even show dropped packets for the real interfaces.
The virtual ones have some but those do not increase during times of low transfer speeds.

When this happens, does a local iPerf still go fast? (iperf3 -s in one window, iperf3 -c 127.0.0.1 in another.)
When this happens, does iPerf from a jail still go fast?

I do not see any difference.
Local iperf3 is at about 50 GBit/s.
Jail (client) to local is at about 5 GBit/s.
Regardless of the issue we're currently looking at happening.

Does this still happen without the bridge (and jails and VMs) active on the interface?
Disable autostart for the jails and VMs, reboot.
Verify that bridge0 wasn't created.
Try the iPerf test again with the bare interface.

It would also be interesting to test connected back-to-back with the other device to rule out intermediate networking gear.

Will try this next.

edit: btw If I try start the iperf3 server in a jail and the client on my desktop pc the test fails after some time with the error "bad file descriptor".
If the desktop pc is running the server and the jail is the client, the issue does not appear at all.

 

[codelett]

I disabled autostart for all Jails and VMs and rebooted.
No bridge interfaces had been created.
But unfortunately the issue persisted.

Before that I moved all jails and VMs to a separate ethernet interface, but unsurprisingly that didn't help either.

Well, even though it doesn't make sense to me right now, because the transfer from the pc to the jail wasn't an issue,
I just removed a networking device in between as you suggested aaaaand.. the transfer speed did not dip down since then.
I'm pretty sure this was one of the first things I tried when the issue came up initially. :/

However, fingers crossed it stays this way.
Thanks for your help :) I really appreciate it and do not take it for granted.