Extremely slow pkg transfers inside jails after FreeNAS 11.3U5 to 12.0-U2.1

[JHBoricua]

I haven't been able to find anything in my searches so I'm forced to ask here. I just upgraded my FreeNAS server 11.3-U5 to TrueNAS Core 12.0-U2.1. That went without a hitch. Then I did a iocage upgrade on my plex jail which also completed.

After that I entered the jail console and I'm trying to do a pkg update/upgrade and I'm getting literally slower than good-old-days dial-up speeds downloading the packages. I'm talking anywhere from 8kB/s to 24kB/s at the most. I'm on a 300MB fiber internet connection.

Prior to doing the FreeNAS 11.3 to TrueNAS 12.0 upgrade on the server, I did a pkg update/upgrade inside my jail and it downloaded packages blazingly fast. So it seems to have been fine when I was on FreeNAS 11.3U5.

On TrueNAS, I've even tried different mirrors in the jail without a change in behavior. I also spun up a fresh jail and it's having the same issue, which seems to rule out something odd going on with the upgraded jail. As a matter of fact, because my existing jail is still trying to download, the new one won't even make any progress if I attempt to do a pkg update/upgrade.

Any thoughts on what could possibly be going on?

 

[HolyK]

@JHBoricua Long story short ... Go to "Network -> Interfaces" and edit the physical interface you're using for your Jails. Tick the "Disable hardware offloading" checkbox, confirm the warning, save settings and apply. Then try again your Jails.

 

[-fun-]

@HolyK I came across this (though I do not have the OPs issue currently) and have a followup question regarding the warning: What performance impact DOES this have on regular traffic?

 

[HolyK]

@-fun- That Warning refers mainly to CPU utilization. HW offloading allows NIC to calculate checksums by itself (within its own chip) so CPU does need to spend cycles on that. If you disable the HW offloading all of the checksums needs to be calculated by CPU (plus the extra steps to/from NIC).

So if you're not using VNET then disabling HW offload does nothing for network speed (because it will be already OK) but it adds some extra load on CPU.

But if you do use VNET then you're most probably facing Bug 235607 in FreeBSD 12 so it is recommended to disable HW offload. That will bypass the bug and make the network speed as it should be for a cost of (slightly?) higher CPU utilization.
How much exactly depends on how large/heavy your infra is. If you have just few Jails and the CPU is slacking most of the time you will not see any serious impact. If you have dozen of Jails and your network is heavily utilized and your CPU is constantly under pressure then you might have an issue...

Just example in my case:
With HW offload enabled: 4.19 Mbits/sec (and it completely fcks up the tcp segments, if you run tcpdump/wireshark there are CHKsum errors all over the place)
With HW offload disabled: 941 Mbits/sec

CPU utilization went up by ~7% while running iperf in both of the tests above. Anyway my infrais pretty slim and CPU is mostly doing nothing so i don't care much about few extra % as long as my network runs fast :]

 

[-fun-]

Great explanation, thank you!

 

[cegha04]

@JHBoricua Long story short ... Go to "Network -> Interfaces" and edit the physical interface you're using for your Jails. Tick the "Disable hardware offloading" checkbox, confirm the warning, save settings and apply. Then try again your Jails.

Thank you for this. You saved me. I updated from FreeNAS 11 to TrueNAS 12 over the weekend. Was downloading some stuff through SABNZBD and my downloads went from 110MB/s (on fiber) to 900KB/s. I was wondering what happened and finally found this thread. I use dual physical connections to a LAGG. Turned on the "Disable hardware offloading" setting on both physical connections and I'm back to full download speed albeit at a 8-11% CPU usage.

Looks like this FreeBSD bug has been opened for almost 3 years and it's still not fixed? Is there anyway the TrueNAS team can try to prioritize a fix on this with FreeBSD team? Kinda super annoying this wasn't a issue in 11.x but it is in 12.x.