Slow download of TrueNAS Scale ISO

[BhSimon]

I have been struggling to download the TrueNAS Scale 22.12.3.3 ISO for days. The download speed is only 32KB/s and the connection is regularly dropped. I am connected via ethernet to a fibre router.

This is not related to my internet connection speed. By comparison, TrueNAS CORE ISO will download in a couple of minutes, at MB/s, on the same machine.

Is this a geographical issue? I am in Sydney, Australia. Any other occupants of the great southern land affected by this?

I did a bit of searching and noted other people mentioning this problem in relation to nightly builds and some updates, but it did seem to be isolated. I also looked for a torrent file, but couldn't find one.

 

[HoneyBadger]

Hey @BhSimon

I'll send you a DM with some requests about network traces, maybe we can figure out if something's failing along the line.

 

[xtj7]

I have the same problem, I get about 13 kb/s on an 800mbit fiber connection, the download is going to take about 2 days. I am located in Malaysia.

Speedtest to a server in Malaysia: 796 Mbps down
Speedtest to a server in Germany: 158 Mbps down
Speedtest to a server in NY, USA: 249 Mbps down
Speedtest to a server in CA, USA: 178 Mbps down

There appears to be no generic peering issue. Would be happy to perform further test to get behind this.

 

[xtj7]

As this might help others: I have just enabled NordVPN on my computer (Germany, but probably any other country outside your geographic area would work?) and I'm downloading closer to 10 MB/s.

 

[HoneyBadger]

Hey @xtj7

What kind of download speeds do you get from this CDN in comparison to the other one?

https://download.truenas.com/TrueNAS-SCALE-Bluefin/22.12.3.3/TrueNAS-SCALE-22.12.3.3.iso

 

[MrGuvernment]

From Calgary,AB in Canada, good for me direct from website.

Hey @xtj7

What kind of download speeds do you get from this CDN in comparison to the other one?

https://download.truenas.com/TrueNAS-SCALE-Bluefin/22.12.3.3/TrueNAS-SCALE-22.12.3.3.iso

From yours:

 

[xtj7]

Hey @xtj7

What kind of download speeds do you get from this CDN in comparison to the other one?

https://download.truenas.com/TrueNAS-SCALE-Bluefin/22.12.3.3/TrueNAS-SCALE-22.12.3.3.iso

About 5.5 MB/s

 

[Dominick_Storj]

Just tested form New Zealand which we do realize is quite far from Sydney. We got 13MB/s with spikes to 17MB/s. It would be most helpful to get a traceroute @BhSimon. Please advise if you need any assistance with this.

 

[CH4Pz]

Im in South Australia, experiencing the same issue, speeds less than 100kb/s, Speedtest showing over 50mbps, last few attempts has the download stopping at 18mb, saying completed?!

 

[CH4Pz]

Connected to nordvpn instantly went to 6MB/s (50ish mb/s), fyi

 

[Dominick_Storj]

Appreciate the alert @CH4Pz, could you provide a traceroute to download.truenas.com with your VPN off?

 

[Dominick_Storj]

If this cant be replicate please report. We are aware of a shot term issue with Hurricane Electric yesterday which could have been the cause of this. Signing on the VPN would change the route.

 

[CH4Pz]

Appreciate the alert @CH4Pz, could you provide a traceroute to download.truenas.com with your VPN off?

Didnt really know what a traceroute was, so had to look it up, is this what you were after? starts ok, then drops off with bunch of timeouts

 

[MrGuvernment]

Thats some rough routing, note, trace route is not always definitive as some cores they will drop ping, icmp et cetera packets, but that showing some and * on others, tells me there is prob some serious issues going on between you and the download server.

 

[CH4Pz]

Thats some rough routing, note, trace route is not always definitive as some cores they will drop ping, icmp et cetera packets, but that showing some and * on others, tells me there is prob some serious issues going on between you and the download server.

Thats what i was thinking, yet from the vpn, even on australian server, downloads normally?! My speedtest OFF the vpn looks normal too?!

 

[Dominick_Storj]

Thank you, the origin for this download and the speedtest are different places. Speedtests are often very few hops (close). To improve performance in Australia we are in the process of of scaling out additional capacity in region.

Short term we enabled additional ISPs with more routes into region. Please advise if still slow.

 

[MrGuvernment]

Ya, ISP's often give priority to speed test sites to show that they are FASTTTTTTTTTTTTT, why I like to test against other companies vs the defaults most give.

Similar to the VPN, the routing it takes to reach the provider can differ also.

 

[DD4711]

Current speed from Germany 100-120KB/s and the connection is also regularly dropped.

 

[thoriig]

Greetings,

yep germany to download.truenas.com is all time below 80 kb/sec.
adding my traces, only the UDP ones are slow af.
but its not only slow, it interupts and updates getting invalid because of checksum mismatch.

Hope it helps to get further with the investigation. if you need any additional information just drop a pm.

admin@hypercube[~]$ sudo traceroute -I download.truenas.com
traceroute to download.truenas.com (136.0.77.2), 30 hops max, 60 byte packets
1 192.168.1.2 (192.168.1.2) 1.000 ms 1.027 ms 1.024 ms
2 * (*.*.*.*) 4.151 ms 4.410 ms 4.504 ms
3 f-ed14-i.F.DE.NET.DTAG.DE (217.5.70.46) 16.600 ms 16.652 ms 16.705 ms
4 80.157.201.198 (80.157.201.198) 7.807 ms 8.067 ms 8.211 ms
5 be3186.ccr41.fra03.atlas.cogentco.com (130.117.0.1) 7.607 ms 7.668 ms 8.157 ms
6 be2813.ccr41.ams03.atlas.cogentco.com (130.117.0.121) 63.833 ms 62.041 ms 62.079 ms
7 be2795.rcr21.b047839-0.ams03.atlas.cogentco.com (154.54.58.22) 14.885 ms 14.813 ms 14.891 ms
8 * * fdcservers.demarc.cogentco.com (149.11.203.10) 19.294 ms
9 136.0.77.2 (136.0.77.2) 13.461 ms 13.329 ms 13.272 ms

admin@hypercube[~]$ sudo traceroute -T download.truenas.com
traceroute to download.truenas.com (136.0.77.2), 30 hops max, 60 byte packets
1 192.168.1.2 (192.168.1.2) 3.221 ms 7.860 ms 3.525 ms
2 * (*.*.*.*) 6.365 ms 6.426 ms 6.493 ms
3 f-ed14-i.F.DE.NET.DTAG.DE (217.5.109.114) 10.906 ms f-ed14-i.F.DE.NET.DTAG.DE (217.0.201.118) 9.856 ms f-ed14-i.F.DE.NET.DTAG.DE (217.5.70.154) 8.181 ms
4 80.157.201.198 (80.157.201.198) 11.637 ms 11.968 ms 12.842 ms
5 be3186.ccr41.fra03.atlas.cogentco.com (130.117.0.1) 13.406 ms be3187.ccr42.fra03.atlas.cogentco.com (130.117.1.118) 10.850 ms 12.959 ms
6 be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141) 15.563 ms 16.164 ms be2813.ccr41.ams03.atlas.cogentco.com (130.117.0.121) 17.868 ms
7 be2795.rcr21.b047839-0.ams03.atlas.cogentco.com (154.54.58.22) 15.765 ms 17.393 ms be2796.rcr21.b047839-0.ams03.atlas.cogentco.com (154.54.58.126) 14.536 ms
8 * * *
9 136.0.77.2 (136.0.77.2) 14.339 ms 14.809 ms *

admin@hypercube[~]$ traceroute download.truenas.com
traceroute to download.truenas.com (136.0.77.2), 30 hops max, 60 byte packets
1 192.168.1.2 (192.168.1.2) 4.559 ms 4.571 ms 4.549 ms
2 * (*.*.*.*) 5.092 ms 6.376 ms 8.035 ms
3 f-ed14-i.F.DE.NET.DTAG.DE (217.0.203.26) 8.893 ms f-ed14-i.F.DE.NET.DTAG.DE (217.0.201.118) 9.570 ms f-ed14-i.F.DE.NET.DTAG.DE (217.5.67.246) 10.438 ms
4 80.157.201.198 (80.157.201.198) 10.865 ms 10.927 ms 10.777 ms
5 be3186.ccr41.fra03.atlas.cogentco.com (130.117.0.1) 11.679 ms 10.940 ms 10.988 ms
6 be2814.ccr42.ams03.atlas.cogentco.com (130.117.0.141) 16.314 ms 12.725 ms be2813.ccr41.ams03.atlas.cogentco.com (130.117.0.121) 15.238 ms
7 be2796.rcr21.b047839-0.ams03.atlas.cogentco.com (154.54.58.126) 16.093 ms 15.369 ms 14.544 ms
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *

 

[Dominick_Storj]

Appreciate the report and the detail. Escalating this internally now.