TimeoutError: TrueNAS was unable to reach update servers.

[aphex3k]

Looks like I have the same issue as posted here: https://www.truenas.com/community/threads/tn-scale-update-timeout-train-is-not-visible.112550/ However, there have been no responses. I would like to provide some addition information beyond what is visible in the UI and what seems to be commonly referred to as checks when a similar issue is mentioned here in the forum.

I'm on 23.10-RC.1 from a fresh install and configuration.

I was able to resolve update.ixsystems.com successfully as well as ping the resolved IP 136.0.77.2.

Show : DNS

Code:

Linux truenas 6.1.50-production+truenas #2 SMP PREEMPT_DYNAMIC Sat Sep 16 01:23:34 UTC 2023 x86_64


        TrueNAS (c) 2009-2023, iXsystems, Inc.
        All rights reserved.
        TrueNAS code is released under the modified BSD license with some
        files copyrighted by (c) iXsystems, Inc.


        For more information, documentation, help or support, go here:
        http://truenas.com


Welcome to TrueNAS
Last login: Fri Oct 20 09:11:03 PDT 2023 on pts/4


Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.


admin@truenas[~]$ dig update.ixsystems.com


; <<>> DiG 9.18.12-1-Debian <<>> update.ixsystems.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60311
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1


;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;update.ixsystems.com.          IN      A


;; ANSWER SECTION:
update.ixsystems.com.   85947   IN      CNAME   link.storjshare.io.
link.storjshare.io.     149     IN      A       136.0.77.2
admin@truenas[~]$ ping 136.0.77.2
PING 136.0.77.2 (136.0.77.2) 56(84) bytes of data.
64 bytes from 136.0.77.2: icmp_seq=1 ttl=54 time=109 ms
64 bytes from 136.0.77.2: icmp_seq=2 ttl=54 time=27.7 ms
64 bytes from 136.0.77.2: icmp_seq=3 ttl=54 time=41.3 ms
64 bytes from 136.0.77.2: icmp_seq=4 ttl=54 time=18.1 ms

 

[aphex3k]

I believe I found the issue. It seems to lie with IPv6 and resolving the update domain to an IPv6 address.

Code:

admin@truenas[~]$ wget https://update.ixsystems.com/scale/trains.json
--2023-10-20 15:15:07--  https://update.ixsystems.com/scale/trains.json
Resolving update.ixsystems.com (update.ixsystems.com)... 2a10:c640:dead::5, 136.0.77.2
Connecting to update.ixsystems.com (update.ixsystems.com)|2a10:c640:dead::5|:443... failed: Connection timed out.
Connecting to update.ixsystems.com (update.ixsystems.com)|136.0.77.2|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1102 (1.1K) [application/json]
Saving to: ‘trains.json’

trains.json                               100%[==================================================================================>]   1.08K  --.-KB/s    in 0s     

2023-10-20 15:17:19 (5.74 MB/s) - ‘trains.json’ saved [1102/1102]

resolving the domain update.ixsystems.com to IPv6 yields 2a10:c640:dead::5.

Code:

admin@truenas[~]$ dig AAAA update.ixsystems.com


; <<>> DiG 9.18.12-1-Debian <<>> AAAA update.ixsystems.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36273
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1


;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;update.ixsystems.com.          IN      AAAA


;; ANSWER SECTION:
update.ixsystems.com.   14489   IN      CNAME   link.storjshare.io.
link.storjshare.io.     246     IN      AAAA    2a10:c640:dead::5


;; Query time: 0 msec
;; SERVER: 192.168.1.155#53(192.168.1.155) (UDP)
;; WHEN: Fri Oct 20 15:19:39 PDT 2023
;; MSG SIZE  rcvd: 109

 

[aphex3k]

Disabling IPv6 in the TrueNAS network settings resolved the issue but that can not be considered a solution.

 

[satrajitdas]

Looks like I have the same issue.

From the Linux shell I can do a nslookup for "update-master.ixsystems.com" and it returns IPv4 "136.0.77.2" and IPv6 "2a10:c640:dead::5".

However, ping to "update-master.ixsystems.com" does not reach the server. Traceroute to "update-master.ixsystems.com" does not complete after step 30.

Trying to update through the TrueNAS GUI errors out saying the server is not reachable.

I can ping/traceroute other addresses (google.com, cloudflare.com) from the Linux shell.

The only way I could make it work was by clearing out the IPv6 gateway (a FE80 address), so that only a IPv4 gateway remained. Then the GUI update check worked.

 

[Dominick_Storj]

Could we get locations for everyone having these issues?

 

[satrajitdas]

I am in Arizona, US.

 

[aphex3k]

Could we get locations for everyone having these issues?

Southern California

 

[Dominick_Storj]

Fixes for the IPv6 routing issues were deployed last night. Please advise if anyone is still having issues.

 

[satrajitdas]

I am still facing the same issues.

 

[satrajitdas]

Also, for reference, this is my nslookup from the TrueNAS shell:

update-master.ixsystems.com canonical name = link.storjshare.io.
Name: link.storjshare.io
Address: 185.244.226.4
Name: link.storjshare.io
Address: 2a10:c640:0:1::4

 

[aphex3k]

I can confirm, that the IPv6 resolves to a different address now. The different address has the same issue though with no reachable http(s) response on port 443.

Code:

admin@truenas[~]$ wget https://update.ixsystems.com/scale/trains.json
--2023-10-26 08:22:55--  https://update.ixsystems.com/scale/trains.json
Resolving update.ixsystems.com (update.ixsystems.com)... 2a10:c640:0:1::4, 185.244.226.4
Connecting to update.ixsystems.com (update.ixsystems.com)|2a10:c640:0:1::4|:443... failed: Connection timed out.

Also, the automatic updater in the UI does not seem to fall back to IPv4 and instead fails before trying.

 

[aphex3k]

This is an IPv6 traceroute on COX

Code:

➜  ~ traceroute6 update.ixsystems.com
traceroute to update.ixsystems.com (2a10:c640:0:1::4), 30 hops max, 80 byte packets
 1  <redacted>::1 (<redacted>::1)  0.250 ms  0.184 ms  0.165 ms
 2  * * *
 3  2001:578:200:7:f:0:6:801c (2001:578:200:7:f:0:6:801c)  12.140 ms  12.154 ms *
 4  2001:578:200:4:2000::2e (2001:578:200:4:2000::2e)  10.723 ms  11.930 ms  15.624 ms
 5  2001:578:1:0:172:17:248:216 (2001:578:1:0:172:17:248:216)  65.670 ms * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

For me it seems to die somewhere in a private network: 2001:578:1:0:172:17:248:216 => 172.17.248.216

 

[flashdrive]

IPv6 hassle...

IPv4 was - looking from outside - somewhat "easier" to handle...

 

[aphex3k]

Currently, IPv6 seems deactivated at least for my ISP

Code:

➜  ~ traceroute6 update.ixsystems.com
update.ixsystems.com: Name or service not known

 

[satrajitdas]

Same here. Only a IPv4 address comes back in nslookup and the update check completes successfully.

 

[jedwest]

I started facing the same issue. Any updates?

 

[jedwest]

I started facing the same issue. Any updates?

I disabled IPv6 and it did not help. After re-enabling IPv6 I now get Truenas update is disabled. Seem this issue is getting worse, not better. At least for me. Any ideas?

2 errors happened
1. Timeouterror
2. [EFAULT] Network activity "Update" is disabled: Automatic update check failed. Please check system network settings.

Trying to go through 23.10.1.3 to 23.10.2

 

[framerate]

I'm seeing this today as well. Monitoring this thread

 

[jedwest]

I'm seeing this today as well. Monitoring this thread

I also manually re-enabled "outbound Network" under global settings to "allow all" a couple of days again and it didn't work, but today I set "outbound network" to "Allow update" and it worked. Go figure???

 

[Azure_Kid]

I'm currently getting this issue. I've taken the suggestions around IPV6 and it has changed nothing for me. I have no ides what to do.