SOLVED Is the GUI update server down/having problems today?

[VictorR]

[moved this from General Questions and Help to more appropriate sub-forum]

Trying do an update of FreeNAS-11.0-U4 (54848d13b) to latest 11.1 stable via GUI, and it just times out.
I can ping update.ixsystems.com from Shell within GUI.
But, update.ixsystems.com/FreeNAS returns "ping: cannot resolve update.ixsystems.com/FreeNAS: Unknown host"

Here's what I get when using the GUI updater:

Code:

Update server could not be reached

<urlopen error timed out>

Traceback
Traceback (most recent call last):
File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open
encode_chunked=req.has_header('Transfer-encoding'))
File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
self.send(msg)
File "/usr/local/lib/python3.6/http/client.py", line 964, in send
self.connect()
File "/usr/local/lib/python3.6/http/client.py", line 936, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/local/lib/python3.6/socket.py", line 722, in create_connection
raise err
File "/usr/local/lib/python3.6/socket.py", line 713, in create_connection
sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./freenasUI/system/views.py", line 1424, in update_check
train=updateobj.get_train(),
File "./freenasUI/system/models.py", line 625, in get_train
trains = conf.AvailableTrains() or []
File "/usr/local/lib/freenasOS/Configuration.py", line 1064, in AvailableTrains
fileref = self.TryGetNetworkFile(file=TRAIN_FILE, reason="FetchTrains")
File "/usr/local/lib/freenasOS/Configuration.py", line 787, in TryGetNetworkFile
raise url_exc
File "/usr/local/lib/freenasOS/Configuration.py", line 761, in TryGetNetworkFile
furl = opener.open(req, timeout=30)
File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
response = self._open(req, data)
File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
'_open', req)
File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
result = func(*args)
File "/usr/local/lib/python3.6/urllib/request.py", line 1346, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error timed out>

 

[Redcoat]

I have the update server open right now at http://update.ixsystems.com/FreeNAS/

 

[VictorR]

Thanks for the reply. There got to be some DNS problems happening on this end. Earlier, i could at least access the update directories via web browser. Now, I can't.

 

[jgreco]

But, update.ixsystems.com/FreeNAS returns "ping: cannot resolve update.ixsystems.com/FreeNAS: Unknown host"

I would imagine so. There is no top level domain called "com/FreeNAS". You cannot ping the latter half of a URL. You need to try to ping "update.ixsystems.com". Ping only understands hostnames and correctly reports what you asked it to ping as an unknown host.

 

[VictorR]

Hi jgreco, I can ping update.ixsystems.com from shell within GUI
But, "Check Now" connecting to http://update.ixsystems.com/FreeNAS times out after one minute with "Update server could not be reached <urlopen error timed out>"

As it turns out, there are some strange DNS things going on here. Cannot access websites like ATT.com or weather.com via browser on Mac desktops via the same connection, even though I can ping them via console

 

[jgreco]

That doesn't sound like "strange DNS things" unless you're getting broken answers, such as all of them resolving to a captive portal. Do they all resolve to the same address?

If you're being redirected to a captive portal, many of them break in today's HTTPS-heavy environment, where they used to assume they could just scam out redirects by intercepting :80.

Otherwise, there are many possibilities, including PMTUD type fun. You may be able to debug some of this by trying to telnet to <website> 80 and seeing if it connects, and if so, whether or not it'll serve up pages that way.

 

[VictorR]

That doesn't sound like "strange DNS things" unless you're getting broken answers, such as all of them resolving to a captive portal. Do they all resolve to the same address?

Trying to browse ATT.com, after 3 minutes (or so), it simply says "Failed to open page"...even though, I can successfully ping it via shell.

I contacted Frontier Networks support late last night. They cannot find the account, even though I can log on to their website and it acknowledges the account exists. But, does not furnish a PIN number for this account. So, they can't do anything until this is cleared up with their accounts department.

Otherwise, there are many possibilities, including PMTUD type fun. You may be able to debug some of this by trying to telnet to <website> 80 and seeing if it connects, and if so, whether or not it'll serve up pages that way.

Here's the output from a Telnet SSH connection:

Code:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for update.ixsystems.com has changed, and the key for the corresponding IP address 104.225.10.13
is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@	WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!	 @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed. The fingerprint for the ECDSA key sent by the remote host is
SHA256:q5RE2fTrr0GHRA03SefvyAp/PM/OZHBaOuWr7wtecKU.
Please contact your system administrator.
Add correct host key in /Users/alledge/.ssh/known_hosts to get rid of this message.

Offending ECDSA key in /Users/alledge/.ssh/known_hosts:1
ECDSA host key for update.ixsystems.com has changed and you have requested strict checking.
Host key verification failed.

[Process completed]

Thanks for the help on this. I know it's something obvious, I'm just not seeing it. And, this seems to happen on this system (or connection) every 3-4 months.

 

[jgreco]

Well that's an ssh connection. What I meant was to use telnet, and to test web retrieval. Many of the protocols on the Internet, especially the older ones, are things that you can actually chat with, because this allowed for interoperable designs and easy testing. So for example, you could check out www.freenas.org:

Code:

% telnet www.freenas.org 80
Trying 198.20.86.3...
Connected to edge.secdn.net.
Escape character is '^]'.
GET http://www.freenas.org/ HTTP/1.0

HTTP/1.1 200 OK
Server: nginx/1.10.1
Date: Sat, 27 Jan 2018 10:15:10 GMT
Content-Type: text/html; charset=UTF-8
Link: <http://www.freenas.org/wp-json/>; rel="https://api.w.org/"
Link: <http://www.freenas.org/>; rel=shortlink
X-Powered-By: ScaleEngine/0.8
Cache-Control: max-age=300
Vary: Accept-Encoding
Age: 366
X-SE-Grace: healthy-fetch
X-SE-Cache: Hit
X-SE-Hits: 1
X-SE-Server: 198.20.86.3
Accept-Ranges: bytes
Content-Length: 78775
Connection: close

<!DOCTYPE html>
<!--[if IE 6]>
<html id="ie6" lang="en-US" prefix="og: http://ogp.me/ns#">
<![endif]-->
<!--[if IE 7]>
<html id="ie7" lang="en-US" prefix="og: http://ogp.me/ns#">
<![endif]-->
<!--[if IE 8]>
<html id="ie8" lang="en-US" prefix="og: http://ogp.me/ns#">
etc etc

So anyways the trick here is to use telnet to connect to port 80. You then issue the command "GET http://www.freenas.org/ HTTP/1.0" and press return twice. The stuff returned is the HTML for www.freenas.org. Two things need to happen. One is that this has to complete, quickly and rapidly, and the other is that it has to appear to be the actual content for FreeNAS, rather than something talking about your ISP. I

 

[VictorR]

Well that's an ssh connection. What I meant was to use telnet, and to test web retrieval.

Sorry for the delay in response, hadn't been back to this office in a while.
Telnet returns:

 

[VictorR]

Spent over an hour on the phone with Frontier tech support. They can't access our router remotely. And, running traceroute, I can see things timing out as it leaves Frontier to the local Level-3 backbone.

We have an appointment for a tech to come out Monday. So, he can say "Yep, that's a problem"

 

[VictorR]

Well, it turned out to be a Frontier problem. Our static IP address number was never fully transferred over(provisioned) from the Verizon purchase. So, a lot of our traffic was getting dropped at their main trunk in los Angeles.
This has been a problem for 2 years, or more. We've had technicians out here at least 5 times. It would seem to go away for short while, then return.

 

[jgreco]

Yep, that's a problem.

(I'm so naughty)

 

[VictorR]

The really crazy thing is that we never had a period of 100% failure. It would work to some domains, and not others. And, there was no pattern to what got dropped