WebGUI Access lost after update to 11-U3

[AzAv8r]

After running FreeNAS for a few years, last week it reported an error on the boot drive (a 16GB USB flash). Over the weekend I pulled the server out of the corner, installed two new 32 GB USB drives, a keyboard and monitor, and upgraded from 9.10-U6 to 11-U2. No other changes were made. All was well. Tuesday morning I got the notification an update was available, so I logged into the WebGUI and started the update to 11-U3.

Alas, I can no longer access the WebGUI. When I attempt to login I get the "An error has occurred" web page from the Nginx directory. The server is up and my NextCloud server is running in it's jail.

I reconnected the keyboard/monitor and found the following message in the nginx error log file:

Code:

[error] 2524#100653: *1 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream client: 192.168.88.30, server: localhost, request: "GET /currentsetting.htm HTTP/1.1" upstream: "fastcgi://127.0.0.1:9042", host: "192.168.88.5" 

Subsequent access attempts result in the "currentsetting.htm" page of the request disappearing, and the server host IP being replaced with the hostname. But the outcome is otherwise the same. And the outcome is independent of the client OS.

Any suggestions?

(Server is a TS140 with an E3, 24 GB ECC. New boot disks are SanDisk 32GB Cruzer Fit)

 

[dirkme]

Thanks for posting this, I was wondering if I can just run the update and was looking for people with trouble. Sadly, I am new to FreeNas and can't help much.

A question though, do you use the 10 version of nextcloud from the plugin or did you build you own jail witch apache and MySQL?

 

[AzAv8r]

SOLVED!

Decided to select the previous version from the boot screen. Much to my surprise I discovered I was running 9.10-U6 again. Checked the update email I'd received from FreeNAS, and it was advertising an "update" to 9.10-U6, not 11.0-U3 (It has worked so seamlessly in the past I just trusted it - that will teach me!) I see that on the Update settings tab although the "Current Train" was set to 11.0 stable, the selection for the update was still at 9.10 Stable. I changed this, and subsequentially updated to 11.0-U3. And can still access the WebGUI. Perhaps a minor bug in the 11.0 installer, in that it leaves the Update train in an inconsistent state. Recommend you check the selected (Drop-Down) train on the Update GUI after upgrade.

Anyway, all is now good, and my painless, easy-to-manage fileserver is once again painless and easy-to-manage.

Regarding NextCloud - I use the available 10.0 plugin. (Ran Owncloud for quite a while, but I'm a bit skeptical about the long-term health of Owncloud so a few months ago I migrated to NextCloud.)
NextCloud will update itself within the 10 series - log in to the NextCloud webgui, select Admin, and you'll be presented with update options, amongst other items. It apparently will not update to 11.0 without first separately updating the PHP version, something I'm not yet ready to take on.

 

[dirkme]

Same here, I ran some updates, i think I went to 11.0 and had trouble afterwards, so right now I am back to 10.0 and leave it there for now.

I might do a self/fresh install in a new jail, just for testing.

If/when I do it, I will post ,my findings and maybe some tutorial/steps how I did it.

 

[Mike Katz]

I experienced exactly the same situation. When I tried to login to the webgui via http I got Error 403 - Forbidden and no response at all when i tried from https.

I rebooted to 11.0u2 and everything works fine. Something is broken in 11.0U3.

 

[Redcoat]

Something is broken in 11.0U3.

I have two systems here that don't agree with you...

 

[Mike Katz]

And there are 2 people here who have a problem. There is probably some kind of issue that is not across the boards for all configurations.

 

[dirkme]

I ran the update yesterday and all is running fine.

 

[Redcoat]

Great. I just don't want anyone to read a thread with a blanket statement "Something is broken in 11.0U3" and assume it applies without qualification...

 

[gromaler]

yesterday I run update and after rebooting no WEBGUI and not in network :(
what I should to do then...

 

[Mike Katz]

Try rebooting and at the FreeNAS grub menu, select 11.0U2.

I have already filed a bug report on this.

 

[Mike Katz]

I am making a blanket statement. Something is broken in 11.0U3 in the upgrade process. Thought it might not be broken for everyone, something is broken none the less. There are now three reports in the last 3 days about this issue. In my opinion as a software developer (40 years experience) and software development manager (10 years experience) something is definitely not working correctly.

As I said before I have already files a bug reports on this issue.

My suggestion to everyone, at the risk of not being able to login to the WebGui anymore would be to NOT upgrade to 11.0U3 until the issue is found and corrected or a work around is published. Since some people are able to upgrade successfully and some people are not, attempting the upgrade before the issue has been resolved is a gamble.

I am not in the habit of gambling on my ability to maintain my servers.

 

[Inxsible]

I am making a blanket statement. Something is broken in 11.0U3 in the upgrade process. Thought it might not be broken for everyone, something is broken none the less. There are now three reports in the last 3 days about this issue. In my opinion as a software developer (40 years experience) and software development manager (10 years experience) something is definitely not working correctly.

As I said before I have already files a bug reports on this issue.

My suggestion to everyone, at the risk of not being able to login to the WebGui anymore would be to NOT upgrade to 11.0U3 until the issue is found and corrected or a work around is published. Since some people are able to upgrade successfully and some people are not, attempting the upgrade before the issue has been resolved is a gamble.

I am not in the habit of gambling on my ability to maintain my servers.

I hear you, BUT I upgraded 2 days ago from U2 to U3 and have had no problems at all. Zero, zilch, nada.

Unless you can provide a way to reproduce the problem there is not much that can be done to figure it out. You said you filed a bug report. Would you provide a link so we can check if you have listed a way to reproduce the problem? Have you put in the logs in that bug report?

When you only say "something is broken", it could be anything from fatal errors to PEBKAC. As a software developer, you should know this.

 

[shamo316]

im also have the same issue on chrome browser, when i inspect site this is the error.

edit: Seems like new UI works the old one does not just a blank screen

Code:

dojo.js?cache=569213313682ba56a9d411be67f2debb:8 EXPERIMENTAL: dojox.timing -- APIs subject to change without notice.
dojo.experimental @ dojo.js?cache=569213313682ba56a9d411be67f2debb:8

dojo.js?cache=569213313682ba56a9d411be67f2debb:8 EXPERIMENTAL: dojox.widget.Standby -- APIs subject to change without notice.
dojo.experimental @ dojo.js?cache=569213313682ba56a9d411be67f2debb:8

dojo.js?cache=569213313682ba56a9d411be67f2debb:8 EXPERIMENTAL: dojox.layout.ResizeHandle -- APIs subject to change without notice.
dojo.experimental @ dojo.js?cache=569213313682ba56a9d411be67f2debb:8

dojo.js?cache=569213313682ba56a9d411be67f2debb:8 Uncaught TypeError: Cannot read property 'decimalFormat' of undefined
	at _7 (NumberTextBox.js?569213313682ba56a9d411be67f2debb:4)
	at NumberTextBox.js?569213313682ba56a9d411be67f2debb:19
	at _ce (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)
	at _37 (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)
	at _37 (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)
	at dojo.js?cache=569213313682ba56a9d411be67f2debb:8
	at _38 (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)
	at _81 (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)
	at _f6 (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)
	at HTMLScriptElement._10b (dojo.js?cache=569213313682ba56a9d411be67f2debb:8)

 

[Mike Katz]

Just because it works for you doesn't mean that it works for everyone else. Broken does not necessarily mean non-functional. I am currently working with the developers via the bug reporting mechanism. I have provided them with my system dump, etc.

Broken does not necessarily mean non-functional.

While I understand that your upgrades worked perfectly, there are now 4 different instances in this thread where people are reporting the same thing.

I can reproduce the error every time by upgrading and downgrading. As the manager of several servers, if I saw someone say that there were some problems with an upgrade that would cause me to lose my ability to maintain my servers I would wait on performing the upgrade until the issue was solved or a work around published.

Though the upgrade works for you, it does not work for me and several others. I will grant you that FreeNAS 11.0-U3 is not broken completely (i.e. nonfunctional for all) but for me and the other people responding here it is broken to the point where the server cannot be managed via the GUI. In my opinion, an un-maintainable server is a broken server, even if other functionality of the server is working. Since managing the server is an important aspect of FreeNAS I will continue to say that something in the 11.0-U2 to 11.0-U3 upgrade is broken for some users.

If you would like to track the bug report it is here: https://bugs.freenas.org/issues/25809.

 

[Ericloewe]

there are now 4 different instances in this thread where people are reporting the same thing.

How many of them have scrubbed the boot pool? Run the verify install function (this one might be difficult without the GUI, but it is available from the Unix shell, IIRC)? Rolled back and tried the update again?

There might be something here (especially the installs on SSD), but there's a lot of noise. And this kind of thing is recurring, an unfortunate side effect of the unreliability of USB flash drives. This does make me think that there's some value in minimizing update sizes, as recent updates have been much larger than they used to be.

 

[Inxsible]

There might be something here (especially the installs on SSD), but there's a lot of noise. And this kind of thing is recurring, an unfortunate side effect of the unreliability of USB flash drives.

Did you mean "installs on USB" instead of "installs on SSD" ?
FWIW, I have a single 16GB USB drive which has my boot pool.

This does make me think that there's some value in minimizing update sizes, as recent updates have been much larger than they used to be.

How large have the updates been? Certainly not enough to fill out 8GB or 16GB drives.

 

[Ericloewe]

Did you mean "installs on USB" instead of "installs on SSD" ?

No, I trust SSDs (well, to some extent). I do not trust USB flash drives to be vaguely reliable, not after three years of data to the contrary.

How large have the updates been? Certainly not enough to fill out 8GB or 16GB drives.

Here's an example. Apologies for the image, but zfs list is harder to interpret due to how boot environments are created (they're clones of snapshots of the original environment, which has been edited over time).

9.10.2-U4 through -U6 are all 650+MB on my other server, which is still on the 9.10.2 train, whereas this one moved to the 11 train before 9.10.2-U4 was released.

(Note that the latest environment is reported with the total size of all the snapshots, not including the datasets cloned from them, so it's slightly lower than the total used space.)

For those who are curious, here's the zfs list output:

Code:

# zfs list -r -t all freenas-boot
NAME											  USED  AVAIL  REFER  MOUNTPOINT
freenas-boot									 6.12G  7.68G	31K  none
freenas-boot/ROOT								6.10G  7.68G	25K  none
freenas-boot/ROOT/11.0-RELEASE					 36K  7.68G   747M  /
freenas-boot/ROOT/11.0-U1						  86K  7.68G   747M  /
freenas-boot/ROOT/11.0-U2						11.6M  7.68G   751M  /
freenas-boot/ROOT/11.0-U3						6.05G  7.68G   740M  /
freenas-boot/ROOT/11.0-U3@2016-08-18-11:46:39	3.04M	  -   613M  -
freenas-boot/ROOT/11.0-U3@2016-09-27-20:37:59	6.83M	  -   619M  -
freenas-boot/ROOT/11.0-U3@2016-10-04-19:37:45	 151M	  -   626M  -
freenas-boot/ROOT/11.0-U3@2016-11-26-12:40:13	 154M	  -   629M  -
freenas-boot/ROOT/11.0-U3@2016-12-21-15:24:51	  44K	  -   630M  -
freenas-boot/ROOT/11.0-U3@2016-12-21-15:41:04	  38K	  -   630M  -
freenas-boot/ROOT/11.0-U3@2016-12-21-16:14:21	8.12M	  -   630M  -
freenas-boot/ROOT/11.0-U3@2017-02-07-01:31:05	 126M	  -   644M  -
freenas-boot/ROOT/11.0-U3@2017-03-07-22:34:45	 103M	  -   645M  -
freenas-boot/ROOT/11.0-U3@2017-04-21-17:02:35	 136M	  -   646M  -
freenas-boot/ROOT/11.0-U3@2017-05-04-19:33:24	 646M	  -   646M  -
freenas-boot/ROOT/11.0-U3@2017-07-04-17:04:52	 745M	  -   747M  -
freenas-boot/ROOT/11.0-U3@2017-07-23-00:11:05	 745M	  -   747M  -
freenas-boot/ROOT/11.0-U3@2017-09-06-16:33:18	 748M	  -   750M  -
freenas-boot/ROOT/9.10.1-U1					  4.84M  7.68G   626M  /
freenas-boot/ROOT/9.10.1-U2						38K  7.68G   629M  /
freenas-boot/ROOT/9.10.1-U4					  8.33M  7.68G   630M  /
freenas-boot/ROOT/9.10.1-U4@2016-12-21-18:19:32	  0	  -   630M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-21-18:22:15	  0	  -   630M  -
freenas-boot/ROOT/9.10.1-U4@2016-12-21-18:22:56	  0	  -   630M  -
freenas-boot/ROOT/9.10.2						   42K  7.68G   644M  /
freenas-boot/ROOT/9.10.2-U1						47K  7.68G   636M  /
freenas-boot/ROOT/9.10.2-U2					  9.07M  7.68G   646M  /
freenas-boot/ROOT/9.10.2-U3					  8.90M  7.68G   646M  /
freenas-boot/ROOT/Initial-Install				   1K  7.68G   613M  legacy
freenas-boot/ROOT/default						  36K  7.68G   619M  legacy
freenas-boot/grub								6.45M  7.68G  6.45M  legacy

 

[dirkme]

I just did a fresh install on 2 USB drives with U3 and it is all running fine.

 

[Inxsible]

I ran the update yesterday and all is running fine.

I just did a fresh install on 2 USB drives with U3 and it is all running fine.

So did you update or did a fresh install? and if the update worked, why do a fresh install?