Unable to access WebUI after failed update

[Lorem-Ipsum]

I attempted to install the FreeNAS 9.10.2 update earlier and after waiting for over an hour stuck at step 1/5 decided something must be wrong and cancelled it.

I was unable to remove the FreeNAS 9.10.2 boot environment in the GUI so rebooted and chose 9.10.1u4 from Grub.

It booted up OK and I was able to access 2 of my 3 jails. I was initially able to browse smb shares but this stopped working a few minutes later.
However I was and am completely unable to access the WebUI.

Troubleshooting:

• I am unable to browse the FreeNAS WebUI at all. It partially loads but I can't log in and the page times out in multiple browsers.
• I am able to ssh in and memory/cpu usage etc.. appears normal.
• I have run a scrub on my boot pool which completed successfully.
• No errors appear on the console and iLo health checks appear good.

I'm a bit unsure how to troubleshoot this. At first I thought it might me either the USB drive or SDcard in the boot pool but a scrub of freenas-boot completed successfully. Also bootup time is normal.

• I have plenty of space available on freenas-boot (over 70% free).
• I could see that in zfs list and bdadm list there was a dataset and boot environment for 9.10.2.
• I was able to remove this with beadm destroy 9.10.2.
• The webui is still not responsive.
• I rebooted the FreeNAS server again
• The webUI still does not load.
• SMB shares appear online again and I am able to access the same 2/3 jails.
• Am able to ping and resolve DNS from FreeNAS.
• ifconfig shows that networking appears to be correct.
• /var/db/system/syslog-bf937a0da5564cb596eff5f8aba08500/log/nginx-error.log does not show any recent errors.

I have also found the following:

Code:

> ls -al /
total 84
-rw-r--r--   1 root		 wheel	 0 Sep 11 12:40 ??SELECT???plugin_name,???plugin_jail,???plugin_path,???plugin_enabled??FROM???plugins_plugins?? WHERE ( plugin_jail = 'couchpotato_1' AND plugin_name = 'couchpotato' )
-rw-r--r--   1 root		 wheel	 0 Sep 11 14:31 ??SELECT???plugin_name,???plugin_jail,???plugin_path,???plugin_enabled??FROM???plugins_plugins?? WHERE ( plugin_jail = 'emby_1' AND plugin_name = 'emby' )
-rw-r--r--   1 root		 wheel	 0 Sep 25 21:13 ??SELECT???plugin_name,???plugin_jail,???plugin_path,???plugin_enabled??FROM???plugins_plugins?? WHERE ( plugin_jail = 'transmission_1' AND plugin_name = 'transmission' )
drwxr-xr-x  20 root		 wheel	31 Dec 20 22:30 ./
drwxr-xr-x  20 root		 wheel	31 Dec 20 22:30 ../
-rw-r--r--   1 root		 wheel   898 Nov 10 03:48 .cshrc
-rw-r--r--   1 root		 wheel   188 Nov 10 03:48 .profile
-rw-------   1 root		 wheel  1024 Mar 17  2016 .rnd
-r--r--r--   1 root		 wheel  6142 Nov 10 03:48 COPYRIGHT
-rw-r--r--   1 root		 wheel	 0 Nov 20 16:29 SELECT tun_var, tun_value FROM system_tunable WHERE tun_enabled = 1 AND tun_type = 'sysctl' ORDER BY id
drwxr-xr-x   2 root		 wheel	47 Nov 10 03:48 bin/
drwxr-xr-x  10 root		 wheel	57 Nov 10 03:56 boot/
drwxr-xr-x   3 root		 wheel	 3 Mar 25  2016 compat/
drwxr-xr-x   4 root		 wheel	 4 Mar 17  2016 conf/
drwxr-xr-x   7 www		  www	  14 Dec 20 22:32 data/
dr-xr-xr-x   9 root		 wheel   512 Dec 20 22:29 dev/
-rw-------   1 root		 wheel  4096 Dec 20 22:30 entropy
drwxr-xr-x  28 root		 wheel  7808 Dec 20 22:32 etc/
drwxr-xr-x   3 root		 wheel	51 Nov 10 03:48 lib/
drwxr-xr-x   3 root		 wheel	 4 Nov 10 03:48 libexec/
drwxr-xr-x   2 root		 wheel	 2 Nov 10 03:51 media/
drwxr-xr-x   3 root		 wheel   128 Dec 20 22:30 mnt/
drwxr-xr-x   3 pinapples	wheel	12 Mar 26  2016 nonexistent/
dr-xr-xr-x   2 root		 wheel	 2 Nov 10 03:51 proc/
drwxr-xr-x   2 root		 wheel   143 Nov 10 03:48 rescue/
drwxr-xr-x   3 root		 wheel	17 Dec 20 21:51 root/
drwxr-xr-x   2 root		 wheel   128 Nov 10 03:48 sbin/
lrwxr-xr-x   1 root		 wheel	11 Nov 10 03:48 sys@ -> usr/src/sys
lrwxr-xr-x   1 root		 wheel	 8 Nov 10 03:48 tmp@ -> /var/tmp
drwxr-xr-x  16 root		 wheel	16 Nov 10 03:52 usr/
drwxr-xr-x  31 root		 wheel  1984 Dec 20 22:31 var/

Any ideas how best to proceed?

 

[Starion]

Same exact problem here. Step 1/5 (20%) has been stuck for over an hour.

 

[Lorem-Ipsum]

I have now done a completely fresh install of 9.10.2

It appeared to work normally but when I restored my previous configuration backup the same issue returns.

I have no idea how to proceed now.

 

[Brian Bialik]

Same exact problem here. Step 1/5 (20%) has been stuck for over an hour.

Identical to my problem. I believe this is a botched update and needs to be fixed ASAP.

 

[Starion]

Response I've seen so far is to let it run. May take HOURS.

 

[Starion]

It took HOURS. But it is working now.

 

[Stux]

Just found this file

Code:

-rw-r--r--   1 root	 wheel		 0 Oct 24  2016 SELECT tun_var, tun_value FROM system_tunable WHERE tun_enabled = 1 AND tun_type = 'sysctl' ORDER BY id

Empty file, from Oct 24 2016. I assume it was from a bad/failed update. Anyway, deleted it. If you need to delete it make sure you use double quotes around the filename!

rm -rf "SELECT tun_var, tun_value FROM system_tunable WHERE tun_enabled = 1 AND tun_type = 'sysctl' ORDER BY id"