SOLVED Blank web gui with no images and unusable

[gilgha]

Ahoy FreeNAS community !

My FreeNAS installation ran and upgraded perfectly for more than 1 year but today I am facing an annoying issue. In fact, I just realised the web gui is totally broken !

I am running FreeNAS 9.10.2-U4 as you can see from the second screenshot. I did the upgrade recently but I did not notice any issue afterwards. Of course, I already tried to reboot but same result. I also don't think the browser is the issue because I tried on Chrome, Safari and finally Firefox Dev Edition which has no plugin enabled, pure Firefox engine.

While writing this post I just realised that I cannot access smb shares anymore since I rebooted the system :-( ! I am however able to use the owncloud web gui running in a jail on the system and access my files from there. I also have no issue connecting with ssh and access the files. The smb service seems to be listening on the network:


[root@freenas] ~# netstat -an | grep 445
tcp4 0 0 *.445 *.* LISTEN
tcp6 0 0 *.445 *.* LISTEN


I'm kind of stuck right now. I don't have any idea where to start to troubleshoot this. I am quite confident with the command line of FreeBSD I manage multiple jails on this box. However, I never had to manage the FreeNAS system through the cli except for some simple ZFS tasks.

Does someone has already faced the same issue here?

I would really appreciate some help :) ! My FreeNAS installation is quite complex with multiple jails, backup cron jobs and such and it would be really painful if I had to reinstall the whole thing :-(

Thanks in advance

 

[danb35]

The most likely explanation is that your boot device has died. You have a backup of your config file, right?

 

[gilgha]

The most likely explanation is that your boot device has died. You have a backup of your config file, right?

I do have a configuration backup from a month ago generated from the web gui. I don't think anything as changed since then except for the FreeNAS version which was 9.10.2-U2 at that time.

I also just grabbed the database config file from the running system in the following location: /data/freenas-v1.db. Is this equivalent to a configuration backup generated from the web gui?

What's next then? Should I reinstall the FreeNAS software completely and restore the config?

Still, I don't see any clue that the boot device has failed since I was able to reboot the box and I have no issue reading files in / :

[root@freenas] ~# df -h | grep boot
freenas-boot/ROOT/9.10.2-U4 3.6G 647M 3.0G 17% /
freenas-boot/grub 3.0G 6.3M 3.0G 0% /boot/grub

[root@freenas] ~# gpart status
Name Status Components
ada0p1 OK ada0
ada0p2 OK ada0
ada1p1 OK ada1
ada1p2 OK ada1
ada2p1 OK ada2
ada2p2 OK ada2
ada3p1 OK ada3
ada3p2 OK ada3
da0p1 OK da0
da0p2 OK da0


 

[joeschmuck]

Grab yourself a different USB Flash drive (assuming that is the device) and install FreeNAS 9.10.2-U5 to the device. After you boot up, restore the configuration file you created 1 month ago. This should restore your system to normal operation.

 

[gilgha]

Grab yourself a different USB Flash drive (assuming that is the device) and install FreeNAS 9.10.2-U5 to the device. After you boot up, restore the configuration file you created 1 month ago. This should restore your system to normal operation.

I will try to restoration process now and keep you updated.

However, I just found more useful troubleshooting information. I am actually facing the exact same issue as described in this thread:
https://forums.freenas.org/index.php?threads/system-compromised.43081/

The guy finally decided to restore his system and did not seem to have found the root cause of the issue.

On my system, the first error log message of /var/log/auth.log is the following:
Jun 8 03:01:01 freenas sudo: root : unable to stat /usr/local/etc/sudoers : Permission denied ; TTY=unknown ; PWD=/root ; USER=root ;

This message is repeating multiple times every seconds. The file /usr/local/etc/sudoers has the following content:


... (truncated) ...
# Let find_alias_for_smtplib.py runs as root (it needs database access)
ALL ALL=(ALL) NOPASSWD: /etc/find_alias_for_smtplib.py


Which correspond to the process consuming a lot of CPU power in htop:

 

[gilgha]

Alright, I reinstalled the FreeNAS image on an other USB stick, restored the config and the system is back to normal.

Thank you very much for your danb35 and joeschmuck

 

[danb35]

The guy finally decided to restore his system and did not seem to have found the root cause of the issue.

"The guy" was me, and as you see in that thread, there were a number of weird things going on. I was torn between trying to figure it out, and wanting to get the system running again; the latter won out. Glad your system's back.

 

[gilgha]

"The guy" was me, and as you see in that thread, there were a number of weird things going on. I was torn between trying to figure it out, and wanting to get the system running again; the latter won out. Glad your system's back.

Indeed... Sorry about that, I did not notice :-D !

 

[joeschmuck]

Indeed... Sorry about that, I did not notice :-D !

I noticed you @danb35 , I noticed you ;)