import libxml2 fails and causes webgui/ix-nignx to fail as well

[cout]

Hi everyone

I have an 8.3 release on an amd x64 which fails to start the webgui. I rebooted today to replace a failed drive in a zraid and what I did, though I doubt it that it has anything with my issue, was offlined the disk from the webgui, shut the machine down and physically replaced the disk and once it booted back up again I ran into all sorts of errors. I also tried to delete/recreate the nginx files by doing:

# rm /usr/local/etc/nginx/nginx.conf*
# service ix-nginx start
# nginx

Below is the log, if someone can give me a hand here. Thank you.

Code:

May 20 15:43:45 nas.local syslogd: restart
May 20 15:43:45 nas.local syslogd: kernel boot file is /boot/kernel/kernel
May 20 15:43:46 nas.local kernel: Setting hostuuid: 00020003-0004-0005-0006-00070008009.
May 20 15:43:46 nas.local kernel: Setting hostid: 0x81f4ec68.
May 20 15:43:46 nas.local kernel: /dev/ufs/FreeNASs4: FILE SYSTEM CLEAN; SKIPPING CHECKS
May 20 15:43:46 nas.local kernel: /dev/ufs/FreeNASs4: clean, 37750 free (30 frags, 4715 blocks, 0.1% fragmentation)
May 20 15:43:46 nas.local kernel: Traceback (most recent call last):
May 20 15:43:46 nas.local kernel: File "/usr/local/www/freenasUI//notifier.py", line 42, in <module>
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: import libxml2
May 20 15:43:46 nas.local kernel: ValueError
May 20 15:43:46 nas.local kernel: :
May 20 15:43:46 nas.local kernel: bad marshal data (unknown type code)
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: Traceback (most recent call last):
May 20 15:43:46 nas.local kernel: File "/usr/local/www/freenasUI//notifier.py", line 42, in <module>
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: import libxml2
May 20 15:43:46 nas.local kernel: ValueError
May 20 15:43:46 nas.local kernel: :
May 20 15:43:46 nas.local kernel: bad marshal data (unknown type code)
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: Traceback (most recent call last):
May 20 15:43:46 nas.local kernel: File "/usr/local/www/freenasUI//notifier.py", line 42, in <module>
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: import libxml2
May 20 15:43:46 nas.local kernel: File "/usr/local/www/freenasUI//notifier.py", line 42, in <module>
May 20 15:43:46 nas.local kernel: File "/usr/local/www/freenasUI//notifier.py", line 42, in <module>
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: import libxml2
May 20 15:43:46 nas.local root: /etc/rc: WARNING: Dump device does not exist.  Savecore not run.
May 20 15:43:46 nas.local root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
May 20 15:43:46 nas.local kernel: /etc/rc: WARNING: Dump device does not exist.  Savecore not run.
May 20 15:43:46 nas.local kernel: /etc/rc: WARNING: failed precmd routine for vmware_guestd
May 20 15:43:46 nas.local ntpd[1762]: ntpd 4.2.4p5-a (1)
May 20 15:43:46 nas.local kernel: Updating motd:
May 20 15:43:46 nas.local kernel: .
May 20 15:43:46 nas.local kernel: Starting ntpd.
May 20 15:43:46 nas.local ntpd[1763]: precision = 1.676 usec
May 20 15:43:46 nas.local ntpd[1763]: Listening on interface #0 wildcard, 0.0.0.0#123 Disabled
May 20 15:43:46 nas.local ntpd[1763]: Listening on interface #1 wildcard, ::#123 Disabled
May 20 15:43:46 nas.local ntpd[1763]: Listening on interface #2 nfe0, 10.113.74.4#123 Enabled
May 20 15:43:46 nas.local ntpd[1763]: Listening on interface #3 lo0, fe80::1#123 Enabled
May 20 15:43:46 nas.local ntpd[1763]: Listening on interface #4 lo0, ::1#123 Enabled
May 20 15:43:46 nas.local ntpd[1763]: Listening on interface #5 lo0, 127.0.0.1#123 Enabled
May 20 15:43:46 nas.local ntpd[1763]: Listening on routing socket on fd #26 for interface updates
May 20 15:43:46 nas.local ntpd[1763]: kernel time sync status 2040
May 20 15:43:46 nas.local kernel: 16+0 records in
May 20 15:43:46 nas.local kernel: 16+0 records out
May 20 15:43:46 nas.local kernel: 16 bytes transferred in 0.000047 secs (340654 bytes/sec)
May 20 15:43:46 nas.local kernel: Generating a 1024 bit RSA private key
May 20 15:43:46 nas.local kernel: .
May 20 15:43:46 nas.local last message repeated 7 times
May 20 15:43:46 nas.local kernel: +
May 20 15:43:46 nas.local last message repeated 5 times
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: .
May 20 15:43:46 nas.local last message repeated 5 times
May 20 15:43:46 nas.local kernel: +
May 20 15:43:46 nas.local last message repeated 5 times
May 20 15:43:46 nas.local kernel:
May 20 15:43:46 nas.local kernel: writing new private key to '/etc/ssl/freenas/CA/private/cakey.key'
May 20 15:43:46 nas.local kernel: -----
May 20 15:43:46 nas.local kernel: writing RSA key
May 20 15:43:48 nas.local kernel: Removing stale Samba tdb files:
May 20 15:43:48 nas.local kernel: .
May 20 15:43:48 nas.local kernel: done
May 20 15:43:48 nas.local kernel: Starting nmbd.
May 20 15:43:48 nas.local kernel: Starting smbd.
May 20 15:43:48 nas.local kernel: Starting smartd.
May 20 15:43:54 nas.local ntpd[1763]: synchronized to 198.55.111.50, stratum 2
May 20 15:43:54 nas.local ntpd[1763]: time reset -0.272198 s
May 20 15:43:59 nas.local kernel: Traceback (most recent call last):
May 20 15:43:59 nas.local kernel: File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: execute_from_command_line(sys.argv)
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: utility.execute()
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: self.fetch_command(subcommand).run_from_argv(self.argv)
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: self.execute(*args, **options.__dict__)
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 231, in execute
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: self.validate()
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 266, in validate
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: num_errors = get_validation_errors(s, app)
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/core/management/validation.py", line 30, in get_validation_errors
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: for (app_name, error) in get_app_errors().items():
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/db/models/loading.py", line 158, in get_app_errors
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: self._populate()
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/db/models/loading.py", line 64, in _populate
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: self.load_app(app_name, True)
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/db/models/loading.py", line 88, in load_app
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: models = import_module('.models', app_name)
May 20 15:43:59 nas.local kernel: File "/usr/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: __import__(name)
May 20 15:43:59 nas.local kernel: File "/usr/local/www/freenasUI/../freenasUI/account/models.py", line 31, in <module>
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: from freenasUI..notifier import notifier
May 20 15:43:59 nas.local kernel: File "/usr/local/www/freenasUI/../freenasUI//notifier.py", line 42, in <module>
May 20 15:43:59 nas.local kernel:
May 20 15:43:59 nas.local kernel: import libxml2
May 20 15:43:59 nas.local kernel: ValueError
May 20 15:43:59 nas.local kernel: :
May 20 15:43:59 nas.local kernel: bad marshal data (unknown type code)
May 20 15:43:59 nas.local kernel:
May 20 15:44:00 nas.local kernel: Starting django.
May 20 15:44:05 nas.local kernel: Waiting for http...
May 20 15:44:10 nas.local kernel: Failed.

 

[William Grzybowski]

That means the USB flash drive is corrupted.

Save config and reinstall.

 

[cout]

Oh man, it can't be that easy!!!! I am still amazed and going through the drives here after restoring and I am shocked at how easy it was. In fact it was so easy that I am amazed it still works. What I did:

Code:

ssh -l root mylocalnas
#cd /data
#ftp mysite.com
#put freenas-v1.db

After copying/installing on a new flash drive the same steps again after I've started ssh through the web interface and changed the admin/root password:

Code:

ssh -l root my_new_nas
#cd /data
#ftp mysite.com
#get freenas-v1.db
#reboot

As you can see I didn't even bother to delete/rename the freeenas-v1.db that was already there. Rebooted and it's magic. I still can't believe it!!!! That was fun. One thing though: you have to be very careful to install/burn the exact same image that you had before the crash. Always keep the image that you've originally installed someplace safe, which I didn't, but I knew I had the 8.3.0. I don't know what the r12701M means, FreeNAS-8.3.0-RELEASE-x64 (r12701M), I'm guessing it's a release number for the 8.3.0 version. Anyhow, thanks, I figured I could replace the flash drive and start a reinstall but I was afraid and thinking it's too complicated that's why I tried to fix the django and the libxml problems first.
The guide to restore a freenas from command line? That's pretty much it, no big deal, copy/restore the /data/freenas-v1.db from the old install to the new one and you're done :). Oh yeah, and when I rebooted, my disk was still showing offline and I just had to replace it from the webgui and now it's resilvering. Amazing!!!
Thank you!!!

 

[William Grzybowski]

You dont need to use the same image, as long as you update the config file using the WebGUI.

 

[cout]

Hmmmm... that's also interesting. I will test that to see if it works on an offline machine. I remember reading about devastating issues to be aware when upgrading to a newer image, zfs upgrades seem to be scary, and I assumed this is the case. I did have the conf saved previously from the webgui but I looked at it and it didn't have the .db extension like the one in the /data folder, it is probably parsed by the system after the upload. It was just a Freenas-8.3.0 filename and I was afraid to try and restore from that. I thought the safest way was to copy the .dat file and then re-copy back, which in my case did the trick. BTW still waiting for resilvering 2.92 TB of data on the failed drive. I also see that it did the zfs snapshot last night while it was running on the damaged USB even though I didn't get an email last night with the status of pool, security, etc. Probably cron failed to start as well.
Thank you for your help.

 

[paleoN]

I remember reading about devastating issues to be aware when upgrading to a newer image, zfs upgrades seem to be scary, and I assumed this is the case. I did have the conf saved previously from the webgui but I looked at it and it didn't have the .db extension like the one in the /data folder, it is probably parsed by the system after the upload.

The administrator is required to manually upgrade zpool/zfs versions. It doesn't happen automatically. FreeNAS 8.3.X all are using the 8.3 kernel. One would assume you would have accounted for the issue already.

At least several people "upgrade" by doing fresh installs to alternate USB sticks and restoring the latest previous settings.

 

[cout]

The administrator is required to manually upgrade zpool/zfs versions. It doesn't happen automatically. FreeNAS 8.3.X all are using the 8.3 kernel. One would assume you would have accounted for the issue already.

At least several people "upgrade" by doing fresh installs to alternate USB sticks and restoring the latest previous settings.

I've never had any issues with this unit. I knew about the conf restoring but I thought of it only for disaster situations, which didn't seem to me to be the case, unfortunately it was. This unit is a used 24 bay supermicro bought from ebay, many people got them about a year ago with boards and ram already on them, that's been running ever since. Never thought of upgrading the zfs, it's been working for us ever since, and I have iSCSI turned on it, four-five servers back themselves up every other night, file sharing is turned on, snapshots, it's a workhorse in other words. Even if I were to upgrade the zfs on it I'd triple backup all of the stuff on it and then attempt it. Why would I update in the first place? I don't need any sophisticated disk encryption, that being one of the reasons for an upgrade that I read somewhere, it's not exposed to the outside and it's been so good to me so far. I don't want to bother it and I wouldn't have if it weren't for that failed disk.

 

[cyberjock]

Upgrading the zpool has nothing to do with encryption(unless you are using Oracle products). Whatver you read you either misunderstood or is in error.

Upgrading is very simple, not particularly risky, but improves alot of things with your zpool. Not the least of which is reliability.

I tell people that if FreeNAS isn't doing anything wrong with the new version there's no reason not to upgrade the zpool. You can even do it with the zpool online!

 

[cout]

I currently sit at ZFS 5 with ZFS pool 28 and it seems to be enough. You were right about the encryption thing, I remember reading something somewhere.

Code:

Encryption

With Oracle Solaris, the encryption capability in ZFS[59] is embedded into the I/O pipeline. During writes, a block may............

http://en.wikipedia.org/wiki/ZFS

My last experience with upgrading/updating was with a brand new 9.1 FreeBSD, in my attempts to replace an 8.2 box that was working just fine, installed from scratch on a machine and the pain that followed because libpcre would fail to adjust to apache/php5.x and the whole thing would crumble. I tried everything, downgrading php/apache it wouldn't work, or at least I couldn't get it to work and I'm still on the 8.2 for that matter. My experience has been that the conservative side is the winner and I try to avoid as much as I can bleeding edge stuff, especially on production machines. I didn't know, however, that FreeNAS is that versatile between versions.

If it's just saving the conf file, replacing the thumbdrive with the latest version, every once in a while, then that seems to be awesome but I'd keep that only for when issues arise. I see people with way more complicated setups than mine, ZIL's on ssd's, etc. and that's a really scary thought for me when I think of upgrading, but they seem to be ok.

 

[cyberjock]

If you are using v28 then you've already upgraded at some point. The whole "upgrade" was from v15 to v28. :P