SOLVED Update from 9.3 stable to 9.3.1 failure.

[PhilZJ81]

EDIT: Changed to solved, I ended up buying a new usb stick and doing a clean install. The SaveConfiguration feature was causing several UnicodeDecode errors during bootup after importing the new configuration so had to redo the config by hand. Nevertheless, thanks for the help and suggestions!

In another thread I mentioned that i had a failure during my update process going from 9.3 to 9.3.1.

here is my update.failed output, can someone please give me an hand, I don't even know where to start.

I'm reverting back to my original boot now.

Code:

ps: cannot mmap corefile
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
ps: empty file: Invalid argument
Running migrations for api:
- Nothing to migrate.
- Loading initial data for api.
Installed 0 object(s) from 0 fixture(s)
Running migrations for freeadmin:
- Nothing to migrate.
- Loading initial data for freeadmin.
Installed 0 object(s) from 0 fixture(s)
Running migrations for services:
- Migrating forwards to 0190_dup_webdav.
> storage:0051_auto__add_field_task_task_excludesystemdataset
> services:0156_auto__add_field_iscsitargetextent_iscsi_target_extent_blocksize__add_f
> services:0157_auto__del_field_iscsitarget_iscsi_target_logical_blocksize
> services:0158_auto__add_field_afp_afp_srv_homename
> services:0159_auto__add_field_nfs_nfs_srv_v4_krb
> services:0160_pwencrypt
- Migration 'services:0160_pwencrypt' is marked for no-dry-run.
> services:0161_auto__chg_field_cifs_cifs_srv_bindip
> services:0162_auto__add_field_iscsitargetextent_iscsi_target_extent_rpm
> services:0163_auto__add_field_ssh_ssh_host_dsa_key_cert_pub__add_field_ssh_ssh_host_
> services:0164_nfs_srv_v4_krb_fix
- Migration 'services:0164_nfs_srv_v4_krb_fix' is marked for no-dry-run.
> services:0165_auto__add_field_snmp_snmp_v3__add_field_snmp_snmp_v3_username__add_fie
> services:0165_fix_ddns_domains_delimiters
- Migration 'services:0165_fix_ddns_domains_delimiters' is marked for no-dry-run.
> services:0166_pwencrypt
- Migration 'services:0166_pwencrypt' is marked for no-dry-run.
> services:0167_auto__add_field_ups_ups_powerdown
> services:0168_snmp_v3_literal
- Migration 'services:0168_snmp_v3_literal' is marked for no-dry-run.
> services:0169_auto__add_field_iscsitargetportal_iscsi_target_portal_discoveryauthmet
> services:0170_copy_discovery_fields
- Migration 'services:0170_copy_discovery_fields' is marked for no-dry-run.
> services:0171_auto__del_field_iscsitargetglobalconfiguration_iscsi_discoveryauthmeth
> services:0172_auto__add_field_iscsitargetextent_iscsi_target_extent_serial
> services:0173_auto__del_field_iscsitarget_iscsi_target_serial
> services:0174_auto__add_iscsitargetgroups
> services:0175_target_to_groups
- Migration 'services:0175_target_to_groups' is marked for no-dry-run.
> services:0176_auto__del_field_iscsitarget_iscsi_target_portalgroup__del_field_iscsit
> services:0177_auto__del_field_iscsitargetgroups_iscsi_target_type
> services:0178_auto__add_fiberchanneltotarget
> services:0179_auto__add_field_iscsitarget_iscsi_target_mode
> services:0180_auto__del_unique_iscsitargettoextent_iscsi_extent__add_unique_iscsitar
> services:0181_auto__add_field_snmp_snmp_v3_authtype__add_field_snmp_snmp_v3_privprot
> services:0182_auto__add_unique_iscsitargetgroups_iscsi_target_iscsi_target_portalgro
> services:0183_auto__chg_field_fiberchanneltotarget_fc_target
> services:0184_auto__del_fiberchanneltotarget__add_fibrechanneltotarget
> services:0185_auto__add_field_iscsitargetextent_iscsi_target_extent_ro
> services:0186_auto__add_field_ssh_ssh_kerberosauth
> services:0187_auto__add_field_nfs_nfs_srv_16
> services:0188_auto__add_field_ssh_ssh_bindiface
> services:0189_auto__add_field_nfs_nfs_srv_v4_v3owner
> services:0190_dup_webdav
- Migration 'services:0190_dup_webdav' is marked for no-dry-run.
- Loading initial data for services.
Installed 0 object(s) from 0 fixture(s)
Running migrations for vcp:
- Migrating forwards to 0002_auto__add_vcenterconfiguration.
> vcp:0001_initial
- Migration 'vcp:0001_initial' is marked for no-dry-run.
> vcp:0002_auto__add_vcenterconfiguration
- Loading initial data for vcp.
Installed 0 object(s) from 0 fixture(s)
Running migrations for support:
- Migrating forwards to 0003_auto__add_field_support_support_email.
> support:0003_auto__add_field_support_support_email
- Loading initial data for support.
Installed 0 object(s) from 0 fixture(s)
Running migrations for directoryservice:
- Migrating forwards to 0057_auto__add_field_activedirectory_ad_disable_freenas_cache.
> directoryservice:0041_auto__add_field_ldap_ldap_schema
> directoryservice:0042_auto__add_kerberossettings
> directoryservice:0043_auto__chg_field_ldap_ldap_binddn
> directoryservice:0044_auto__add_field_idmap_rfc2307_idmap_rfc2307_ssl__add_field_idmap_rfc23
> directoryservice:0045_auto__add_field_activedirectory_ad_netbiosname_b
> directoryservice:0045_auto__add_field_idmap_rfc2307_idmap_rfc2307_ldap_user_dn_password
> directoryservice:0046_auto__add_kerberosprincipal
> directoryservice:0047_migrate_kerberos_keytabs_to_principals
- Migration 'directoryservice:0047_migrate_kerberos_keytabs_to_principals' is marked for no-dry-run.
> directoryservice:0048_auto__add_field_activedirectory_ad_kerberos_principal__add_field_ldap_
> directoryservice:0049_populate_kerberos_principals
- Migration 'directoryservice:0049_populate_kerberos_principals' is marked for no-dry-run.
> directoryservice:0050_auto__del_field_activedirectory_ad_kerberos_keytab__del_field_ldap_lda
> directoryservice:0051_auto__del_field_kerberoskeytab_keytab_principal
> directoryservice:0052_change_ad_timeout_defaults
- Migration 'directoryservice:0052_change_ad_timeout_defaults' is marked for no-dry-run.
> directoryservice:0053_auto__del_field_activedirectory_ad_netbiosname__add_field_activedirect
> directoryservice:0054_auto__add_field_activedirectory_ad_allow_dns_updates
> directoryservice:0055_auto__add_field_ldap_ldap_netbiosname_a__add_field_ldap_ldap_netbiosna
> directoryservice:0056_migrate_ldap_netbiosname
- Migration 'directoryservice:0056_migrate_ldap_netbiosname' is marked for no-dry-run.
> directoryservice:0057_auto__add_field_activedirectory_ad_disable_freenas_cache
- Loading initial data for directoryservice.
Installed 0 object(s) from 0 fixture(s)
Running migrations for account:
- Migrating forwards to 0023_auto__add_field_bsdusers_bsdusr_microsoft_account.
> account:0023_auto__add_field_bsdusers_bsdusr_microsoft_account
- Loading initial data for account.
Installed 0 object(s) from 0 fixture(s)
Running migrations for network:
- Migrating forwards to 0018_auto__add_field_alias_alias_vip__add_field_alias_alias_v4address_b__ad.
> network:0016_auto__add_field_interfaces_int_vip__add_field_interfaces_int_vhid__add
> network:0017_auto__add_field_globalconfiguration_gc_hostname_b
> network:0018_auto__add_field_alias_alias_vip__add_field_alias_alias_v4address_b__ad
- Loading initial data for network.
Installed 0 object(s) from 0 fixture(s)
Running migrations for sharing:
- Migrating forwards to 0034_fix_wizard_cifs_vfsobjects.
> sharing:0032_auto__add_field_cifs_share_cifs_storage_task
> sharing:0033_add_periodic_snapshot_task
- Migration 'sharing:0033_add_periodic_snapshot_task' is marked for no-dry-run.
> sharing:0034_fix_wizard_cifs_vfsobjects
- Migration 'sharing:0034_fix_wizard_cifs_vfsobjects' is marked for no-dry-run.
- Loading initial data for sharing.
Installed 0 object(s) from 0 fixture(s)
Running migrations for system:
- Migrating forwards to 0100_auto__chg_field_tunable_tun_value.
> system:0093_auto__add_field_advanced_adv_boot_scrub
> system:0094_add_model_backup
> system:0095_auto__add_field_advanced_adv_periodic_notifyuser
> system:0096_auto__add_field_systemdataset_sys_uuid_b
> system:0097_auto__add_field_alert_timestamp
> system:0098_auto__add_field_alert_node__del_unique_alert_message_id__add_unique_al
> system:0099_auto__add_field_certificate_cert_chain__add_field_certificateauthority
> system:0099_email_pwenc
- Migration 'system:0099_email_pwenc' is marked for no-dry-run.
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.

! You *might* be able to recover with:   (migration cannot be dry-run; cannot discover commands)
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: system:0099_email_pwenc
The following content types are stale and need to be deleted:

    system | registration

Any objects related to these content types by a foreign key will also
be deleted. Are you sure you want to delete these content types?
If you're unsure, answer 'no'.

    Type 'yes' to continue, or 'no' to cancel: Traceback (most recent call last):
  File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 331, in migrate_many
    result = self.migrate(migration, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
    result = self.run(migration, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
    return self.run_migration(migration, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 84, in run_migration
    migration_function()
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
    return (lambda: direction(orm))
  File "/usr/local/www/freenasUI/../freenasUI/system/migrations/0099_email_pwenc.py", line 24, in forwards
    do_encrypt(orm['system.Email'], 'em_pass')
  File "/usr/local/www/freenasUI/../freenasUI/system/migrations/0099_email_pwenc.py", line 18, in do_encrypt
    setattr(obj, field, notifier().pwenc_encrypt(getattr(obj, field)))
  File "/usr/local/www/freenasUI/../freenasUI/middleware/notifier.py", line 5740, in pwenc_encrypt
    nonce = get_random_bytes(8)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/__init__.py", line 41, in get_random_bytes
    return _UserFriendlyRNG.get_random_bytes(n)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 228, in get_random_bytes
    return _get_singleton().read(n)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 215, in _get_singleton
    _singleton = _LockingUserFriendlyRNG()
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 159, in __init__
    _UserFriendlyRNG.__init__(self)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 86, in __init__
    self._ec = _EntropyCollector(self._fa)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/_UserFriendlyRNG.py", line 53, in __init__
    self._osrng = OSRNG.new()
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/OSRNG/posix.py", line 83, in new
    return DevURandomRNG(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/Crypto/Random/OSRNG/posix.py", line 44, in __init__
    f = open(self.name, "rb", 0)
IOError: [Errno 2] No such file or directory: '/dev/urandom'
Exception AttributeError: "'DevURandomRNG' object has no attribute 'closed'" in <bound method DevURandomRNG.__del__ of <Crypto.Random.OSRNG.posix.DevURandomRNG object at 0x80b62a0d0>> ignored

I'm also getting a this in my console:

Code:

May 22 20:46:20 BNAS manage.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x8101bf0d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:46:32 BNAS manage.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x8101bf0d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:46:33 BNAS syslog-ng[5943]: syslog-ng starting up; version='3.5.6'
May 22 20:46:33 BNAS kernel: pid 1971 (syslog-ng), uid 0: exited on signal 6 (core dumped)
May 22 20:47:11 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:48:11 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:49:12 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:50:12 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:51:12 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:51:20 BNAS manage.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x8101bf0d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:52:12 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:53:12 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete
May 22 20:54:13 BNAS alert.py: [system.alert:219] Alert module '<replication_status.ReplicationStatusAlert object at 0x80b3d23d0>' failed: no such column: storage_replication.repl_followdelete

 

[dlavigne]

Did you switch trains between a STABLE and a nightly at any point? What's the exact build version that you tried to update from (from System -> Information) and update to?

 

[PhilZJ81]

This is what my update window looked when i clicked the update now button:

View attachment 11820

What you say is correct about the 9.10 train. I did try to update to that, it failed, I reverted back to original boot, and then switched trains to 9.3 again and tried that update which also failed.

thanks

 

[PhilZJ81]

Did you switch trains between a STABLE and a nightly at any point? What's the exact build version that you tried to update from (from System -> Information) and update to?

Added clarification, I didn't switch to nightly, I remained on STABLE, but i tried 9.10 and then I tried 9.3.1

 

[dlavigne]

So, going from 9.10 to 9.3.1 is a downgrade and will result in weird errors like that. Do you have a boot environment from before you tried upgrading to 9.10? If so, if you boot into that and try to upgrade to 9.3.1 does that work?

Also, what sort of errors did you get when you tried to upgrade to 9.10? That is the recommended option as 9.3.x is in maintenance mode only.

 

[PhilZJ81]

So, going from 9.10 to 9.3.1 is a downgrade and will result in weird errors like that. Do you have a boot environment from before you tried upgrading to 9.10? If so, if you boot into that and try to upgrade to 9.3.1 does that work?

Also, what sort of errors did you get when you tried to upgrade to 9.10? That is the recommended option as 9.3.x is in maintenance mode only.

First, thank you for your help, and sorry to make you ask questions that I should have clarified in my opening post.

Currently I'm running:
Build FreeNAS-9.3-STABLE-201412090314
that's my default boot and my original configuration.

From that build I tried 2 things
1) went in Updates, selected 9.10 train, clicked update now, failed (I don't remember the error, I have to change my boot, but I can't now I'm running a scrub). When that failed I reverted back to my original Boot. (I tried this prior to having PLEX installed, or any jails, it was a pretty new build at that time. I only had my CIFS setup, my datasets, email configured.

[a couple of weeks later]
2) I went to Updates, Selected the 9.3 train (again I'm in my original default boot) clicked check now, then Update now and the errors that were a result of that process were listed at the top of this thread. At this point all I have on my FreeNAS, is CIFS share, a few datasets, a zfs volume and a standalone drive and a UPS setup. I also have a scrub setup and Smart test jobs.

I'm sure I'm forgetting more important details, so let me know what else I can provide

Thanks again

 

[dlavigne]

If you have another stick, I'd suggest doing a fresh install of the latest 9.10, boot from that, import your config, and see if all goes well. If it doesn't, you still have your original stick to fall back on. This may or may not work as you'll be getting nearly a year and a half's worth of updates.

Or to be more cautious, you could first try a fresh install of the latest 9.3.1 (again, on another stick), and if that likes your config, try updating to the latest 9.10.

 

[PhilZJ81]

If you have another stick, I'd suggest doing a fresh install of the latest 9.10, boot from that, import your config, and see if all goes well. If it doesn't, you still have your original stick to fall back on. This may or may not work as you'll be getting nearly a year and a half's worth of updates.

Or to be more cautious, you could first try a fresh install of the latest 9.3.1 (again, on another stick), and if that likes your config, try updating to the latest 9.10.

I'll have to research some more on how to copy all the config files and then re-import them. Is there any chance that doing that will hose the data on the drive and ZFS arrays?
Also, when doing the installation, should I have all my drives in and spinning, or should I pull the cord on them, do the install and then plug them back in?

I've not had this box 'that' long but I'd be pretty frustrated if it all went fubar. I have it setup pretty much as I wanted with the exception of shadow copies.

thanks

 

[dlavigne]

Super simple: System -> General -> Save Config and afterwards, System -> General -> Upload Config.

The whole point of separating the OS (and the config) from the storage drives is for scenarios just like this: a messed up OS doesn't affect the data stored on the drives.

 

[PhilZJ81]

Super simple: System -> General -> Save Config and afterwards, System -> General -> Upload Config.

The whole point of separating the OS (and the config) from the storage drives is for scenarios just like this: a messed up OS doesn't affect the data stored on the drives.

Great thanks, I did as you suggested.
I started with the latest 9.3.1 build, and imported my saved config. I do have a couple of errors during bootup, could you guys help me with them?

Attached screen shots.

thanks !!

 

[dlavigne]

Mmm, tracebacks are bad...

Just to verify, you installed 9.3.1/STABLE/201605170422?

 

[PhilZJ81]

Mmm, tracebacks are bad...

Just to verify, you installed 9.3.1/STABLE/201605170422?

oh no :(

I have

FreeNAS-9.3-STABLE-201605170422 yes

 

[dlavigne]

Well, you have a few options here:

1. you could report a bug which may or may not be fixed as 9.3 is in maintenance-only mode

2. you could try a fresh install of latest 9.10 and see if the config also blows that up

3. you could cut your losses, do a fresh install of latest 9.10 and recreate your config from scratch

 

[PhilZJ81]

Well, you have a few options here:

1. you could report a bug which may or may not be fixed as 9.3 is in maintenance-only mode

2. you could try a fresh install of latest 9.10 and see if the config also blows that up

3. you could cut your losses, do a fresh install of latest 9.10 and recreate your config from scratch

ok thanks, i'm trying option 2. During import configs for both i get errors related to directoryService_activedirectory... bla bla no such column. but I'm not using AD...
Do you know what those errors are for though? as far as I could tell everything was working on the NAS. the web interface for plex worked, and my CIFS shares were visible on the network, user credentials and everything looked good...

thanks

 

[PhilZJ81]

Alright, 9.10 has the same error...
I found a couple of bugs about it, but it doesn't tell me how to fix them. I tried to reset the root password via the gui, I also reset the password for the 1 user i created for my cifs shares. That's all the passwords i have...

https://bugs.pcbsd.org/issues/12811

And also, https://bugs.pcbsd.org/issues/12717

That bug talks about copying /data/pwenc_secret from the old boot device.

I can certainly do that if someone can help me out with how to do that and give it a shot.. thanks

 

[PhilZJ81]

I also just now realized that eventhough PLEX is responding, it isn't listed under my installed plugins.

Could it be related? What would you recommend? should I just re-install PLEX and see what happens? How do I remove it if it's not listed in the installed plugins?

 

[PhilZJ81]

Well, I did a fresh install of freeNAS 9.10 and re-did my setup from scratch..

unfortunately now I'm unable to install plex.

here's my thread about that. https://forums.freenas.org/index.php?threads/jail-installation-under-9-10.43682/

any help is greatly appreciated.