Was the 9.10.1-U1 update tested? Update failure discussion

[m0nkey_]

Pretty much the subject. Keep seeing reports on IRC, forum and bugs of critical errors when updating.

Examples:
https://forums.freenas.org/index.php?threads/critical-alert-update-failed-9-10-1-to-9-10-1-u1.46338/
https://bugs.freenas.org/issues/17789
https://forums.freenas.org/index.php?threads/bug-with-latest-9-10-updates.46327/
https://forums.freenas.org/index.php?threads/update-to-9-10-u1-failed-stuck-grub-rescue.46340/

Edit: Adding to the examples list

 

[Ericloewe]

Yeah, I saw those just after I'd pressed the upgrade button on my secondary server... and it all went well, huh.

I wonder what's specific to those setups... In my case, it was a brand-new installation on USB drives in an UEFI environment. Speaking of which, lemme check if I have a console again... nope, still no local console. Off to file some bug ticket stuff I go.

Edit: Config regeneration with grub-mkconfig -o /boot/grub/grub.cfg did the trick (Bug #17815)

 

[Ericloewe]

I'm going to go ahead and sticky this for now, to try and contain the barrage of threads.

 

[Jailer]

I'm sooooo glad I read the forums before performing any update.

 

[Ericloewe]

I'm sooooo glad I read the forums before performing any update.

Yeah, definitely not updating my main server that soon.

 

[SavageAUS]

Same issue as many others with the U1 update, rolled backed to FreeNAS-9.10.1 (d989edd) until issue has been fixed.

 

[STREBLO]

Shit, just updated, got big red flashing error, should I roll back?

Edit: Ahh found the other threads

Code:

less  /data/update.failed
Running migrations for api:
- Nothing to migrate.
- Loading initial data for api.
Installed 0 object(s) from 0 fixture(s)
Running migrations for freeadmin:
- Nothing to migrate.
- Loading initial data for freeadmin.
Installed 0 object(s) from 0 fixture(s)
Running migrations for support:
- Nothing to migrate.
- Loading initial data for support.
Installed 0 object(s) from 0 fixture(s)
Running migrations for vcp:
- Nothing to migrate.
- Loading initial data for vcp.
Installed 0 object(s) from 0 fixture(s)
Running migrations for directoryservice:
- Nothing to migrate.
- Loading initial data for directoryservice.
Installed 0 object(s) from 0 fixture(s)
Running migrations for services:
- Migrating forwards to 0198_iscsi_expunge_lun_auto.
> services:0198_iscsi_expunge_lun_auto
- Migration 'services:0198_iscsi_expunge_lun_auto' is marked for no-dry-run.
- Loading initial data for services.
Installed 0 object(s) from 0 fixture(s)
Running migrations for storage:
- Nothing to migrate.
- Loading initial data for storage.
Installed 0 object(s) from 0 fixture(s)
Running migrations for account:
- Migrating forwards to 0024_add_media_user_and_group.
> account:0024_add_media_user_and_group
- Migration 'account:0024_add_media_user_and_group' is marked for no-dry-run.
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.

! You *might* be able to recover with:   (migration cannot be dry-run; cannot discover commands)
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: account:0024_add_media_user_and_group
Traceback (most recent call last):
  File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
    ignore_ghosts = ignore_ghosts,
  File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
    success = migrator.migrate_many(target, workplan, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
    result = migrator.__class__.migrate_many(migrator, target, migrations, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 331, in migrate_many
    result = self.migrate(migration, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
    result = self.run(migration, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
    return self.run_migration(migration, database)
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 84, in run_migration
    migration_function()
  File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
    return (lambda: direction(orm))
  File "/usr/local/www/freenasUI/../freenasUI/account/migrations/0024_add_media_user_and_group.py", line 14, in forwards
    group.save()
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 545, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 573, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 654, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 687, in _do_insert
    using=using, raw=raw)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 232, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1514, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/www/freenasUI/../freenasUI/freeadmin/sqlite3_ha/base.py", line 376, in execute
    execute = Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: account_bsdgroups.bsdgrp_group

 

[Pheran]

Dammit, was hoping 9.10.1-U1 was going to fix things. I guess I'm going to 9.10-20160627 this weekend, that will at least get me to the new OS/plugins version.

 

[picklefish]

I can no longer launch my VM in phpVirtualBox so I had to roll back.

Not sure if this is related but I'm getting this when I try to launch:


paste here: http://pastebin.com/raw/u9Y16JDY since I can't get formatting to work right.

Code:

Exception Object
(
    [message:protected] => The virtual machine 'VPNUbuntuMate' has terminated unexpectedly during startup with exit code 1
    [string:Exception:private] =>
    [code:protected] => 0
    [file:protected] => /usr/local/www/phpvirtualbox/lib/vboxconnector.php
    [line:protected] => 2382
    [trace:Exception:private] => Array
        (
            [0] => Array
                (
                     => /usr/local/www/phpvirtualbox/lib/vboxconnector.php
                    [line] => 928
                    [function] => remote_progressGet
                    [class] => vboxconnector
                    [type] => ->
                    [args] => Array
                        (
                            [0] => Array
                                (
                                    [progress] => 6f102e7fd11e546c-0000000000000053
                                    [_persist] => Array
                                        (
                                            [vboxHandle] => 6f102e7fd11e546c-0000000000000051
                                        )

                                    [fn] => progressGet
                                )

                        )

                )

            [1] => Array
                (
                     => /usr/local/www/phpvirtualbox/lib/ajax.php
                    [line] => 309
                    [function] => __call
                    [class] => vboxconnector
                    [type] => ->
                    [args] => Array
                        (
                            [0] => progressGet
                            [1] => Array
                                (
                                    [0] => Array
                                        (
                                            [progress] => 6f102e7fd11e546c-0000000000000053
                                            [_persist] => Array
                                                (
                                                    [vboxHandle] => 6f102e7fd11e546c-0000000000000051
                                                )

                                            [fn] => progressGet
                                        )

                                    [1] => Array
                                        (
                                            [0] => Array
                                                (
                                                    [data] => Array
                                                        (
                                                            [responseData] => Array
                                                                (
                                                                    [progress] => 6f102e7fd11e546c-0000000000000053
                                                                    [info] => Array
                                                                        (
                                                                            [completed] => 1
                                                                            [canceled] =>
                                                                            [description] => Starting VM
                                                                            [operationDescription] => Creating process for virtual machine "VPNUbuntuMate" (headless)
                                                                            [timeRemaining] => Array
                                                                                (
                                                                                )

                                                                            [timeElapsed] => Array
                                                                                (
                                                                                    [days] => 17072
                                                                                    [hours] => 7
                                                                                    [minutes] => 52
                                                                                    [seconds] => 46
                                                                                )

                                                                            [percent] => 0
                                                                        )

                                                                )

                                                            [success] => 1
                                                            [key] => 53474945cfd7ac0bf8114767c209c2a6
                                                        )

                                                    [errors] => Array
                                                        (
                                                        )

                                                    [persist] => Array
                                                        (
                                                        )

                                                    [messages] => Array
                                                        (
                                                        )

                                                )

                                        )

                                )

                        )

                )

            [2] => Array
                (
                     => /usr/local/www/phpvirtualbox/lib/ajax.php
                    [line] => 309
                    [function] => progressGet
                    [class] => vboxconnector
                    [type] => ->
                    [args] => Array
                        (
                            [0] => Array
                                (
                                    [progress] => 6f102e7fd11e546c-0000000000000053
                                    [_persist] => Array
                                        (
                                            [vboxHandle] => 6f102e7fd11e546c-0000000000000051
                                        )

                                    [fn] => progressGet
                                )

                            [1] => Array
                                (
                                    [0] => Array
                                        (
                                            [data] => Array
                                                (
                                                    [responseData] => Array
                                                        (
                                                            [progress] => 6f102e7fd11e546c-0000000000000053
                                                            [info] => Array
                                                                (
                                                                    [completed] => 1
                                                                    [canceled] =>
                                                                    [description] => Starting VM
                                                                    [operationDescription] => Creating process for virtual machine "VPNUbuntuMate" (headless)
                                                                    [timeRemaining] => Array
                                                                        (
                                                                        )

                                                                    [timeElapsed] => Array
                                                                        (
                                                                            [days] => 17072
                                                                            [hours] => 7
                                                                            [minutes] => 52
                                                                            [seconds] => 46
                                                                        )

                                                                    [percent] => 0
                                                                )

                                                        )

                                                    [success] => 1
                                                    [key] => 53474945cfd7ac0bf8114767c209c2a6
                                                )

                                            [errors] => Array
                                                (
                                                )

                                            [persist] => Array
                                                (
                                                )

                                            [messages] => Array
                                                (
                                                )

                                        )

                                )

                        )

                )

        )

    [previous:Exception:private] =>
)

 

[John_G]

I can no longer launch my VM in phpVirtualBox so I had to roll back.

Not sure if this is related but I'm getting this when I try to launch:
<...>

Same here. Rolled back to previous version.

 

[Mirfster]

Meh, this is where I am glad my laziness got the better of me (and that I do test updates on a VM instance of FreeNAS). :)

 

[GairyS]

I just had a failure as well, updating from 9.10.1 (d989edd) to 9.10.1-U1 (ff51a5d). Rebooting to old environment brought me back up, but a second attempt to update yielded the same result.

Running migrations for api:
- Nothing to migrate.
- Loading initial data for api.
Installed 0 object(s) from 0 fixture(s)
Running migrations for freeadmin:
- Nothing to migrate.
- Loading initial data for freeadmin.
Installed 0 object(s) from 0 fixture(s)
Running migrations for directoryservice:
- Nothing to migrate.
- Loading initial data for directoryservice.
Installed 0 object(s) from 0 fixture(s)
Running migrations for services:
- Migrating forwards to 0198_iscsi_expunge_lun_auto.
> services:0198_iscsi_expunge_lun_auto
- Migration 'services:0198_iscsi_expunge_lun_auto' is marked for no-dry-run.
- Loading initial data for services.
Installed 0 object(s) from 0 fixture(s)
Running migrations for support:
- Nothing to migrate.
- Loading initial data for support.
Installed 0 object(s) from 0 fixture(s)
Running migrations for vcp:
- Nothing to migrate.
- Loading initial data for vcp.
Installed 0 object(s) from 0 fixture(s)
Running migrations for account:
- Migrating forwards to 0024_add_media_user_and_group.
> account:0024_add_media_user_and_group
- Migration 'account:0024_add_media_user_and_group' is marked for no-dry-run.
! Error found during real run of migration! Aborting.
! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.
! You *might* be able to recover with: (migration cannot be dry-run; cannot discover commands)
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: account:0024_add_media_user_and_group
Traceback (most recent call last):
File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
ignore_ghosts = ignore_ghosts,
File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 331, in migrate_many
result = self.migrate(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
result = self.run(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
return self.run_migration(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 84, in run_migration
migration_function()
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
return (lambda: direction(orm))
File "/usr/local/www/freenasUI/../freenasUI/account/migrations/0024_add_media_user_and_group.py", line 26, in forwards
user.save()
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 545, in save
force_update=force_update, update_fields=update_fields)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 573, in save_base
updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 654, in _save_table
result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 687, in _do_insert
using=using, raw=raw)
File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 232, in _insert
return insert_query(self.model, objs, fields, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 1514, in insert_query
return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 903, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/../freenasUI/freeadmin/sqlite3_ha/base.py", line 376, in execute
execute = Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: UNIQUE constraint failed: account_bsdusers.bsdusr_username

 

[JoshDW19]

I've pointed this thread to QA and Kris' attention. I'll let you all know if I hear anything.

 

[GairyS]

I've pointed this thread to QA and Kris' attention. I'll let you all know if I hear anything.

Hi JoshDW19, I appreciate your immediate attention to this. I just found https://bugs.freenas.org/issues/17789 and was about to add some detail to the case. From what I can tell with my error, it looks like mine (as a few others in the bug report) are failing due to a media user. Mine was created by SickRage.

This is a home server that I'm about to revert back to my original image, but I can update again later if anyone would benefit from the collected logs.

Code:

Error in migration: account:0024_add_media_user_and_group

~% cat /etc/passwd | grep media
media:*:816:1001:Media for SickRage:/nonexistent:/bin/csh

 

[Kris Moore]

The two issues referenced have already been fixed in source, and will be in -U2. Specifically they relate to updating if you've manually added a "media" user. If so, you can wait for -U2, or remove the user before updating.

The second issue is related to UEFI installs only (And not all of them, I still cannot reproduce it here locally). It too has been fixed in source and will be in -U2. We are looking at -U2 being in the next ~30 days or so. Anyway, thanks for the bug reports, we appreciate your help in tracking these down.

 

[m0nkey_]

Specifically they relate to updating if you've manually added a "media" user. If so, you can wait for -U2, or remove the user before updating.

This is going to cover a wide user base. Asking users to wait (especially when there is a critical Samba CVE) or asking people to remove any 'media' account they may have created, possibly breaking workflows, access, etc. in my view should not be an acceptable answer.

Would it be a better idea to pull the update and re-issue it when it's fixed?

 

[Kuro Houou]

My system had the issue with the media account being there, although it shows the U1 update is the current running version. My question is, what is the side effect of leaving the system at U1 for now until U2 comes out, or will things not work and I need to roll back to the version before U1?

 

[JoshDW19]

According to Kris if you want to be completely safe it would probably be best to roll back until we know more. The devs will update the ticket as more information becomes available.

 

[rosabox]

I have the same problem as picklefish
I can no longer launch my VM in phpVirtualBox but I didn't roll back because the VM is just for testing.
Should I roll back or wait for U2?

 

[Kuro Houou]

I just decided to activate the last boot version and will stay of U1 until U2 comes out, sounds like it will be out in 30 days.. maybe less if were lucky to fix this issue. I don't get why we can't have a U1b or something that just fixes this issue in U1. Seems like the right thing to do to fix a bug in a release.