9.3: possible to re-run database upgrade?

[heupink]

Hi all,

We have updated to 9.3 today, and after the reboot, the system seemed stuck a bit after the 'upgrading database' bit. We waited a while, and then, expecting the machine to be frozen, I pressed the powerdown button, and that's what it did: it powered down.

Then, second boot attempt went a lot faster, even the gui loaded, but gives all kinds of errors, like unable to load /api/v1.0/storage/task/ status:500, and: OperationalError, Exception Value:no such column: jails_jailsconfiguration.jc_ipv4_dhcp

So I guess the database upgrade is not complete.

Stupid enough I did NOT create a configuration backup before the upgrade. Yes I know, stupid stupid.

So, is there a way to RETRY/RERUN the database upgrade?

 

[heupink]

Update.failed contains:
[root@nas] /data# cat update.failed
FATAL ERROR - The following SQL query failed: CREATE TABLE "_south_new_services_afp" ("afp_srv_homedir_enable" bool NOT NULL, "afp_srv_homedir" varchar(255), "afp_srv_global_aux" text NOT NULL, "afp_srv_guest_user" varchar(120), "afp_srv_dbpath" varchar(255), "afp_srv_connections_limit" integer NOT NULL DEFAULT 50, "afp_srv_bindip" varchar(255) NOT NULL, "afp_srv_guest" bool, "id" integer PRIMARY KEY)
The error was: table "_south_new_services_afp" already exists
Running migrations for api:
- Nothing to migrate.
- Loading initial data for api.
Installed 0 object(s) from 0 fixture(s)
Running migrations for freeadmin:
- Nothing to migrate.
- Loading initial data for freeadmin.
Installed 0 object(s) from 0 fixture(s)
Running migrations for support:
- Nothing to migrate.
- Loading initial data for support.
Installed 0 object(s) from 0 fixture(s)
Running migrations for sharing:
- Nothing to migrate.
- Loading initial data for sharing.
Installed 0 object(s) from 0 fixture(s)
Running migrations for directoryservice:
- Migrating forwards to 0040_auto__del_field_ldap_ldap_use_default_domain.
> services:0149_auto__add_field_afp_afp_srv_bindip
! Error found during real run of migration! Aborting.

! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.

! You *might* be able to recover with:
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS (one that supports DDL transactions)
! NOTE: The error which caused the migration to fail is further up.
Error in migration: services:0149_auto__add_field_afp_afp_srv_bindip
Traceback (most recent call last):
File "/usr/local/www/freenasUI/manage.py", line 42, in <module>
execute_from_command_line(sys.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
utility.execute()
File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 392, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 242, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 285, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.7/site-packages/south/management/commands/migrate.py", line 111, in handle
ignore_ghosts = ignore_ghosts,
File "/usr/local/lib/python2.7/site-packages/south/migration/__init__.py", line 220, in migrate_app
success = migrator.migrate_many(target, workplan, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 256, in migrate_many
result = migrator.__class__.migrate_many(migrator, target, migrations, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 331, in migrate_many
result = self.migrate(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 133, in migrate
result = self.run(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 114, in run
return self.run_migration(migration, database)
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 84, in run_migration
migration_function()
File "/usr/local/lib/python2.7/site-packages/south/migration/migrators.py", line 60, in <lambda>
return (lambda: direction(orm))
File "/usr/local/www/freenasUI/../freenasUI/services/migrations/0149_auto__add_field_afp_afp_srv_bindip.py", line 14, in forwards
keep_default=False)
File "/usr/local/lib/python2.7/site-packages/south/db/sqlite3.py", line 38, in add_column
field.column: (self._column_sql_for_create(table_name, name, field, False), field_default)
File "/usr/local/lib/python2.7/site-packages/south/db/generic.py", line 47, in _cache_clear
return func(self, table, *args, **opts)
File "/usr/local/lib/python2.7/site-packages/south/db/sqlite3.py", line 110, in _remake_table
", ".join(["%s %s" % (self.quote_name(cname), ctype) for cname, ctype in definitions.items()]),
File "/usr/local/lib/python2.7/site-packages/south/db/generic.py", line 282, in execute
cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py", line 99, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py", line 53, in execute
return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 452, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: table "_south_new_services_afp" already exists
[root@nas] /data#

 

[fracai]

I'd just try re-importing the config to your current 9.3 from backup.

 

[dlavigne]

It also worth creating a bug report at bugs.freenas.org indicating the version you upgraded from and that error. If you make one, post the issue number here.

 

[ikke]

I recall doing upgrade manually long ago. At the time I went manually through the upgrade process, reading automatic update source and fixing database schema manually as I saw errors. Likely you can still do it. I need to do it too, as the same error occurs to me too. But off to work first, let's see when I get to continue.

Please post here if you find the schema update script first.

 

[cyberjock]

I doubt a reimport or reupgrade would work. The problem is you probably rebooted while the config file was mid-write and you probably have a corrupted config file. So now you have no good config file to go on. I know when I did an upgrade to 9.3 it seemed to stop for like 5 minutes on upgrading, but it did eventually complete. No doubt if you have a slower boot device than mine it could take even longer.

In any case I don't have any advice. If you want to do the upgrade again all you have to do is go to the CLI and do:

cd /data
touch need-update
shutdown -r now

It should attempt to import your config file again.

 

[ikke]

Update from my case, it works now. I just didn't have enough faith. It took somewhere from 40 to 50 minutes to do the update database schema -part. It just sit quietly doing it, no progress prompts, like stuck.

 

[heupink]

Yep, I was too unpatient, and simply should have waited longer, probably. In the end, I created a new config from start, and since then, freenas has been running great. Love the new update/boot mechanisms!

Thanks for the help.