11.3-2 -> 12Beta upgrade sqlite issues

[eirikz]

Hi.

After upgrading to the beta I seem to have sqlite issues, previous threads suggests this is related to failing storage, however I run the OS on a pair of SAS drives mirrored, doesn't seem to be the issue.

Opening the sqlite the file appears fine

Code:

mirai% sqlite3 freenas-v1.db 

but whenever I try to do changes to config, like tunables or acl settings of a smb share I get the following

Code:

  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/datastore/connection.py", line 62, in execute_write
    return await self.middleware.run_in_executor(self.thread_pool, self._execute_write, sql, binds)
  File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1102, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/datastore/connection.py", line 65, in _execute_write
    result = self.connection.execute(sql, binds)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 976, in execute
    return self._execute_text(object_, multiparams, params)
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1145, in _execute_text
    ret = self._execute_context(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1287, in _execute_context
    self._handle_dbapi_exception(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _handle_dbapi_exception
    util.raise_(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
    raise exception
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1247, in _execute_context
    self.dialect.do_execute(
  File "/usr/local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 590, in do_execute
    cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) unable to open database file
[SQL: INSERT INTO system_tunable (tun_value, tun_type, tun_comment, tun_enabled, tun_var) VALUES (?, ?, ?, ?, ?)]
[parameters: ('YES', 'loader', '', 1, 'fuse_load')]
(Background on this error at: http://sqlalche.me/e/e3q8)

Code:

(sqlite3.OperationalError) unable to open database file [SQL: INSERT INTO system_tunable (tun_value, tun_type, tun_comment, tun_enabled, tun_var) VALUES (?, ?, ?, ?, ?)] [parameters: ('YES', 'loader', '', 1, 'fuse_load')] (Background on this error at: http://sqlalche.me/e/e3q8)

I have logged a jira, but didn't get picked up by anyone yet

https://jira.ixsystems.com/browse/NAS-106644?filter=-2

 

[Yorick]

previous threads suggests this is related to failing storage

Usually, yes. What do you get when you run a scrub on the boot pool?

 

[eirikz]

Usually, yes. What do you get when you run a scrub on the boot pool?

Code:

  pool: freenas-boot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 0 days 00:15:17 with 0 errors on Fri Jul  3 23:53:33 2020
config:

    NAME                                            STATE     READ WRITE CKSUM
    freenas-boot                                    ONLINE       0     0     0
      mirror-0                                      ONLINE       0     0     0
        gptid/fdeef42e-d7cd-11e9-a652-782bcb47b119  ONLINE       0     0     0
        gptid/b08c5079-d7cf-11e9-a652-782bcb47b119  ONLINE       0     0     0

errors: No known data errors

 

[eirikz]

A second reboot appears to have solved it, so something hit the wall after the first reboot after upgrade, some scripts that might need to be added if there is some dB upgrade happening? Or maybe run a less error prone dB perhaps.