Unable to run alert source 'HasUpdate‘

[bigdadda06]

So today I’ve been getting a few of these alerts. Anyone know what they mean?
The boot pool is currently doing a resilver? Could the be the cause?

Running 11.2 on a i3 with 8GB RAM, it was running fine with no alerts until I added another USB drive to the boot pool, so that is probably it.
It seems to be taking a long time to resolver the 16GB drive however.

Code:

New alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
   return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
   execute = self.locked_retry(Database.Cursor.execute, query, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
   raise e
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
   rv = method(self, *args, **kwargs)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
   alerts = (await alert_source.check()) or []
File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
   return await self.middleware.run_in_thread(self.check_sync)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 25, in check_sync
   self.middleware.call_sync("datastore.query", "system.update", None, {"get": True})
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
   return fut.result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
   return self.__get_result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
   raise self._exception
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
   return await run_method(methodobj, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
   return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
   select=options.get('select'),
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize
   for i in qs:
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__
   self._fetch_all()
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
   self._result_cache = list(self.iterator())
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__
   results = compiler.execute_sql()
File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
   cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
   return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
   six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
   raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
   return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
   execute = self.locked_retry(Database.Cursor.execute, query, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
   raise e
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
   rv = method(self, *args, **kwargs)
django.db.utils.OperationalError: database is locked


Alerts:
* Unable to run alert source 'HasUpdate'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
   return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
   execute = self.locked_retry(Database.Cursor.execute, query, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
   raise e
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
   rv = method(self, *args, **kwargs)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
   alerts = (await alert_source.check()) or []
File "/usr/local/lib/python3.6/site-packages/middlewared/alert/base.py", line 96, in check
   return await self.middleware.run_in_thread(self.check_sync)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/update.py", line 25, in check_sync
   self.middleware.call_sync("datastore.query", "system.update", None, {"get": True})
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1139, in call_sync
   return fut.result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 425, in result
   return self.__get_result()
File "/usr/local/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
   raise self._exception
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
   return await run_method(methodobj, *args)
File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
   return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
   result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
   return f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
   select=options.get('select'),
File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize
   for i in qs:
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__
   self._fetch_all()
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
   self._result_cache = list(self.iterator())
File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__
   results = compiler.execute_sql()
File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
   cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
   return self.cursor.execute(sql, params)
File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
   six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
   raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
   return self.cursor.execute(sql, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
   execute = self.locked_retry(Database.Cursor.execute, query, params)
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
   raise e
File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
   rv = method(self, *args, **kwargs)
django.db.utils.OperationalError: database is locked

 

[Inxsible]

I have similar issues over last month as well.

I have seen alerts related to 'HasUpdate', 'Samba4', 'ZPoolCapacity', 'smartd'

The Samba4 and smartd alerts --- I get a "Gone alert" email within 1 minute -- indicating that the alert has been resolved.
ZPoolCapacity -- I get a "Gone alert" email in 5 minutes
HasUpdate -- I get a "Gone alert" email in 1 hour


What is causing these alerts? How do they fix themselves?

How can I stop having these alerts--- Do I have to reschedule the timings of my scrubs etc?

 

[Chris Moore]

Please put long text listings like this inside code tags like this:

that way they come out looking much more readable.

 

[bigdadda06]

I have similar issues over last month as well.
I have seen alerts related to 'HasUpdate', 'Samba4', 'ZPoolCapacity', 'smartd'
The Samba4 and smartd alerts --- I get a "Gone alert" email within 1 minute -- indicating that the alert has been resolved.
ZPoolCapacity -- I get a "Gone alert" email in 5 minutes
HasUpdate -- I get a "Gone alert" email in 1 hour

What is causing these alerts? How do they fix themselves?
How can I stop having these alerts--- Do I have to reschedule the timings of my scrubs etc?

I think I have seen a couple of different versions as well. i.e. for HasUpdate as well as Replication, and I usually get a "Alert Gone" message as well.

My resolver has stopped now, but I think I'm still getting the messages.

 

[dlavigne]

It looks like the most likely culprit is slow USB boot devices: https://redmine.ixsystems.com/issues/69159. U2 will add a timeout to help minimize those alerts, a better solution is replace slow boot devices with better ones.

 

[bigdadda06]

It looks like the most likely culprit is slow USB boot devices: https://redmine.ixsystems.com/issues/69159. U2 will add a timeout to help minimize those alerts, a better solution is replace slow boot devices with better ones.

That appears to be correct, I removed the "new" USB stick and the alerts appear to have stopped.
I say "new" becuase I was using the same USB stick with 11.1 U6.3 without error.
Or maybe I should say without error that it was telling me about???

 

[Inxsible]

That's interesting. I didn't change the config of my setup at all and upgrading seems to have caused the start of these Alerts.

Unfortunately, in this rig, I don't have any spare sata ports to get rid of my USB :( So I might have to just live with it until the upgrade to U2 -- as @dlavigne suggests