FreeNAS sent strange alerts, went unresponsive, and now won't boot

[DarkClaw]

Hey everyone, I'm a new FreeNAS user who's having issues and can't seem to resolve them through Googling. Please hear me out, but also feel free to chastise me for everything I'm doing wrong so I'll learn. I'll also preface this with the fact that I'm running my FreeNAS off of a USB drive, as many people recommended.

So I recently setup an older computer that had a decent enough amount of RAM with a few NAS drives and FreeNAS. So far, there were few issues in installation and everything was as I wanted functionality-wise (though I didn't realize that one could and should also save a backup of the config until after I suffered this issue... oops). As I was on my way home today, I got an e-mail with the alert "Unable to run alert source 'Samba4'", with some mentions to a database being locked. I can post the full error, if necessary; however, I got an e-mail only a minute or so later saying that the alert had gone. I then got a third e-mail with a new alert: "The boot volume state is ONLINE: One or more devices are faulted in response to IO failures." I figured I'd login to the FreeNAS GUI when I got home to check for anything stange.

When I got to my PC and checked the web GUI for anything out of place, as well as Googling as to why one might be receiving such errors, I noticed that the web GUI stopped responding to any input altogether. I only managed to get to a few pages before this happened, such as the Pools page (which, gladly enough, stated my RAID1 pool was healthy). After an hour wait with an unresponsive GUI (though I could still access the Windows share), I figured it was time to try a reboot... and I got the error in the attached picture. While I'm not (yet) panicking since I know my pool with all my data reported being healthy and that FreeNAS can import pools if I happen to need to do a fresh install, I would like to know why this might have happened and maybe even how to fix this if possible.

If you'd like more details, please feel free to ask. Thanks.

Edit - Just read the introductory message one gets from signing up, and realized I've already broke some of the guidelines... I'll post the full error I got from the first e-mail and also my hardware config:

Code:

* Unable to run alert source 'Samba4'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
    execute = self.locked_retry(Database.Cursor.execute, query, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
    raise e
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
    alerts = (await alert_source.check()) or []
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/samba4.py", line 11, in check
    if not await self.middleware.call("datastore.query", "storage.volume"):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
    return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
    select=options.get('select'),
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize
    for i in qs:
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
    execute = self.locked_retry(Database.Cursor.execute, query, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
    raise e
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
django.db.utils.OperationalError: database is locked


Alerts:
* Unable to run alert source 'Samba4'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
    execute = self.locked_retry(Database.Cursor.execute, query, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
    raise e
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/alert.py", line 358, in __run_source
    alerts = (await alert_source.check()) or []
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/../alert/source/samba4.py", line 11, in check
    if not await self.middleware.call("datastore.query", "storage.volume"):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1113, in call
    return await self._call(name, serviceobj, methodobj, params, app=app, pipes=pipes, io_thread=True)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1063, in _call
    return await run_method(methodobj, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1009, in run_in_thread
    return await self.loop.run_in_executor(executor, functools.partial(method, *args, **kwargs))
  File "/usr/local/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 175, in query
    select=options.get('select'),
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/datastore.py", line 88, in __queryset_serialize
    for i in qs:
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 256, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 1087, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/usr/local/lib/python3.6/site-packages/django/db/models/query.py", line 54, in __iter__
    results = compiler.execute_sql()
  File "/usr/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 835, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/usr/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 412, in execute
    execute = self.locked_retry(Database.Cursor.execute, query, params)
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 403, in locked_retry
    raise e
  File "/usr/local/www/freenasUI/freeadmin/sqlite3_ha/base.py", line 389, in locked_retry
    rv = method(self, *args, **kwargs)
django.db.utils.OperationalError: database is locked

Hardware Configuration:
Intel Core i7-920
Gigabyte GA-EX58-UD5
6x 2GB DDR3 1066
2x 4TB WD Red HDDs (RAID1, plugged into SATA ports that go to ICHR10, not the JMB322 controller)
1x Verbatim 32GB USB drive (plugged into motherboard USB port, not front panel header)

 

[Chris Moore]

1x Verbatim 32GB USB drive (plugged into motherboard USB port, not front panel header)

This is the most likely cause of the problem. I had some of those drives. They don't hold up to the constant workload of the FreeNAS boot pool. The one I used didn't last nine months before it died.
You probably need to reinstall on a new device and reload your config.db from backup.

 

[Chris Moore]

I am not sure if it is in here, but you might want to have a look:

FreeNAS® Quick Hardware Guide
https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

This is the kind of module that has been most reliable for me:

https://www.digikey.com/product-detail/en/swissbit/SFUI016GJ1AB1TO-I-QT-211-STD/1052-1264-ND/6553848

I have some of these in the 16GB variety that have been going for over five years now with no problems at all.

 

[DarkClaw]

This is the most likely cause of the problem. I had some of those drives. They don't hold up to the constant workload of the FreeNAS boot pool. The one I used didn't last nine months before it died.
You probably need to reinstall on a new device and reload your config.db from backup.

I am not sure if it is in here, but you might want to have a look:

FreeNAS® Quick Hardware Guide
https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

This is the kind of module that has been most reliable for me:

https://www.digikey.com/product-detail/en/swissbit/SFUI016GJ1AB1TO-I-QT-211-STD/1052-1264-ND/6553848

I have some of these in the 16GB variety that have been going for over five years now with no problems at all.

It doesn't seem like there's any specific USB key recommendations in either hardware guide, but the full guide recommends something that can indeed handle the strain of the random writes that can occur in some situations. It even mentions mirroring, which I didn't even know you could do with USB drives, but it makes sense. Your option seems a lot slicker, but I don't have that kind of money right now unfortunately. I may end up using a SATA port and a spare SSD for the time being, since I have one kicking around, as it could probably handle the throughput much better; it's just a shame to lose all that extra space as well as the port that might get used in the future.

Unfortunately, I learned a little too late that you could backup all the settings as a config file. Luckily, I was mainly using it as a file server for the time being, so there won't be much to re-setup, but I'll be sure to make myself a copy this time.

I appreciate the professional opinion, thanks a bunch.