Unable to view or mount storage pool after power outage

[marcosaluzzo]

FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz
8043MB

OS on 60GB SSD
An encrypted volume volume1 with Raidz1 on 3x 1GB Seagate barracuda disks
The system has been running since 2014. No upgrades or changes have been made to the OS. I have a backup of the configuration made in December 2019, the GELI key and GELI recovery key.

There was a power outage this morning. When the power came back on I booted the freenas server and waited to connect to the GUI. I was unable to connect to the GUI so I connected a monitor and I saw the following info on the screen:

F1 FreeBSD
F2 FreeBSD
F5 Drive 1
F6 PXE

F2 ###########################################################################

with the screen filling up with more hashes. Pressing F2 made more hashes appear more quickly. Pressing F1 or F5 gave an error message which I can't remember, but some about it not being a bootable drive. After several reboot attempts and went into the bios checked that the configuration was correct, saved and rebooted. This time F1 was selected and the OS booted normally.

I entered the GUI and went to to storage to unencrypt the volume but there wasn't any option to unencrypt the volume. So I clicked on Auto Import Volume by selecting the disks, using the GELI.key and passphrase. It then shows me the volume1 to import with ZFS id. However is stays that "You already have a volume with same name".

So after searching on the forum and internet I ran the following commands:

zfs mount - nothing returned

zfs list
no datasets found

zpool list
no pools available

zpool status
no pools available

zpool import
pool: volume1
id: 16006840839950837346
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:

volume1 ONLINE
raidz1-0 ONLINE
gptid/00bbdf18-e7ce-11e3-8682-e03f4949e218.eli ONLINE
gptid/014576cb-e7ce-11e3-8682-e03f4949e218.eli ONLINE
gptid/01cb7380-e7ce-11e3-8682-e03f4949e218.eli ONLINE

I checked to see what there is the mount folder and there is the following:

/mnt# ls -la
total 2
drwxr-xr-x 3 root wheel 512 Apr 23 2014 ./
drwxr-xr-x 21 root wheel 512 Apr 23 2014 ../
drwxrwxr-x 2 root operator 512 Mar 23 13:52 .snap/
-rw-r--r-- 1 root wheel 5 Apr 23 2014 md_size

I have found similar issues but they were all related to upgrades. So before I attempted any else I want to ask the community their assistance. What are your suggestions for getting the pool working?

Thanks

 

[sretalla]

How does it look with zfs mount -a ?

Also share the output from zfs list

EDIT: Actually I see that the pool is listed under zpool import, so you won't see anything with those commands.

You should try to import it to change the name... zpool import volume1 volume1a

Then export it zpool export volume1a.

Then try again to import it from the GUI

If that works, you can repeat the process in reverse to export, re-import at CLI with the original name, then export and impot from GUI.

 

[marcosaluzzo]

Thanks for your reply.
[root@darksaber] ~# zpool import volume1 volume1a cannot mount '/volume1a': failed to create mountpoint cannot mount '/volume1a/.system': failed to create mountpoint cannot mount '/volume1a/.system/cores': failed to create mountpoint cannot mount '/volume1a/.system/samba4': failed to create mountpoint cannot mount '/volume1a/.system/syslog': failed to create mountpoint cannot mount '/volume1a/database': failed to create mountpoint cannot mount '/volume1a/films': failed to create mountpoint cannot mount '/volume1a/jails': failed to create mountpoint cannot mount '/volume1a/music': failed to create mountpoint cannot mount '/volume1a/photos': failed to create mountpoint cannot mount '/volume1a/private': failed to create mountpoint cannot mount '/volume1a/work': failed to create mountpoint cannot mount '/volume1a/work/deanima': failed to create mountpoint cannot mount '/volume1a/work/englishpower': failed to create mountpoint

I then ran zpool export volume1a

I then went to the GUI and ran autoimport. After choosing the disks, geli.key and passphrase I had the option to import volume1a. I clicked okay and then it gave the follow error:

Request Method:	POST
Request URL:	http://192.168.2.102/storage/auto-import/
Software Version:	FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
Exception Type:	IntegrityError
Exception Value:	column encrypted_provider is not unique
Exception Location:	/usr/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py in execute, line 450
Server time:	Mon, 23 Mar 2020 19:59:25 +0100

Traceback
Environment:

Software Version: FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
Request Method: POST
Request URL: http://192.168.2.102/storage/auto-import/


Traceback:
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
107. response = middleware_method(request, callback, callback_args, callback_kwargs)
File "/usr/local/www/freenasUI/../freenasUI/freeadmin/middleware.py" in process_view
158. return login_required(view_func)(request, *view_args, **view_kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
22. return view_func(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
69. return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/formtools/wizard/views.py" in dispatch
236. response = super(WizardView, self).dispatch(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
87. return handler(request, *args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/formtools/wizard/views.py" in post
297. return self.render_done(form, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/contrib/formtools/wizard/views.py" in render_done
350. done_response = self.done(final_form_list, **kwargs)
File "/usr/local/www/freenasUI/../freenasUI/storage/forms.py" in done
1051. ed.save()
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py" in save
545. force_update=force_update, update_fields=update_fields)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
573. updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
654. result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py" in _do_insert
687. using=using, raw=raw)
File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py" in _insert
232. return insert_query(self.model, objs, fields, **kwargs)
File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py" in insert_query
1511. return query.get_compiler(using=using).execute_sql(return_id)
File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
898. cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py" in execute
53. return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/utils.py" in __exit__
99. six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/util.py" in execute
53. return self.cursor.execute(sql, params)
File "/usr/local/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py" in execute
450. return Database.Cursor.execute(self, query, params)

Exception Type: IntegrityError at /storage/auto-import/
Exception Value: column encrypted_provider is not unique

Request information
GET

No GET data
POST

Variable	Value
2-__all__	u''
2-volume_disks	u'volume1a|16006840839950837346'
__form_id	u'dijit_form_Form_3'
auto_import_wizard-current_step	u'2'

FILES

No FILES data
COOKIES

Variable	Value
csrftoken	'y2ZDjqyQybflYlRQCwgJm1nAsjDc5H3g'
sessionid	'47os5eji6zkm014izs0t4tlrp3c8s4jj'
fntreeSaveStateCookie	'root%2Croot%2F15%2Croot%2F42%2Croot%2F42%2F44%2Croot%2F58%2Croot%2F104%2Croot%2F58%2F66%2Croot%2F1%2F9%2Croot%2F15%2F36%2Croot%2F15%2F31%2Croot%2F148%2Croot%2F15%2F19%2Croot%2F43%2Croot%2F43%2F45'

META

Variable	Value
wsgi.multiprocess	False
HTTP_REFERER	'http://192.168.2.102/'
REDIRECT_STATUS	'200'
SERVER_SOFTWARE	'nginx/1.4.4'
SCRIPT_NAME	u''
HTTP_X_CSRFTOKEN	'y2ZDjqyQybflYlRQCwgJm1nAsjDc5H3g'
REQUEST_METHOD	'POST'
PATH_INFO	u'/storage/auto-import/'
HTTP_ORIGIN	'http://192.168.2.102'
SERVER_PROTOCOL	'HTTP/1.1'
QUERY_STRING	''
CONTENT_LENGTH	'119'
HTTP_USER_AGENT	'Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0'
HTTP_CONNECTION	'keep-alive'
HTTP_COOKIE	'csrftoken=y2ZDjqyQybflYlRQCwgJm1nAsjDc5H3g; sessionid=47os5eji6zkm014izs0t4tlrp3c8s4jj; fntreeSaveStateCookie=root%2Croot%2F15%2Croot%2F42%2Croot%2F42%2F44%2Croot%2F58%2Croot%2F104%2Croot%2F58%2F66%2Croot%2F1%2F9%2Croot%2F15%2F36%2Croot%2F15%2F31%2Croot%2F148%2Croot%2F15%2F19%2Croot%2F43%2Croot%2F43%2F45'
SERVER_NAME	'localhost'
REMOTE_PORT	'48210'
wsgi.url_scheme	'http'
SERVER_PORT	'80'
SERVER_ADDR	'192.168.2.102'
DOCUMENT_ROOT	'/usr/local/etc/nginx/html'
HTTP_CONTENT_LENGTH	'119'
HTTP_X_REQUESTED_WITH	'XMLHttpRequest'
DOCUMENT_URI	'/storage/auto-import/'
wsgi.input	<flup.server.fcgi_base.InputStream object at 0x8121b1890>
HTTP_HOST	'192.168.2.102'
wsgi.multithread	True
HTTP_CONTENT_TYPE	'application/x-www-form-urlencoded'
REQUEST_URI	'/storage/auto-import/'
HTTP_ACCEPT	'*/*'
wsgi.version	(1, 0)
GATEWAY_INTERFACE	'CGI/1.1'
wsgi.run_once	False
wsgi.errors	<flup.server.fcgi_base.TeeOutputStream object at 0x811532510>
REMOTE_ADDR	'192.168.2.100'
HTTP_ACCEPT_LANGUAGE	'en-GB,en;q=0.7,it;q=0.3'
CONTENT_TYPE	'application/x-www-form-urlencoded'
CSRF_COOKIE	u'y2ZDjqyQybflYlRQCwgJm1nAsjDc5H3g'
HTTP_ACCEPT_ENCODING	'gzip, deflate'

I've not done anything else at this point. What are the suggestions?

 

[marcosaluzzo]

However after the failed autoimport via the GUI the follow commands give:

Code:

zfs list
NAME                         USED  AVAIL  REFER  MOUNTPOINT
volume1a                     850G   976G   277K  /mnt/volume1a
volume1a/.system            17.9M   976G   224K  /mnt/volume1a/.system
volume1a/.system/cores       192K   976G   192K  /mnt/volume1a/.system/cores
volume1a/.system/samba4     3.94M   976G  3.94M  /mnt/volume1a/.system/samba4
volume1a/.system/syslog     13.6M   976G  13.6M  /mnt/volume1a/.system/syslog
volume1a/database           7.91G   976G  7.91G  /mnt/volume1a/database
volume1a/films               275G   976G   275G  /mnt/volume1a/films
volume1a/jails               192K   976G   192K  /mnt/volume1a/jails
volume1a/music               176G   976G   176G  /mnt/volume1a/music
volume1a/photos             50.9G   976G  50.9G  /mnt/volume1a/photos
volume1a/private            97.9G   976G  97.9G  /mnt/volume1a/private
volume1a/work                241G   976G   352K  /mnt/volume1a/work
volume1a/work/deanima        192G   976G   192G  /mnt/volume1a/work/deanima
volume1a/work/englishpower  48.6G   976G  46.9G  /mnt/volume1a/work/englishpower

Code:

zpool list
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
volume1a  2.72T  1.25T  1.47T    45%  1.00x  ONLINE  /mnt

Code:

zpool status
  pool: volume1a
 state: ONLINE
  scan: resilvered 17.2G in 0h16m with 0 errors on Mon Mar 23 20:07:17 2020
config:

    NAME                                                STATE     READ WRITE CKSUM
    volume1a                                            ONLINE       0     0     0
      raidz1-0                                          ONLINE       0     0     0
        gptid/00bbdf18-e7ce-11e3-8682-e03f4949e218.eli  ONLINE       0     0     0
        gptid/014576cb-e7ce-11e3-8682-e03f4949e218.eli  ONLINE       0     0     0
        gptid/01cb7380-e7ce-11e3-8682-e03f4949e218.eli  ONLINE       0     0     0

errors: No known data errors

So everything looks online, however nothing is showing up in the GUI. Under active volumes I don't see anything, as well as nothing under "view disks".

What would be advisable now?

 

[sretalla]

OK, so the pool is mounted, just not "owned by the GUI", so not displayed in the GUI.

It seems the leftovers from volume1 make it impossible to bring the same pool back into play.

There's a good chance that this will be helped by first exporting volume1a (zpool export volume1a)

Let's see if we can try a few things to move it along:
First, let's try zpool clear volume1
Then zpool status -v volume1
If neither of those produces any output, then try zpool export --discard volume1

 

[marcosaluzzo]

Thanks for the reply. The volume1a is exported. Running the the command zpool clear volume1 and zpool status -v volume1 both returned "cannot open 'volume1': no such pool"

Running zpool export --discard volume1 returned invalid option.

Did you mean zpool export -f volume1?

 

[sretalla]

You can try the -f... if that doesn't help, we're going to be talking about editing the config database or starting with a fresh config... not good either way depending on how complex your setup is.

 

[marcosaluzzo]

-f didn't do anything. :( The goods news is the config is not so complicated. I also have a backup of the configure from December 2019. Nothing has changed since then.

 

[marcosaluzzo]

In the end I reset to factory settings, re-imported the volume and reconfigured manually the users, groups and shares. The shares are working again!! However the GUI interface is still not working properly. i.e. the right window doesn't display information about interface, share, storage or disks. The windows are empty except for the configuration buttons.

I'm going to take this opportunity to update the server to a more recent version of freenas. Thank you for your input and I'lll let you know how I get on.

 

[marcosaluzzo]

First I did a upgrade install to Freenas 9.3. I had exactly the same problem as after the power outage. I was unable to import the pool volume1a because there was already a pool with the same name. Again the GUI was not showing the disks or showing the volume in the volume window. So I thought at this point I'd do a clean install of the new version of FreeNAS-9.3-STABLE-201412090314. After I enter the GUI I canceled the Wizard and imported the volume through the GUI (I've not upgrade the pool yet). However even if I have done a clean install in the GUI storage tab under volumes it does display the volume and under the "view disks" it does not display the the view disks. If I go to add a Smart test for example it shows the disks, or when I do an volume import it sees the disks and let's me select them.

So at this point I am perplexed as to why it is not possible to display the disk or volume information after the it has worked for years without any problems on exactly the same hardware.

Does anyone have any ideas?