Hello all,
I am at a loss for what has happened to a pool that i've had running for several years without issue. The pool is a 4TB mirror with 1 of the 2 drives presently failed and physically removed (and has been for quite some months).
Anyway, I recently upgraded to TrueNAS from FreeNAS mostly successfully. After the upgrade, both of my pools came up. One of the pools (not the one I am having issues with now) had a disk IO problem that resolved after I replaced a SATA cable.
After a couple of other reboots, I noticed that my 4TB pool was showing offline in the GUI. The only option on the GUI was to Export/Disconnect. Before doing this, I tried importing the pool on the CLI. and can see that it is showing ONLINE (and still is):
However, when I try to import it with options it fails:
After some trying of many things, I exported the pool from the GUI and tried re-adding via CLI with some success:
Then, so I could check if everything came up again, I rebooted and I have not been able to bring back the pool since.
Now, whenever I try to add the pool through the GUI I get the following error:
OR the error "one or more devices is currently unavailable".
And through the CLI:
I think the problem is that zpool import sees the pool as ONLINE whereas it clearly isn't.
I have performed SMART long test and it completed without errors and there are no SMART errors logged, so I am pretty sure the disk is good.
Any ideas?
FYI, I am on the current version of TrueNAS-12.0-U3
I'm off to bed, but will check in the AM...
Thanks in advance.
I am at a loss for what has happened to a pool that i've had running for several years without issue. The pool is a 4TB mirror with 1 of the 2 drives presently failed and physically removed (and has been for quite some months).
Anyway, I recently upgraded to TrueNAS from FreeNAS mostly successfully. After the upgrade, both of my pools came up. One of the pools (not the one I am having issues with now) had a disk IO problem that resolved after I replaced a SATA cable.
After a couple of other reboots, I noticed that my 4TB pool was showing offline in the GUI. The only option on the GUI was to Export/Disconnect. Before doing this, I tried importing the pool on the CLI. and can see that it is showing ONLINE (and still is):
Code:
root@sol:~ # zpool import pool: vol_Data-1 id: 16557953576057239499 state: ONLINE status: One or more devices contains corrupted data. action: The pool can be imported using its name or numeric identifier. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J config: vol_Data-1 ONLINE mirror-0 ONLINE gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78 ONLINE 2827245120054948424 UNAVAIL corrupted data root@sol:~ #
However, when I try to import it with options it fails:
root@sol:~ # zpool import vol_Data-1
cannot import 'vol_Data-1': one or more devices is currently unavailable
root@sol:~ #
After some trying of many things, I exported the pool from the GUI and tried re-adding via CLI with some success:
Code:
oot@sol:~ # zpool import -F vol_Data-1 root@sol:~ # zpool status pool: freenas-boot state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: scrub repaired 0B in 00:03:26 with 0 errors on Thu Apr 29 03:48:26 2021 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 da0p2 ONLINE 0 0 0 errors: No known data errors pool: vol_Data-1 state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-2Q scan: scrub repaired 0B in 21:38:21 with 0 errors on Sun Mar 28 21:38:41 2021 config: NAME STATE READ WRITE CKSUM vol_Data-1 DEGRADED 0 0 0 mirror-0 DEGRADED 0 0 0 gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78 ONLINE 0 0 0 2827245120054948424 UNAVAIL 0 0 0 was /dev/ada1 errors: No known data errors pool: vol_Data-2 state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(5) for details. scan: resilvered 204K in 00:00:01 with 0 errors on Wed Apr 28 22:48:43 2021 config: NAME STATE READ WRITE CKSUM vol_Data-2 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/fd9cff80-60e4-11ea-a188-00247e0a3e78 ONLINE 0 0 0 gptid/fe615845-60e4-11ea-a188-00247e0a3e78 ONLINE 0 0 0 gptid/03261724-60e5-11ea-a188-00247e0a3e78 ONLINE 0 0 0 errors: No known data errors root@sol:~ #
Then, so I could check if everything came up again, I rebooted and I have not been able to bring back the pool since.
Now, whenever I try to add the pool through the GUI I get the following error:
Code:
Error: concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 94, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool self.logger.error( File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 365, in import_pool zfs.import_pool(found, new_name or found.name, options, any_host=any_host) File "libzfs.pyx", line 1095, in libzfs.ZFS.import_pool File "libzfs.pyx", line 1123, in libzfs.ZFS.__import_pool libzfs.ZFSException: permission denied """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 367, in run await self.future File "/usr/local/lib/python3.8/site-packages/middlewared/job.py", line 403, in __run_body rv = await self.method(*([self] + args)) File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 973, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 1411, in import_pool await self.middleware.call('zfs.pool.import_pool', pool['guid'], { File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1238, in call return await self._call( File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1203, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1209, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) libzfs.ZFSException: ('permission denied',)
OR the error "one or more devices is currently unavailable".
And through the CLI:
Code:
oot@sol:~ # zpool import -F vol_Data-1 cannot import 'vol_Data-1': one or more devices is currently unavailable root@sol:~ # zpool import pool: vol_Data-1 id: 16557953576057239499 state: ONLINE status: One or more devices contains corrupted data. action: The pool can be imported using its name or numeric identifier. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J config: vol_Data-1 ONLINE mirror-0 ONLINE gptid/72b29f53-d72b-11e7-98cf-00247e0a3e78 ONLINE 2827245120054948424 UNAVAIL corrupted data root@sol:~ # zpool import vol_Data-1 cannot import 'vol_Data-1': permission denied Destroy and re-create the pool from a backup source. root@sol:~ #
I think the problem is that zpool import sees the pool as ONLINE whereas it clearly isn't.
I have performed SMART long test and it completed without errors and there are no SMART errors logged, so I am pretty sure the disk is good.
Any ideas?
FYI, I am on the current version of TrueNAS-12.0-U3
I'm off to bed, but will check in the AM...
Thanks in advance.