SOLVED Can't put disk offline

[avalon60]

I have a pool of just 1 disk for cctv , which has been in use for about 7 years now. It is now reported as being Unhealthy in the list of pools, and it i unreadable. I can't access any data on it now since 2 days ago.

After reading up on how to replace the disk with the same size, I found I can't put it offline.
I get this when trying to do so:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 232, in __zfs_vdev_operation
op(target, *args)
File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 232, in __zfs_vdev_operation
op(target, *args)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 248, in <lambda>
self.__zfs_vdev_operation(name, label, lambda target: target.offline())
File "libzfs.pyx", line 2203, in libzfs.ZFSVdev.offline
libzfs.ZFSException: pool I/O is currently suspended

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 248, in offline
self.__zfs_vdev_operation(name, label, lambda target: target.offline())
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 234, in __zfs_vdev_operation
raise CallError(str(e), e.code)
middlewared.service_exception.CallError: [EZFS_POOLUNAVAIL] pool I/O is currently suspended
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 139, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1236, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 1146, in offline
await self.middleware.call('zfs.pool.offline', pool['name'], found[1]['guid'])
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1279, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1244, in _call
return await self._call_worker(name, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1250, in _call_worker

Help please
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1169, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [EZFS_POOLUNAVAIL] pool I/O is currently suspended

This is the ouput of zpool ststus for the said disk

Code:

        NAME                                          STATE     READ WRITE CKSUM
        CCTV                                          ONLINE       0     0     0
          gptid/d647d843-4792-11e6-a4b6-0cc47a0c1eef  ONLINE       4   284     0
errors: List of errors unavailable: pool I/O is currently suspended
       

I have tried Scrub Pool, but get this error:

Scrub Pool​

Error: [EFAULT] concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 303, in scrub_action pool.stop_scrub() File "libzfs.pyx", line 402, in libzfs.ZFS.__exit__ File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 301, in scrub_action pool.start_scrub() File "libzfs.pyx", line 2967, in libzfs.ZFSPool.start_scrub libzfs.ZFSException: pool I/O is currently suspended During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.9/concurrent/futures/process.py", line 246, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 111, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 979, in nf return f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/zfs.py", line 305, in scrub_action raise CallError(str(e), e.code) middlewared.service_exception.CallError: [EZFS_POOLUNAVAIL] pool I/O is currently suspended """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 355, in run await self.future File "/usr/local/lib/python3.9/site-packages/middlewared/job.py", line 391, in __run_body rv = await self.method(*([self] + args)) File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf return await f(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/pool.py", line 4203, in scrub await self.middleware.call('zfs.pool.scrub_action', name, action) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1279, in call return await self._call( File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1244, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1250, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1169, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1152, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) middlewared.service_exception.CallError: [EZFS_POOLUNAVAIL] pool I/O is currently suspended

 

[danb35]

libzfs.ZFSException: pool I/O is currently suspended

The disk is sufficiently damaged that ZFS has stopped the pool. Rebooting might bring it back into a (temporarily) usable state. But since it's a single-disk pool, you won't be able to offline that disk. You should be able to replace it, though.

 

[avalon60]

Well amazingly, the disk came back online after a reboot and is now readable.

zpool status:

Code:

errors: No known data errors

  pool: CCTV
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
 scan: scrub repaired 0B in 00:04:46 with 0 errors on Sat May 27 08:53:11 2023
config:
        NAME                                          STATE   READ WRITE CKSUM
        CCTV                                          ONLINE     0     0     0

Should I still replace the disk now or could I wait a bit longer.

Thanks

 

[Dice]

Should I still replace the disk now or could I wait a bit longer.

Should have been replaced a while ago.

 

[Davvo]

You have to actually run a smart long test in order to know, but you probably should replace it. Be aware that if you replace it you will lose all data since it's a single-disk pool

 

[danb35]

Be aware that if you replace it you will lose all data since it's a single-disk pool

Nonsense. Replacing the disk will resilver the data onto the replacement, assuming it can read the data.

 

[Davvo]

Nonsense. Replacing the disk will resilver the data onto the replacement, assuming it can read the data.

My bad, completely missed the data being safe. Sorry

 

[danb35]

Well amazingly, the disk came back online after a reboot and is now readable.

Not amazing at all. Pool errors are generally cleared at reboot, so it's to be expected that it would come back up on reboot. That doesn't mean the disk or your data are OK, just that ZFS doesn't currently know of any errors. A scrub would likely find them, and SMART data would show if the disk itself were reporting errors. But if it's as bad as it sounded like in your OP, it's still likely (past) time to replace the disk.

 

[avalon60]

I still can't put the said disk Offline, as I am getting the same error message as before. So should I just take it out and put the new in in, and then add the datasets etc

 

[Davvo]

I still can't put the said disk Offline, as I am getting the same error message as before. So should I just take it out and put the new in in, and then add the datasets etc

You need to replace it, not offline it.
Add the other disk into the system and execute the replace operation.

 

[danb35]

I still can't put the said disk Offline

No, of course not--you can't offline a single-disk pool, as I said above. But you can replace the disk--install the new disk, go to the pool status page, do the replacement through the GUI, and let it finish. Once it completes, remove the old disk.

 

[avalon60]

Thanks for the help and information, and the new disk is installed and running fine after resilvering