In the WebUI, Storage ; Snapshots. You have all your snapshots there and this is the place to delete it. The pool being full, the process may fail but give it a try.
I was able to use the WebGUI to delete the snapshot, so one small(ish) victory.
Good. That way, you can extract whatever valuable data you have in it. You may also search for whatever filled that partition but to recover your data is by far the most important.
You may have to do that backup manually then...
This is a serious problem. I tried running ROBOCOPY again, but all it was able to do was re-create the directories on the backup storage. Content copying is not really happening.
I also tried a simple copy using FILE EXPLORER, but again, another no go.
So Yes, you can try it :
--Power off the Windows iSCSI client (to be sure it will not keep filling the zvol until a hard limit protects the pool. Dealing with a critical situation, better be safe than sorry)
--Add the vDev in your pool to extend it (hope ZFS will be able to do that despite the pool being 100% but it may very well be)
--Configure the zVol to cap its max size below the new maximum (cap it to 15T if your new max is 16T) (to protect the pool against being loaded to 100% again)
--Boot up the Windows iSCSI
--Try to extend that partition but remain below the new zvol's limit (extend it to 14.5T if zvol is extended to 15T) (again, I hope you learned the lesson about pushing things to their limits...)
I really hope that this works.
If all of that work, recover your data and know that you will MUST destroy that pool and re-create it. In that new pool, create a zvol with the appropriate volume for your actual need (you said that you have a.5 TB, so 5 TB is surely more than enough).
After that, beware to never again fill a pool up to its limit and also start designing your backup plan. It is for situations like this (and many others) that you need it and that it is important to have it in a separate support.
Absolutely, a lesson that I don't see how I'll be able to forget.
Here's another question, that may or may not be related this issue (a potential cause?)
I was tinkering with the write protection options for the problematic zpool; changing it to OFF (although zfs/zpool get readonly *pool* shows OFF), but when I click on save, I get the following error (which I seem to get whenever I try to do anything).
Code:
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 138, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/service.py", line 495, in update
rv = await self.middleware._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 974, in nf
args, kwargs = clean_and_validate_args(args, kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 932, in clean_and_validate_args
value = attr.clean(args[args_index + i])
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 607, in clean
data[key] = attr.clean(value)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 167, in clean
value = super(Str, self).clean(value)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 53, in clean
raise Error(self.name, f'Invalid choice: {value}')
middlewared.schema.Error: [compression] Invalid choice: ON
I also get this error while trying to check for updates:
Code:
[Errno 28] No space left on device: '/var/db/system/tmpf67thovp': Automatic update check failed. Please check system network settings.