Recovering from 100% disk full

[licious]

After updating to 13.0, it seems the zfs pool is full and can't proceed. All of my regular processes that write to disk have quotas in place to leave plenty of space, which makes me think it was maybe part of the automated upgrade process / snapshots that filled it up?

I found a past thread indicating that once a volume is 100% full, it's essentially toast: https://www.truenas.com/community/threads/unable-to-destroy-snapshot-io-error.95452/post-664778
It does mention reconstituting from recent data, but I'm not sure there's a way to do that in place. Trying to remove snapshots or truncate/remove files fails (see below). If anyone has suggestions on how to proceed it'd be very appreciated. It's a relatively large pool so destroying and restoring from backup is not very desirable (though from what I've seen so far that seems to be all that's left besides swapping out for larger disks and expanding in place).

Code:

freenas# zfs destroy vol0/iocage/jails/unifi@ioc_update_12.1-RELEASE-p13_2022-03-20_19-09-55
cannot destroy snapshots: I/O error

Code:

freenas# df -h
...
vol0                                                18T     18T      0B   100%    /mnt/vol0

Code:

freenas# echo "" > bigolfile
-bash: bigolfile: No space left on device

Code:

freenas# rm SABnzbd_nzf_726atrpp 
rm: SABnzbd_nzf_726atrpp: No space left on device

 

[Samuel Tai]

In FreeNAS 11, it was possible to recover space from deleted files by running zfs remap <root dataset>. However, upstream OpenZFS removed that capability as it was rarely used. Thus TrueNAS 12 and higher lack remap, and the only options for your situation are the ones you’ve already identified.

 

[Apollo]

It should still be possible to recover some space if you can manage destroying snapshots that contains deleted files.
Datasets with quota can be a pain during replication process, especially when replicating to a smaller pool.
If you are running iocage jails, some of the root folders can grow in size on a daily basis and it should be safe enough destroying most of the older snapshots.