delete 11.3-RELEASE folder : "Device or resource busy"

[zmaster9877]

Hey,
A bit of a long story, I had a hard drive failure in a Mirrored pool "Pool1" - I bought a new hard disk and replaced/resilvered the failed disk (sdb). After the resilver completed the ZFS Health is showing as "Pool is not healthy". The disks are showing 44 errors per disk in the pool.

So i ran a:

Code:

root@truenas[~]# zpool status -v Pool1
  pool: Pool1
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: resilvered 2.89T in 04:31:37 with 22 errors on Sat Oct 14 15:12:15 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        Pool1                                     ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            17a282ff-a536-41d4-a3d2-75d2c14b9187  ONLINE       0     0    44
            5ad75c16-9cf6-4f0c-a47e-99caefaf8b80  ONLINE       0     0    44

errors: Permanent errors have been detected in the following files:

        Pool1/mediaset:<0x2df39>
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/netconfig
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/disktab
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/blacklistd.conf
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/mac.conf
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/group
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/rc.bsdextended
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/Makefile.depend
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/etc/csh.cshrc
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/lib/libc.a
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/stand/efi/include/Protocol/EdidActive.h
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/stand/efi/include/Protocol/EdidOverride.h
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/stand/efi/libefi/env.c
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/stand/efi/libefi/efi_driver_utils.c
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/stand/efi/libefi/devicename.c
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/stand/efi/libefi/Makefile
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/crypto/openssh/regress/unittests/sshkey/testdata/ecdsa_2.pub
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/crypto/openssh/regress/unittests/sshkey/testdata/rsa1_1_pw
        Pool1/iocage/releases/11.3-RELEASE/root@DLNA:/usr/src/crypto/openssh/regress/unittests/sshkey/testdata/ed25519_1

This system was originally Freenas version 11 however I installed Truenas Scale over the top. With the detected files with permanent errors i had one file within the mediaset that i have deleted and i figured the remaining files are files i never cleaned up from freenas and I can just delete them too. However i can't:

Code:

root@truenas[~]# cd /mnt/Pool1/iocage/releases
root@truenas[/mnt/Pool1/iocage/releases]# ls -al
total 11
drwxr-xr-x 5 root root  5 Jan 26  2022 .
drwxr-xr-x 9 root root 12 Oct  3  2021 ..
drwxr-xr-x 3 root root  3 Oct 14 16:48 11.3-RELEASE
drwxr-xr-x 3 root root  3 Oct  4  2021 12.2-RELEASE
drwxr-xr-x 3 root root  3 Jan 26  2022 12.3-RELEASE
root@truenas[/mnt/Pool1/iocage/releases]# rm -rf 11.3-RELEASE
rm: cannot remove '11.3-RELEASE/root': Device or resource busy
root@truenas[/mnt/Pool1/iocage/releases]# cd 11.3-RELEASE
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# ls -al
total 2
drwxr-xr-x 3 root root 3 Oct 14 16:48 .
drwxr-xr-x 5 root root 5 Jan 26  2022 ..
drwxr-xr-x 2 root root 2 Oct 14 16:32 root
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# rm -rf root
rm: cannot remove 'root': Device or resource busy
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# cd root
root@truenas[...ol1/iocage/releases/11.3-RELEASE/root]# ls -al
total 1
drwxr-xr-x 2 root root 2 Oct 14 16:32 .
drwxr-xr-x 3 root root 3 Oct 14 17:43 ..
root@truenas[...ol1/iocage/releases/11.3-RELEASE/root]# cd ..
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# mkdir abc
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# ls -al
total 2
drwxr-xr-x 4 root root 4 Oct 14 17:43 .
drwxr-xr-x 5 root root 5 Jan 26  2022 ..
drwxr-xr-x 2 root root 2 Oct 14 17:43 abc
drwxr-xr-x 2 root root 2 Oct 14 16:32 root
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# rmdir abc
root@truenas[/mnt/Pool1/iocage/releases/11.3-RELEASE]# ls -al
total 2
drwxr-xr-x 3 root root 3 Oct 14 17:43 .
drwxr-xr-x 5 root root 5 Jan 26  2022 ..
drwxr-xr-x 2 root root 2 Oct 14 16:32 root

The directory is empty, i can create a new directory and remove it, so something has a lock on the directory. I am a bit confused as to what that is and how to remove the lock - any ideas?

Thanks in advance