zpool permission denied

[jlcash]

I have a ZFS filesystem from TrueNAS that was stored on a ESXi VM (I know, bad idea). I am planning to migrate to a dedicated NAS server (TrueNAS Mini) but am unable to access the data in the pool. I have already deleted the ESXi host.

I have installed Ubuntu onto the PC that ESXi was previously installed onto, which has access to the disk, solely for the purpose of getting the data from the ZFS pool. This will be rebuilt as a XEN host without NAS (which will be on the aforementioned server)

The pool was a single disk.

I have mounted the vmfs file system in ubuntu under /mnt/vmfs using

sudo vmfs-fuse /dev/sda1 /mnt/vmfs

I can browse the vmfs and can see the -flat.vmdk and .vmdk files containing the ZFS filesystem. These are named NAS001-flat.vmdk and NAS001.vmdk. I then attached the ZFS filesystem to /dev/ndb1 using the following command:

sudo modprobe nbd
sudo qemu-nmd -r -c /dev/nbd1 /mnt/vmfs/NAS001/NAS001-flat.vmdk

sudo fdisk -l

now shows the ZFS filesystem on device /dev/nbd1p2 as type FreeBSD ZFS.

sudo zpool import

shows the pool as

pool001 ONLINE
nbd1p2 ONLINE

However I am unable to import the pool using

zpool import pool001 -f

I receive the following error message:

cannot import 'pool001': permission denied
Destroy and re-create the pool from a backup source

Is there a way to access the data in the ZFS filesystem, as I would like to back up, and then import into the new TrueNAS server. Just importing the disk won't work due to the wrapper VMFS filesystem.

 

[sretalla]

You could try first to show the full output from zpool import

That way we can see what's recognized or not.

 

[sretalla]

However I am unable to import the pool using

zpool import -f pool001 would be the right way around...

Next would be that you may not have the rights to create the mountpoint... /mnt/pool001 (if it doesn't exist, try creating it).

 

[jlcash]

zpool import -f pool001 would be the right way around...

Next would be that you may not have the rights to create the mountpoint... /mnt/pool001 (if it doesn't exist, try creating it).

The output of zpool importis
pool: pool001 id: 12979328589932627785 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY config: pool001 ONLINE nbd2p2 ONLINE pool: M2 id: 7665506777843331956 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY config: M2 ONLINE 18d12772-02ca-40de-903f-f8cd089efc38 ONLINE

I then manually created the mount point and run the import command.
sudo mkdir /mnt/pool001 sudo zpool import -f pool001

This returns the same error message
cannot import 'pool001': permission denied Destroy and re-create the pool from a backup source.

 

[jlcash]

Does anyone know of a solution for this. I don't see any obvious permission issues and would like to back up this data before I rebuild my NAS.

 

[Samuel Tai]

You don't need to create the mount point. zpool import will do it for you. What's likely going on is you're trying to mount on the root directory /.

First, rmdir /mnt/pool001. Then zpool import -f -c /data/zfs/zpool.cache -R /mnt pool001. Note the -R, which sets the altroot property, and forces the pool to mount inside /mnt.

 

[jlcash]

This did not work. I received the same permission denied error. I also didn't have the .cache file so tried excluding the -c option

Code:

sudo rmdir /mnt/pool001/
sudo zpool import -f -c /data/zfs/zpool.cache -R /mnt pool001

failed to open cache file: No such file or directory
cannot import 'pool001': no such pool available

sudo zpool import -f -R /mnt pool001
cannot import 'pool001': permission denied
        Destroy and re-create the pool from
        a backup source.

 

[Samuel Tai]

Oh, I see from the OP you already imported it. You can't import an already-imported pool. Try zpool export pool001 and then zpool import -f -R /mnt pool01.

 

[jlcash]

The pool is not imported as it failed during the import. I tried your suggestion with no success.

Code:

sudo zpool export pool001
cannot open 'pool001': no such pool

sudo zpool import -f -R /mnt pool001
cannot import 'pool001': permission denied
        Destroy and re-create the pool from
        a backup source.

 

[Samuel Tai]

Unfortunately, it sounds like the VMDK might be damaged. What does zpool import -f -F -n -R /mnt pool001 report? This simulates a more aggressive import attempt without actually performing the import.

 

[jlcash]

This doesn't return anything

Code:

jlcash@lab:/$ sudo zpool import -f -F -n -R /mnt pool001
jlcash@lab:/$

 

[Samuel Tai]

Is the VMDK still attached? Does fdisk still see the partition?

 

[jlcash]

Yes. Relevant part of output below:

Code:

Disk /dev/nbd2: 14.55 TiB, 15997894183936 bytes, 31245887078 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7B14D8B7-5DCA-11EC-9B60-000C291B04F5

Device        Start         End     Sectors  Size Type
/dev/nbd2p1     128     4194431     4194304    2G FreeBSD swap
/dev/nbd2p2 4194432 31245887039 31241692608 14.5T FreeBSD ZFS

 

[Samuel Tai]

OK, the fact the simulation didn't return an error is a good sign. Try the same command without the -n parameter to perform it for real. If it completes without an error, then see if the pool is online via zpool status -v pool001.

 

[jlcash]

No luck. Without the -n, I get the same error as before:

Code:

jlcash@lab:/$ sudo zpool import -f -F -R /mnt pool001
cannot import 'pool001': permission denied
        Destroy and re-create the pool from
        a backup source.

 

[Samuel Tai]

Unfortunately, we're now at the point where we'll need to use 3rd party tools, like Klennet, or a professional data recovery service. You've just proven that VMDKs aren't reliable for ZFS, and the proper means to safely provide disks to a virtualized TrueNAS is via a passed through HBA.

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[jlcash]

Thanks for trying. Looks like I will need to start over, this time outside of a vm.

 

[jlcash]

I finally got this working. After mounting the vm file system and attaching the virtual disk, I did a bit copy to a blank disk, and mounted the ZFS filesystem from there. I have described the steps below in case someone else encounters this problem.

1) Install a new disk of the same size as the one containing your TRUENAS zfs filesystem and boot to a non virtualised Linux OS (not TrueNAS). The steps below have the original disk containing the TrueNAS zfs filesystem on vm virtual drive mapped as /dev/sda, and the new disk mapped as /dev/sdb.

2) You should be root for the following, or prefix all commands below with sudo.

Code:

sudo su

3) Create mount point for the VM file system and mount it

Code:

mkdir /mnt/vmfs
vmfs6-fuse /dev/sda1 /mnt/vmfs

4) The virtual disk should be available under /mnt/vmfs/xxxx/xxxx.vmdk, where xxxx is the name of the virtual machine that you used to run TrueNAS. In my case this was /mnt/vmfs/NAS001/NAS001.vmdk. Attach this to a nbd device. fdisk -l should now show two partitions, the swap partition under /dev/nbd1p1 and and the ZFS filesystem under /dev/nbd1p2.

Code:

modprobe nbd
qemu-nbd -f vmdk -r -c /dev/nbd1 /mnt/vmfs/NAS001/NAS001.vmdk
fdisk -l

Code:

Disk /dev/nbd1: 14.55 TiB, 15997894183936 bytes, 31245887078 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7B14D8B7-5DCA-11EC-9B60-000C291B04F5

Device        Start         End     Sectors  Size Type
/dev/nbd1p1     128     4194431     4194304    2G FreeBSD swap
/dev/nbd1p2 4194432 31245887039 31241692608 14.5T FreeBSD ZFS

5) Copy the whole device, bit by bit, to the new disk. This will take a really long time if your disk is large. My 16TB drive took 39 hours. status=progress will display the progress of the copy.

Code:

dd if=/dev/nbd1 of=/dev/sdb bs=64k conv=noerror,sync status=progress

6) Once this is finished, fdisk -l should show the ZFS filesystem on /dev/sdb. Remove the nbd device to avoid any conflicts and run zpool import, which should show your pool. In my case, this was listed as pool001.

Code:

rm /dev/nbd1
zpool import

Code:

pool: pool001
     id: 12979328589932627785
  state: ONLINE
status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and
        the '-f' flag.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
 config:

        pool001     ONLINE
          sdb2      ONLINE

7) Import the zfs filesystem from the new disk, specifying where to mount. I mounted as /mnt/pool001

Code:

zpool import -f -F -R /mnt pool001

8) Your zfs pool should now be mounted and the mount point should now contain the contents of your TrueNAS zfs filesystem. Backup what you need then you can reuse your original and new disks in your next TrueNAS installation![/CODE]

 

[Samuel Tai]

Great news!