pool not importable after fresh reinstall

[tld8102]

Hi everyone.
I'm kind of new to Truenas. At the moment I'm playing around with virtualization. Unfortunately I somehow corrupted the root partition of my Proxmox installtion with it virtualising TrueNAS core at the time. I had to do a fresh install of proxmox.
Now I am unable to import the pool into Truenas with the GUI. I am I need some help here. I do not have a backup of this data. I think it's still there on the drive but I can't seem to access it.

Context:
x1 QEMU virtual hard disk Truenas OS was booting off of [ERASED SSD]
x3 WD Blue 3TB hard drives passed through to the virtual machine (original pool)
x3 Seagate Ironwolf 8TB hard drives passed through to the virtual machine (expanded pool)
x2 QEMU virtual hard disk in mirror used as a SSD Write cache [ERASED SSD]

root@truenas[~]# zpool import pool: Starship_Trenzalore id: 10199485811422978749 state: UNAVAIL status: One or more devices are missing from the system. action: The pool cannot be imported. Attach the missing devices and try again. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-6X config: Starship_Trenzalore UNAVAIL missing device raidz1-0 ONLINE gptid/344e4aa0-5824-11ea-9b1e-55332b0379a7 ONLINE gptid/3475db2e-5824-11ea-9b1e-55332b0379a7 ONLINE gptid/347ebce7-5824-11ea-9b1e-55332b0379a7 ONLINE raidz1-1 ONLINE gptid/bce44b6d-6891-11ec-8178-2dd58ff8e014 ONLINE gptid/bd58c7fd-6891-11ec-8178-2dd58ff8e014 ONLINE gptid/2c4de4e7-6cff-11ec-989e-cd606c961446 ONLINE logs mirror-2 UNAVAIL insufficient replicas 8899443486727901022 UNAVAIL cannot open 14789519229959048440 UNAVAIL cannot open Additional devices are known to be part of this pool, though their exact configuration cannot be determined.

All 6 of the passed through hard drives seem to be good. Im assuming, I somehow have to remove the write cache. but first i think i need to to get it to mount first before removing the logs. Am i on the right track here?
Is there any hope of recovering my files? What do i have to do next?

All suggestions are welcome and Merry Christmas in advance :)

 

[artlessknave]

problems you have set yourself up for:
#1. virtualizing TrueNAS is generally not recommended. if you must do so, only VMware ESX is considered reliable.
#2. there is no write cache disk in ZFS and a SLOG only applies to sync writes. the SLOG is a core part of the pool. you cannot import a pool with missing disks. as a SLOG is nearly 100% writes, you need a resilient, fast drive of nearly negligible size.
#3. no backups. nuff said.
#4. I am not sure how proxmox "passes" drives through. the semi-supported method to access drives on virtualisation is using PCIE passthrough of the entire storage controller.
#5. it looks like you passed disks to the VM, but then put your SLOG on virtualized drives? that is a recipe for disaster. *all* data and metadata drives should be directly accessable, with the exception of L2ARC, as the pool can function with no impact if those go missing.
#6. the QEMU SSDs most definitely do NOT have power protection, which if you dont have you should not use a SLOG.

now, a possible solution I believe is to remove the SLOG, i think done on import, but I have never been in such a position myself so I am very not familiar with it. you would have to force the pool to abandon anything on the SLOG, and depending on how you had it cobbled together to "work", might destroy the pool anyway. VM's could get corrupted.

I did some searching but was unable to determine how you would remove these. the people most likely to know tend to be less interested in helping when the recommendations were not followed. sometimes they will follow up when someone tries to help.

 

[tld8102]

problems you have set yourself up for:
#1. virtualizing TrueNAS is generally not recommended. if you must do so, only VMware ESX is considered reliable.
#2. there is no write cache disk in ZFS and a SLOG only applies to sync writes. the SLOG is a core part of the pool. you cannot import a pool with missing disks. as a SLOG is nearly 100% writes, you need a resilient, fast drive of nearly negligible size.
#3. no backups. nuff said.
#4. I am not sure how proxmox "passes" drives through. the semi-supported method to access drives on virtualisation is using PCIE passthrough of the entire storage controller.
#5. it looks like you passed disks to the VM, but then put your SLOG on virtualized drives? that is a recipe for disaster. *all* data and metadata drives should be directly accessable, with the exception of L2ARC, as the pool can function with no impact if those go missing.
#6. the QEMU SSDs most definitely do NOT have power protection, which if you dont have you should not use a SLOG.

now, a possible solution I believe is to remove the SLOG, i think done on import, but I have never been in such a position myself so I am very not familiar with it. you would have to force the pool to abandon anything on the SLOG, and depending on how you had it cobbled together to "work", might destroy the pool anyway. VM's could get corrupted.

I did some searching but was unable to determine how you would remove these. the people most likely to know tend to be less interested in helping when the recommendations were not followed. sometimes they will follow up when someone tries to help.

Thank you for taking the time to look into this. I understand it's all my fault. I shouldn't have cheapen out at the start of my freenas journey. Thank you for explaining it more clearly to me about the write caches. you’ve pointed me in a direction with your research and i think i’ve stumbled upon a lead. correct me if i wrong but i have to force import the pool without the slog (didn’t know that was an option), then remove the mirror from the pool.

zpool import -f -m <pool> zpool remove <pool> <mirror-X> <device1> <device2> zpool online <pool>

After this, i think there’s no going back and flying blind. Im prepared to say goodbye to the data on the SLOG as i don’t think they were being utilised at the time. i don't know if this would work as other peoples examples have been different as thier slog were online when they did it. so i’d like to run this idea by you. :)

 

[jgreco]

[mod note: moved to virtualization subforum, the listed problems are due to a botched virtualization -JG]

 

[jgreco]

Not to steal @artlessknave 's thunder and fundamentally correct answer, but please do have a look at

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

You should be able to drop your two SLOG virtual disks and then it is likely you can recover this. However, you haven't mentioned how you are giving your disks to FreeNAS. Please make sure you are using PCIe passthru.

 

[tld8102]

Not to steal @artlessknave 's thunder and fundamentally correct answer, but please do have a look at

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

You should be able to drop your two SLOG virtual disks and then it is likely you can recover this. However, you haven't mentioned how you are giving your disks to FreeNAS. Please make sure you are using PCIe passthru.

FIXED IT!!!
I don't think it was PCIe Passthrough. The drives were connected to the motherboard's SATA ports and not with a dedicated PCIe HBA card.

Update: (for anyone else who finds themselves in my position)

• "zpool remove <pool> <mirror-X> <device1> <device2>" command did not work. nor did "zpool online -e <pool> <mirror-X> <device1> <device2>". however "zpool offline -f <pool> <mirror-X> <device1> <device2>" did yield results as it no longer prints out the slog with "zpool status -x <pool>". my pool says it's online and all is healthy apparently.
• I wasn't sure what to do now. my pool still doesn't come up in the GUI.
• I eventually figured to start typing random commands like "sync" and stumbled across "export"
• I managed to get the pool to import using the GUI. I achieved this by first exporting the pool with "zpool export <pool>" then importing using the GUI.

Thanks very much for the support @artlessknave @jgreco

 

[artlessknave]

that is very good.
the next thing to do is make a backup, and then seriously reconsider how you have this designed.

 

[tld8102]

that is very good.
the next thing to do is make a backup, and then seriously reconsider how you have this designed.

yes... backup the pool. all my options seem pretty steep especially with the amount of hard drives i need to buy.

 

[artlessknave]

depending on what you have, you could do a backup via cloud, or get 1 18TB drive and backup to that.
ideally the backup is a robust as the primary, such as what I have setup, but it doesn't have to be. even a single copy on a single drive is far better than no backup at all. even if something does get corrupted on the backup disk(s), ZFS will at least tell you fix files cannot be restored.

 

[artlessknave]

Not to steal @artlessknave 's thunder and fundamentally correct answer, but please do have a look at

nope, more thunder is always better, plus, I was too lazy to go find that page, so it's great to have.

 

[Davvo]

the next thing to do is make a backup

Yup. When in doubt: backup. When you are sure: backup.
Bakcup will never be wrong. Backup is your friend.

 

[jgreco]

I don't think it was PCIe Passthrough. The drives were connected to the motherboard's SATA ports and not with a dedicated PCIe HBA card.

Not quite clear what your setup is, but just to point out, you can use PCIe passthru on most mainboard SATA controllers to give all the SATA ports to TrueNAS.