So, this appears to be a fairly common problem since I spent the last day of my life searching about it and did find others that had the exact same results, but I couldn't find any specific solutions and it sounds like I need to manually recreate some partition info for the zpool to have any hope of recovering my nearly 15TB of data :( Unfortunately I'm having a hard time finding what to do to recreate the zpool info needed, so I'm now resorting to posting here and hoping someone can help, which I am willing to pay for if needed.
Hardware: Supermicro mobo, Xeon E3-1231v3, 16GB ECC RAM, 4x 6TB (HGST HDN726060ALE614) ZFS RAID array (raidz1 I think? my usable space is basically 3x 6TB)
I was running 9.10.2-U6 on a USB drive for years, which suddenly got unrecoverable errors the other day, so I figured this was a good time to upgrade to 11.1 and decided to put it on a 250GB SSD this time since I noticed one of the features of 11 is the ability to finally run VMs on this server and I've had lots of similar USB drive issues in the past that I'm hoping to avoid by going with a real SSD.
Side note: I didn't realize at the time that 11.2 was released as stable now, since I had just went straight to https://download.freenas.org/11/latest/x64/ and got the ISO from there.
So, I removed the USB drive (still have it in case something can be recovered from it), put the 11.1-U6.3 ISO on a different USB drive and proceeded to install on the SSD. I definitely picked the right drive when asked where to install, and everything seemed to go smoothly, except my zpool was now missing and not showing up in the GUI for importing, nor does it show up in the CLI using commands like "zpool import" or "zpool status".
After further research, I checked "gpart show" and can see that all of the partitions are still there:
However, the zpool info seems to be gone, including any labels, which I don't think I ever created any specific labels in the past, but I hear it's bad if zbd doesn't see any, such as:
(all the other drives gives the same results, except the ada4p2 on the 250gb drive with the running FreeNAS on it)
I then tried going back to FreeNAS 9.10.2-U6 hoping that it was just an issue with 11 not being able to see my particular zpool, but unfortunately the results are the same there now too. I even tried restoring my FreeNAS config backup, which then shows my "freenas" zpool volume in the GUI, but it's in a LOCKED state and fails to unlock, since it doesn't actually exist anymore.
Other notes:
The data on this volume isn't mission critical and I didn't have anywhere big enough to back it all up to, so no, I don't have any backup that I could use to restore from, but it's still a lot of data that I really don't want to lose so I really hope someone here is able to help me recover it.
I'm not sure if the volume was encrypted, but I did see geli mentioned when I ran "fstyp -u /dev/ada0p2", so I think it may be and I don't remember making any encryption key backup, although maybe the USB drive that failed could have something recovered from it if absolutely necessary, or maybe my FreeNAS config backup has it?
I searched for ZFS recovery software and doesn't seem like there is much available on that end, other than https://www.r-explorer.com/ that looks promising, but I don't want to just start blindly trying things and make the situation worse than it already is.
The most concerning thing about all this is why did installing FreeNAS 11.1 do this to me? Did I do something wrong or is it some bug? Looking back, I guess I should have just unplugged the 6TB drives while doing the new install and I'd probably be fine, but I had no idea that a new FreeNAS install had such risks and have never experienced this before when reinstalling FreeNAS.
Thanks in advance for any help getting me out of this bad situation.
Hardware: Supermicro mobo, Xeon E3-1231v3, 16GB ECC RAM, 4x 6TB (HGST HDN726060ALE614) ZFS RAID array (raidz1 I think? my usable space is basically 3x 6TB)
I was running 9.10.2-U6 on a USB drive for years, which suddenly got unrecoverable errors the other day, so I figured this was a good time to upgrade to 11.1 and decided to put it on a 250GB SSD this time since I noticed one of the features of 11 is the ability to finally run VMs on this server and I've had lots of similar USB drive issues in the past that I'm hoping to avoid by going with a real SSD.
Side note: I didn't realize at the time that 11.2 was released as stable now, since I had just went straight to https://download.freenas.org/11/latest/x64/ and got the ISO from there.
So, I removed the USB drive (still have it in case something can be recovered from it), put the 11.1-U6.3 ISO on a different USB drive and proceeded to install on the SSD. I definitely picked the right drive when asked where to install, and everything seemed to go smoothly, except my zpool was now missing and not showing up in the GUI for importing, nor does it show up in the CLI using commands like "zpool import" or "zpool status".
Code:
[root@freenas] ~# zpool status pool: freenas-boot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 ada4p2 ONLINE 0 0 0 errors: No known data errors [root@freenas] ~# zpool import [root@freenas] ~# zpool import -f freenas cannot import 'freenas': no such pool available
After further research, I checked "gpart show" and can see that all of the partitions are still there:
Code:
[root@freenas] ~# gpart show => 34 11721045101 ada0 GPT (5.5T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 11716850696 2 freebsd-zfs (5.5T) 11721045128 7 - free - (3.5K) => 34 11721045101 ada1 GPT (5.5T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 11716850696 2 freebsd-zfs (5.5T) 11721045128 7 - free - (3.5K) => 34 11721045101 ada2 GPT (5.5T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 11716850696 2 freebsd-zfs (5.5T) 11721045128 7 - free - (3.5K) => 34 11721045101 ada3 GPT (5.5T) 34 94 - free - (47K) 128 4194304 1 freebsd-swap (2.0G) 4194432 11716850696 2 freebsd-zfs (5.5T) 11721045128 7 - free - (3.5K) => 34 488397101 ada4 GPT (233G) 34 6 - free - (3.0K) 40 204800 1 efi (100M) 204840 488192288 2 freebsd-zfs (233G) 488397128 7 - free - (3.5K)
However, the zpool info seems to be gone, including any labels, which I don't think I ever created any specific labels in the past, but I hear it's bad if zbd doesn't see any, such as:
Code:
[root@freenas] ~# zdb -l /dev/ada0 -------------------------------------------- LABEL 0 -------------------------------------------- failed to unpack label 0 -------------------------------------------- LABEL 1 -------------------------------------------- failed to unpack label 1 -------------------------------------------- LABEL 2 -------------------------------------------- failed to unpack label 2 -------------------------------------------- LABEL 3 -------------------------------------------- failed to unpack label 3 [root@freenas] ~# zdb -l /dev/ada0p2 -------------------------------------------- LABEL 0 -------------------------------------------- failed to unpack label 0 -------------------------------------------- LABEL 1 -------------------------------------------- failed to unpack label 1 -------------------------------------------- LABEL 2 -------------------------------------------- failed to unpack label 2 -------------------------------------------- LABEL 3 -------------------------------------------- failed to unpack label 3
(all the other drives gives the same results, except the ada4p2 on the 250gb drive with the running FreeNAS on it)
I then tried going back to FreeNAS 9.10.2-U6 hoping that it was just an issue with 11 not being able to see my particular zpool, but unfortunately the results are the same there now too. I even tried restoring my FreeNAS config backup, which then shows my "freenas" zpool volume in the GUI, but it's in a LOCKED state and fails to unlock, since it doesn't actually exist anymore.
Other notes:
The data on this volume isn't mission critical and I didn't have anywhere big enough to back it all up to, so no, I don't have any backup that I could use to restore from, but it's still a lot of data that I really don't want to lose so I really hope someone here is able to help me recover it.
I'm not sure if the volume was encrypted, but I did see geli mentioned when I ran "fstyp -u /dev/ada0p2", so I think it may be and I don't remember making any encryption key backup, although maybe the USB drive that failed could have something recovered from it if absolutely necessary, or maybe my FreeNAS config backup has it?
I searched for ZFS recovery software and doesn't seem like there is much available on that end, other than https://www.r-explorer.com/ that looks promising, but I don't want to just start blindly trying things and make the situation worse than it already is.
The most concerning thing about all this is why did installing FreeNAS 11.1 do this to me? Did I do something wrong or is it some bug? Looking back, I guess I should have just unplugged the 6TB drives while doing the new install and I'd probably be fine, but I had no idea that a new FreeNAS install had such risks and have never experienced this before when reinstalling FreeNAS.
Thanks in advance for any help getting me out of this bad situation.