@cyberjock: I feel that you're skimming over my previous posts, rather than fully reading everything. The kernel's response to various states of MBR and primary/secondary corruption and availability of "gpart recover" is detailed above.
Onto the main event: I made a FreeNAS VM in VirtualBox:
1x 512GB UFS HDD for constant storage (ada1)
3x 1TB ZFS-Z1 pool for testing, using all the FreeNAS GUI defaults (ada2, ada3 & ada4)
I then booted to Linux and dd'd varying amounts of /dev/zero over the discs. FreeBSD's geom subsystem had locked exclusive access to the base raw /dev/adaX devices and I didn't feel like figuring that out tonight:
ada2 for 512 * 32 bytes (all the primary GPT)
ada3 for 512 * 1 bytes (just the PMBR)
ada4 for 512 * 1000 bytes (all the primary GPT, plus some of the swap partition)
Back in FreeNAS afterwards, (ada2, ada3, ada4) were missing from "gpart list". No zpools available. No lines from "dmesg | grep -i corrupt".
Rewrite the PMBR to each disk in FreeNAS:
dd if=/boot/pmbr-datadisk of=/dev/ada2 (bs=512 count=1, optional as long as the pmbr-datadisk file is 512b)
dd if=/boot/pmbr-datadisk of=/dev/ada3
dd if=/boot/pmbr-datadisk of=/dev/ada4
Reboot FreeNAS.
During bootup, I saw kernel lines saying that ada2 and ada4 had corrupt primary GPTs. There was no mention of ada3 problems - as you'd expect. The ada2 and ada4 problems were shown in "dmesg | grep -i corrupt" after boot.
Fix the two corrupt GPTs:
gpart recover /dev/ada2
gpart recover /dev/ada4
I didn't reboot here, as these legal gpart commands had kicked the kernel to rescan, unlike dd. (ada2, ada3, ada4) were then showing OK in "gpart list" ...best of all, I was able to auto-import the ZFS tank. It also scrubbed with no errors.
So: 100% complete and easy recovery as long as 1) the secondary GPT at the tail of the disc is intact and 2) the ZFS filesystem itself is OK.
In the case of ada4 here, I didn't check if FreeBSD was actually happy with the swap partition. I seem to remember using a Unix OSs that needed a header to this too, while others some use it raw. In any case, it's non-critical and would simply be a standard non-ZFS BSD command to kick it back into place.
@Thomas: hopefully this will fix your system. If you're happy with the head and tail backups you already have, then there's little risk to this step. Your next worry will be what ZFS makes of the FS if you have an intact tail GPT.
To answer another question I set for myself:
Code:
[root@freenas] /mnt/backups# od -c ada2_16kb_head.dat | head -n 30
0000000 E F I P A R T \0 \0 001 \0 \ \0 \0 \0
0000020 S 277 024 254 \0 \0 \0 \0 001 \0 \0 \0 \0 \0 \0 \0
0000040 377 377 377 | \0 \0 \0 \0 " \0 \0 \0 \0 \0 \0 \0
0000060 336 377 377 | \0 \0 \0 \0 326 % 005 331 303 \t 343 021
0000100 243 273 \b \0 ' 9 F 346 002 \0 \0 \0 \0 \0 \0 \0
0000120 200 \0 \0 \0 200 \0 \0 \0 034 % 303 r \0 \0 \0 \0
0000140 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0001000 265 | n Q 317 n 326 021 217 370 \0 002 - \t q +
0001020 % 312 027 331 303 \t 343 021 243 273 \b \0 ' 9 F 346
0001040 200 \0 \0 \0 \0 \0 \0 \0 177 \0 @ \0 \0 \0 \0 \0
0001060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0001200 272 | n Q 317 n 326 021 217 370 \0 002 - \t q +
0001220 232 H . 331 303 \t 343 021 243 273 \b \0 ' 9 F 346
0001240 200 \0 @ \0 \0 \0 \0 \0 336 377 377 | \0 \0 \0 \0
0001260 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0040000
Code:
[root@freenas] /mnt/backups# od -c ada2_16kb_tail.dat | head -n 30
0000000 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0001000 265 | n Q 317 n 326 021 217 370 \0 002 - \t q +
0001020 % 312 027 331 303 \t 343 021 243 273 \b \0 ' 9 F 346
0001040 200 \0 \0 \0 \0 \0 \0 \0 177 \0 @ \0 \0 \0 \0 \0
0001060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0001200 272 | n Q 317 n 326 021 217 370 \0 002 - \t q +
0001220 232 H . 331 303 \t 343 021 243 273 \b \0 ' 9 F 346
0001240 200 \0 @ \0 \0 \0 \0 \0 336 377 377 | \0 \0 \0 \0
0001260 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
*
0040000
...the primary GPT 16KB table at head+512 isn't identical to the secondary GPT 16KB table at tail-16KB. The center (entries) are the same, but the header is seemingly swapped in location. This is actually shown clearly in the diagram here:
http://en.wikipedia.org/wiki/GUID_Partition_Table
No need for panic, as "gpart recover" can readily juggle primary and secondary, automatically cloning out the "bad" with the content from the "good". If you arrived at a situation where you had two "good" ones, you could choose which to keep by selectively zeroing out ther other prior to running "gpart recover".
Hope this helps somebody.
