missing iSCSI extents after upgrade

[skyman]

Just upgraded to RC2 and ran into trouble during the process and it's seems I'm missing my ISCI extents... I was on the prior 8.X release so I couldn't upgrade via the GUI. I tried the CD method, but CD wouldn't boot and after burning and after trying the 3rd CD incase there was a burning error of some kind... I was a little tired and just blew the full install image onto my flash drive.

When I booted with the new image and auto imported the ZFS pool, my iSCSI extents/targets/etc. seem to be missing...

Yes, I forgot to save my config via the GUI... or more to the point, I was sure I had saved it, but for some reason... don't see it... so must have screwed that up...

I have not done anything with the pool other that a zpool clear command at the console to clear an lingering error. I'd like to get a shot at the data that was one that extent as it was only there...

Also I noted that the auto-import doesn't seem to include my hot spare drive for some reason...

UPDATE: Looks like I did export the config after all so I'm not as dense as I feared. However, when the db reloaded I've run into new challenges... For one thing I had to rename some things because apparently uppercase character are no longer supported and the error message doesn't help with that... (specifically target names)

I was doing my extents as files and they have shown up at last. (they weren't visible for some period of time).. or at least the GUI couldn't find them... so now, while I'm better, I'm still in trouble... I'm trying to setup the "Associated Target" and nothing is showing up in the drop down for the extent, just a line of dashes... if I type in the extent, it says it's not valid..

Oh... FYI - This was an 8x2TB zpool setup as 7 drives in RAIDz2 and a hot spare...

UPDATE 2: Well I got it to work eventually (I don't really think I did anything as much as it finally scrubbed itself to a working point) BUT... The iSCSI Target now looks like an uninitialized disk to Windows instead of the 4TB of media that was there before. I has stayed far away from allowing Windows to do anything other than connect to the Target to confirm that that portion of the system was working...

Now about the data... any idea on how to recover it? Windows sees the target as a plain unallocated disk...

I'd appreciate any help...

 

[jgreco]

No ideas, but it seems bad. What might be a little helpful or interesting would be to go in, gingerly, and take a look at the content of the extent files and see if there's reasonable-looking stuff there. You shouldn't see, for example, all NUL's (though many are to be expected). It's safe to use the "less" pager on the file.

Now, from the command line, you can also poke a little at it. Remember, these are essentially disk drives stored as files. Use mdconfig to make them appear to the system:

[jgreco@storagex] /mnt/storagex/iscsi# ls -al
total 10207731
drwxr-xr-x 2 root wheel 3 Sep 24 08:38 ./
drwxr-xr-x 10 root wheel 15 Sep 11 07:31 ../
-rw-r--r-- 1 root wheel 1099511627776 Sep 24 08:38 extent0
[jgreco@storagex] /mnt/storagex/iscsi# mdconfig -a -t vnode -f extent0
md3

That makes the file available to FreeBSD as /dev/md3. So then you can ask about the partitioning of that with fdisk:

[jgreco@storagex] /mnt/storagex/iscsi# fdisk md3
******* Working on device /dev/md3 *******
parameters extracted from in-core disklabel are:
cylinders=133674 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=133674 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 251 (0xfb),(unknown)
start 128, size 2147472682 (1048570 Meg), flag 0
beg: cyl 0/ head 2/ sector 3;
end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

And that's as far as I can take you, since our extents are VMware. But theoretically you could use mount_msdosfs or mount_ntfs to mount the filesystem and look around. I suggest doing that sort of thing on a COPY of the extent, because your risk of modifying things goes up exponentially as you get into more complex poking.

Make sure to release the md when done:

[jgreco@storagex] /mnt/storagex/iscsi# mdconfig -d -u md3

 

[skyman]

Thanks... Seems the partition table is screwy according to fdisk so I have some anxiousness about what to do next to repair it without destroying the data.
I took a shot using mdconfig and fdisk and here's what I got:

******* Working on device /dev/md3 *******
parameters extracted from in-core disklabel are:
cylinders=802048 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=802048 heads=255 sectors/track=63 (16065 blks/cyl)

fdisk: invalid fdisk partition table found
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 4294967233 (2097151 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 85/ head 89/ sector 4
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

 

[jgreco]

"fdisk: invalid fdisk partition table found" probably means that the following partition table has been conjured up by the imagination of the fdisk program, so the rest of the fdisk output is probably useless.

Is there any chance that maybe you had disk encryption or something like that enabled in Windows? I'm trying to think about scenarios that would cause something like this.

 

[skyman]

encryption - not part of the plan...

I had a disk fail in the array and did a replace 24 hours before going through the upgrade.
The first couple boots after loading the new OS, the PCIe SATA card I use for two of the drives wasn't recognizing their presence, so I shut everything down, reseated the card and all of the cables and they came up fine after that.
The GUI wasn't recognizing the disks at that point, so I did an import at the command line, which brought everything in... That was about the time I found my GUI DB export, so I reloaded that and did a reboot... That's how I got here...

As I mentioned Windows sees that the drive is there, but thinks it's blank/no partitions...

BTW: The fdisk data does not match what I expect for used space in the file. It should be about 3.5-4TB of media center files (the complete Battlestar Galactica Run, Dr. Who from Tennant on to the current doctor, a ton of Good Eats and a lot of Norm in the New Yankee Workshop), home photos/video, and my Digital Music collection. The Music and the family pics are all backed up... The roughly 3TB of recorded TV was my only copy...

 

[jgreco]

Do you have other files (not just file extents) stored on the NAS? Do they seem intact?

I haven't seen failures like this where there wasn't also other obvious brokenness, and it isn't clear to me how this is operating if there is severe data damage. Quite frankly I'm not sure what to ask next, in part because there are several things that happened leading up to this that could have been responsible. I hope someone else can step in and offer some suggestions.