Corrupted datastore on iSCSI share

[walvord]

I started the day with a perfectly working iSCSI share on my FreeNAS 9.3 server to an esxi 5.5 host. Then I decided that with some holiday downtime, it'd be the perfect time to swap out the existing quad gb NICs used for iSCSI on my FreeNAS and esxi servers and upgrade them to a pair of intel x520-da2's.

Of course I didn't do my usual legwork and just winged it. Now I can see the LUNs from esxi, but I can't see the datastore.

Code:

~ # partedUtil getptbl /vmfs/devices/disks/naa.6589cfc00000047cd6d350ab993b3e8f
 Error: Connection timed out during read on /dev/disks/naa.6589cfc00000047cd6d350ab993b3e8f
Error: Connection timed out during read on /dev/disks/naa.6589cfc00000047cd6d350ab993b3e8f
Error: Both the primary and backup GPT tables are corrupt.  Try making a fresh table, Or use appropriate tools to recover partitions.
Unable to read partition table for device /vmfs/devices/disks/naa.6589cfc00000047cd6d350ab993b3e8f

In all honesty, it wouldn't be the end of the world if it's lost, but it would mean a lot of wasted time. Is there a way to recover it?


thanks in advance and happy holidays,
will

 

[joeschmuck]

Did you try to revert back to the old nics?

 

[walvord]

First off, I truly appreciate any assistance with this. By winging it, I did more than just swap the nics. I thought I could just drop in the new nics, remove the old vnics and add the new ones to the vswitch on the esxi side...and then on the freenas side just remove the now unused ip's from the iscsi portal.

But I ran into an esxi bug which prevented the iscsi binding and before I stumbled upon the fix (wiping /etc/vmware/vmkiscsid, rebooting, and then trying again). So throughout the course of troubleshooting that...I essentially completely removed the configs on the vmware side. On the freenas side, I didn't have to remove everything. I'm guessing that I removed some vmware crap in the wrong order.

...but on to where I am now. Esxi can see the freenas lun's but not the datastore. When I checked the partition table from esxi, it shows that the partition table is corrupt and it helpfully suggests that I use the "appropriate tools to recover partitions".

Code:

Error: Both the primary and backup GPT tables are corrupt.  Try making a fresh table, Or use appropriate tools to recover partitions.

Any ideas on how to recover the partitions of the .extents file on freenas?

By the way, if this needs to be moved to a different section I'm fine with that. Was torn between 'sharing', 'storage', or 'hardware' but chose hardware since that's what started this fun.


merry christmas,
will

 

[joeschmuck]

Any ideas on how to recover the partitions of the .extents file on freenas?

Unfortunately I do not have that kind of experience so hopefully someone else here might have some.

By the way, if this needs to be moved to a different section I'm fine with that. Was torn between 'sharing', 'storage', or 'hardware' but chose hardware since that's what started this fun.

Well you could make an argument either way so it doesn't matter in my book. You are not in the Introduction section so I'm happy.

Merry Christmas to you as well,
-Mark

 

[walvord]

No worries. I appreciate your help on today of all days.

I'll have to do some more testing, but it looks to be fully resolved. I'm going cross-eyed, but hopefully this might help someone else. Here's a recap from memory and some mediocre notes...

From checking /var/log/vmkernel.log on the esxi server, it looked like the nic's were bouncing. Then while double-checking the network settings on both esxi and freenas, I noticed that the vswitch didn't have jumbo frames enabled. I thought that port group settings overrode the vswitch settings, but apparently not mtu. Once I set that, I was able to successfully set the partition table (first 'partedUtil getUsableSectors' to get the end block followed by 'partedUtil setptbl' to set the new partition table). Then I ran 'vmkfstools -V' and finally I was then able to import the datastore.