Entire LIVE environment is down! DNS problems? help!

[Jaknell1011]

I cut my arm off...

I have 3 esxi hosts all pointing to an ISCSI extent on my FreeNAS box. They use DNS for the ISCSI target. They have my primary domain controller 10.10.1.108 as the DNS server. My problem is that 10.10.1.108 is a VM on the datastore. I brought my esxi hosts and my freenas box down today. When I brought everything back up, my esxi hosts do not see the datastore. I checked in freenas and I can still see my extent. (this summer i accidentally deleted it and lost all of my data once already).

Can anyone help me please make sure my data is still in tact, and help me reconnect to my datastores!

I will be here clicking refresh to answer questions.

And if anyone is willing to do a webex or something similar, let me know. I am getting desperate and I think because I messed this up once before I nervous about clicking anything until I know I haven't lost it all. Maybe I can throw a gift card or something your way for the help!

 

[Jaknell1011]

The only change I have made was on FreeNAS. When I brought it back up, it said it's IP address of 10.10.1.85 had "no carrier" but the second link 10.10.1.87, had a link. So after scrambling for 20 minutes or so, I deleted nic(zero) (which was 10.10.1.85) and made nic(1) that IP address because that is the IP all of my vmware hosts look for for the datastore.

So:
FreeNAS: 10.10.1.85
ESXi1:10.10.1.75
ESXi2:10.10.1.77
ESXi3:10.10.1.79

I will post ISCSI info soon.

 

[Jaknell1011]

ISCSI Target Base Name: iqn.2011-03.example.org.istgt
Discovery Auth: none

Portals:
1: IP Addr: 10.10.1.85
Port: 3260
--I also noticed 3 additional lines under the portal config. In one, the IP address: Box says "-----" with port 3260. The next two say IP Address: 0.0.0.0 port 3260.
Should I delete these blank entries for the portal?

Extent:
Name: extent1 Type: File Path to the extent: /mnt/Vol1/extent1 Size:5TB

Initiators:
Group ID: 1 Initiators: ALL Authorized Network: ALL

Target:
Target name: target1 Target Alias: ISCSI target 1 Target Flags: read-write Portal Group ID:1 Auth Method: Auto

Target1/Extent1


Should I redo my portal and target? I just don't want to mess with this too much until I get the all clear on my extent. Thanks for your help folks!

 

[cyberjock]

I hate to be a buzzkill, but this is probably a complex problem that requires intimate knowledge of your setup. I don't know how much help you are going to get on the forums.


Quite literally though, if your extent exists and you haven't deleted it and recreated an empty extent, then all your data is there. It's just a matter of setting up hard info (static IPs, etc.) to make your networking work enough to bootup the system. Of course, this is almost certainly going to break a bunch of crap once you get back up because your DHCP and/or DNS servers are going to be upset that you did stuff "behind it's back".

This is why basic services like DNS and DHCP should never be VMs. :P

 

[Jaknell1011]

I have the unit up and can access the web interface. I just need to make sure it is ok for me to delete my portal config, target config, and then make new ones and link to my extent. Do you see any harm in doing this? Good to know about DNS after the fact...eh live and learn I guess.

**Edit** Yes I can still see my extent and it's available space and usage, so all my data looks safe.

 

[Jaknell1011]

OK I got DNS working and setup on an old DNS server (physical) and I can now ping the FreeNAS and my ESXi hosts. It's just the fact that my hosts do not see the ISCSI via FreeNAS. Here is the message I am getting in vSphere:

 

[Jaknell1011]

Okay I rebuilt the target and had to remove a few screwy lines in the portal configuration, but I am now back up and running. What should have been a 30 minute job has now officially taken from 10:30AM to 6:30 PM. Oh well it's done and we are working again. Sorry for the chaotic posts.

For the record, and for future googlers out there, if your ESXi hosts cannot see your ISCSI target after a reboot, check your portal and target configurations. In my case I had to remove the target out of the ESXi storage adapter configuration and add it back in. Also, it is safe to edit the portal and target, JUST DON'T TOUCH THE EXTENT!

Hopefully someone will run into this in the future and get a glimmer of hope.

 

[depasseg]

To add to the lesson learned, if your esx hosts have a local storage based datastore, create a DNS vm on a couple of them. I keep an AD server on each of two local datastores and one on my iscsi datastore.