Pool Missing

[Bytor02]

FreeNAS-8.0.3-RELEASE-p1-x64 (9591) / Intel(R) Core(TM)2 CPU 6420 @ 2.13GHz / 2032MB / mfi0 Adapter:PERC 6/E Adapter iSCSI attached to 8x 1TB drives

Hello, I am very new to FreeNAS, so please bear with me. The previous Admin left the company and well...here I am.

The problem I am having is that our esxi servers use FreeNAS for VM storage and for some reason last week the esxi servers lost the connection to the NAS. After many days I got the esxi servers to "see" the NAS, but it would not connect after running Rescan on the esxi server. I then went to Add Storage and the LUN's were being presented from the NAS, after selecting the LUN and hitting next the dialog box informed me that the Disk was blank.

I do not think the data is gone because I have some strange things going on in FreeNAS.
1. The iSCSI drives are NOT listed under "gpart show".
2. The iSCSI drives are NOT listed under "camcontrol devlist".
3. There is nothing to select in the Import Volume GUI.
4. "mfiutil show drives" DOES list the drives.
5. "zpool list" returns "no pools available"
6. "zpool import" does nothing

With my limited knowledge it is like the drive is there, but it cannot see the pool / volume.

I have been up and down the FreeNAS threads for the past week and a half. Now I am stuck.

Any help would be greatly appreciated. Please let me know what other information you might need.

-Clint

 

[joeschmuck]

I am not qualified to discuss iSCSI but many other things I can offer some assistance and I'm certain someone will jump in. Let me start with a few questions.

1) Do you have a backup of your configuration file? I don't mean that you create one this minute, I'm looking for an older one from before the problems started.
2) Post the output from "dmesg".
3) Does the GUI work? Meaning is FreeNAS running?
4) Since you state that you can see the drives with mfiutil, can you see them with smartctl? "smartctl -a /dev/ada0" or whatever the drive lettering is?

I think that will be a good start.

 

[anodos]

Ouch! Upload a debug file "system" -> "advanced" -> "save debug". I think calling the previous IT monkey an "Admin" is overly generous. It's almost like he aggregated all of the "what not to do's" into a list and used it as a purchasing and config guide.

 

[gpsguy]

I don't think there was a "save debug" option on 8.0.3. IIRC, that version was released - Jan 2012.


Sent from my iPhone using Tapatalk

 

[Ericloewe]

I won't lie, this looks like an unmitigated disaster.

Upload a debug file "system" -> "advanced" -> "save debug".

Was that available back then?

 

[anodos]

I won't lie, this looks like an unmitigated disaster.


Was that available back then?

It is listed on page 41 of the FreeNAS 8.0.3 guide here: http://doc.freenas.org/8.0.3/freenas8.0.3_guide.pdf

 

[gpsguy]

I stand corrected, ignore my comment above.


Sent from my iPhone using Tapatalk

 

[Robert Trevellyan]

Regardless of anything else you try, add more RAM, as much as the box will take if possible.

 

[Bytor02]

All, Thanks for the help. It is appreciated.

Robert, yes in all my reading I realize that it is under the minimum. This will be corrected.

1) Do you have a backup of your configuration file? I don't mean that you create one this minute, I'm looking for an older one from before the problems started.
Yes - I had two and tried them both.
2) Post the output from "dmesg".
See attachment
3) Does the GUI work? Meaning is FreeNAS running?
Yes - The GUI works.
4) Since you state that you can see the drives with mfiutil, can you see them with smartctl? "smartctl -a /dev/ada0" or whatever the drive lettering is?
Unknown - I tried smartctl -a /dev/mfid0 and smartctl -a /dev/mfid1 and received the following:

/dev/mfid0: Unable to detect device type
Smartctl: please specify device type with the -d option.

Is using mfid0 and mfid1 correct?

5) Upload a debug file
See attachment

 

[Robert Trevellyan]

It might be a good idea to add RAM before attempting any further recovery actions.

mfid0: <MFI Logical Disk> on mfi0
mfid0: 3813376MB (7809794048 sectors) RAID volume 'SAN1' is optimal
mfid1: <MFI Logical Disk> on mfi0
mfid1: 3813376MB (7809794048 sectors) RAID volume 'VD2' is optimal

Are you using that PERC card in RAID mode? Two 4TB volumes?

mfi0: 20411 (516311588s/0x0008/FATAL) - Battery needs replacement - SOH Bad
mfi0: 20412 (516311588s/0x0008/info) - Battery started charging
mfi0: 20413 (516311588s/0x0008/WARN) - Current capacity of the battery is below threshold

Did you have a power outage recently? If so, that might have trashed your RAID volumes, if the battery was already bad.

Do you have a spare PERC card identical to the one in this machine?

I think you should plan on not seeing this pool again. If you do, it will be a bonus.

 

[Bytor02]

It is possible that we had a power outage as I was not onsite when this was reported.

Are you using that PERC card in RAID mode? Two 4TB volumes?
Yes - but I thought the volumes were SAN and SAN1. I don't recognize VD2

Can the battery on the PERC be replaced (it looks like it can) or do I need to get another card?

 

[Robert Trevellyan]

Are you using that PERC card in RAID mode? Two 4TB volumes?
Yes - but I thought the volumes were SAN and SAN1. I don't recognize VD2

Maybe SAN and SAN1 is how they were named in FreeNAS. My quote from your dmesg shows how they were named by the PERC card.

It's worse than I first realized. With eight 1TB drives configured as two 4TB volumes at the PERC level, you must have two RAID0 volumes, i.e. no redundancy at the PERC level. A single disk failure in either volume would take out that entire virtual device, leaving ZFS nothing to work with.

Can the battery on the PERC be replaced (it looks like it can) or do I need to get another card?

I expect the battery is replaceable. Unfortunately, I don't expect that to fix your missing pool.

I see only one recovery scenario. If the PERC card went bad, or is misbehaving due to a bad battery, there's a chance replacing the battery or the card will resurrect the pool. If the pool comes back, immediately back it up, then build a new FreeNAS following the best practices described in these forums.

 

[Robert Trevellyan]

By the way, what's the 250GB Seagate Barracuda doing in there?

 

[Bytor02]

That drive is local in the FreeNAS box.

 

[Bytor02]

Well.....I ordered 8gigs of RAM and a new battery for the controller. Once they arrive I will rebuild the SAN using RAID1.

Thanks for all the help!

 

[anodos]

You need to use the '-d' switch to retrieve smart information form drives connected to a RAID controller.

Code:

smartctl -a -d megaraid,14 /dev/sdb

In the above command 'megaraid' represents the controller type (I think the perc controllers are megaraid), '14' represents the drive number as identified internally in the RAID controller, '/dev/sdb' represents where the raid controller resides (I usually do this on linux hence the sdx - replace with whatever is appropriate for your system - probably /dev/mfi0).

 

[anodos]

Well.....I ordered 8gigs of RAM and a new battery for the controller. Once they arrive I will rebuild the SAN using RAID1.

Thanks for all the help!

Your server design is still fundamentally flawed. 2GB RAM + ISCSI is a bad idea. 8GB RAM is better. 8GB RAM + RAID controller is still a bad idea. 8GB RAM + ZFS + RAID controller with a failing BBU is a terrible idea. See hardware recommendations thread for more info - https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/

 

[Bytor02]

Anodos, thank you for the link!

 

[Robert Trevellyan]

I will rebuild the SAN using RAID1.

Please don't do this, it will likely end the same way - unless you plan to switch to a different host OS.

 

[Bytor02]

Please don't do this, it will likely end the same way - unless you plan to switch to a different host OS.

Hi Robert, do you mean upgrade to a newer version of FreeNAS?