Tested my mirror, got unexpected results, and not sure what to do next

[rico]

I've been using FreeNAS-8.0.2-RELEASE-amd64 (8288) for many months now with two disks in a mirrored configuration.

Or at least I thought I had a mirrored configuration.

Today I finally got around to testing the rebuild capability of the mirror.

I pulled out the second of the 2 disks (let's call it the disk that was in the second disk slot), replaced it with a new disk of the same make/model/size, brought the system back up and tried to get to the data.

When the system came back up, I logged into the GUI and saw this for the volume:

Code:

  Volume	Path		Used          Available                      Size                       Status
  vol1		/mnt/vol1	None (Error)  Error getting available space  Error getting total space  <forgot to capture this field>

That was the only volume listed. I've only ever had one volume listed though.

At first this lead me to believe that all the data was only on the disk I pulled out (from the second slot) and there was not a copy on the disk in the first slot.

For another check, I moved the new disk from the second slot to the first slot, moved the disk that was originally in the first slot into the second slot, and still kept out the disk that I originally pulled from the second slot.

Now when the system comes back up, the GUI reports this for the volume:

Code:

  Volume  Path         Used           Available  Size      Status
  vol1    /mnt/vol1    21.0 KiB (0%)  1.3 TiB    1.3 TiB   DEGRADED

That's more like what I expected to see.

However, when I mount the volume, it appears to be empty. But before the test I had data on all but about 3 GiB of the 1.5 (1.3) TiB volume.

So should my data already be available?

Or do I need to wait for the system to re-make the mirror (assuming it is doing that)?

[edit] This system is one of those Intel SS4200 home file servers, if that makes a difference. Actually, it is the Siemens SCALEO version with an upgraded CPU and RAM (2 GB). Booting from USB stick.

Thanks in advance for your help!

 

[rico]

How do I determine what the pool name is for using with the 'zpool online' command?

[root@freenas] /mnt/vol1# zpool online /dev/ada0
missing device name
usage:
online <pool> <device> ...
​

I saw via another post that there are some commands I can run and "zpool status -v" suggested I run "zpool online".

[root@freenas] /mnt/vol1# zpool status -v
pool: vol1
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://www.sun.com/msg/ZFS-8000-2Q
scrub: scrub completed after 0h0m with 0 errors on Fri Apr 27 03:01:06 2012
config:

NAME STATE READ WRITE CKSUM
vol1 DEGRADED 0 0 0
mirror DEGRADED 0 0 0
gpt/ada0 ONLINE 0 0 0
14573397685397128675 UNAVAIL 0 0 0 was /dev/gpt/ada1

errors: No known data errors
​

http://www.sun.com/msg/ZFS-8000-2Q no longer works. Thanks Oracle.

For reference, here's the other post I got the "zpool status -v" command from: http://forums.freenas.org/showthread.php?5978-Replacing-a-drive-in-FreeNAS-8-0-3

 

[StephenFry]

I'm not an expert, but have been testing a lot recently with FreeNAS+ZFS, so I'm just throwing in my 2c.

So should my data already be available?
Absolutely, in any normal situation, when one drive of a ZFS mirror pair is unavailable, the other drive will ensure availability of your data.

Your situation might not be normal, for reasons as of yet unclear.

Or do I need to wait for the system to re-make the mirror (assuming it is doing that)?
As I confirmed above, this is a "nope". It also doesn't seem it's doing anything of the sort.

In my personal testing of FreeNAS ZFS mirrors, there was no difference between disabling drive 0 or drive 1. The pools continued functioning in a degraded state.

My suspicion is that you got bitten by the 8.0.2 GUI - it happened to me many times that the GUI and "real" ZFS system seemed not to communicate properly. You might try installing a fresh FreeNAS 8.0.4-p1 on a USB drive and import either or both of your mirror drives and see what happens.

 

[rico]

My suspicion is that you got bitten by the 8.0.2 GUI - it happened to me many times that the GUI and "real" ZFS system seemed not to communicate properly.

OK, first I think I'll try to get back to the configuration I had before I attempted to test the mirror. That is, I'll put the original disks back into the server in the slots they were originally in.

I hope it's not too late for that to work.

[size=+1]Someone please scream if that's going to be a catastrophic mistake![/size]

I haven't changed any data in this test configuration. Although new ZFS snapshots have been getting created. The old ones never showed up.

By the way, the GUI and the CLI both agreed on what snapshots were present before and after the test.

The only time they didn't agree was with particular NFS clients I was using. For example, a Fedora Core 12 client I was using wouldn't show all the snapshots under /mnt/vol1/.zfs/snapshot that I could see using NFS clients in Ubuntu and OSX. And the GUI always showed all the snapshots.

 

[rico]

OK, I didn't want to wait, and I figured that pulling out both disks and putting just the original one back in (the one that I pulled out in the first place to test the mirror) by itself couldn't hurt.

With the server back up now, all my data still seems to be on that disk.

And guess what - even though I only have one half of the mirror pair in the box, the status is still healthy!

[root@freenas] ~# zpool status -v
pool: vol1
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
vol1 ONLINE 0 0 0
gpt/ada0 ONLINE 0 0 0

errors: No known data errors
​

Does this mean that I never had a mirror in the first place?

Should there be a "gpt/ada1" entry as part of that zpool status when I have both disks in?

At this point, might there be any issues with putting the other original disk back in?

I am tempted to just put it back in.

I might check the BIOS as well to see if the second disk isn't recognized.

 

[rico]

I have confirmed that I never had a mirror in the first place.

The BIOS on this SS4200 was set to "ATA/IDE Configuration [Compatible]"

I just figured out that in that mode, disk slots 1 and 2 are not useable together (only disk 2 will be seen by the BIOS), and disk slots 3 and 4 are not useable together (only disk 4 will be seen by the BIOS).

It may be that disks 1 and 4 are useable together, and perhaps disks 2 and 3 are useable together, but I never tried those configurations (with ATA/IDE Configuration = Compatible, that is). I was using the configuration where the disks are in slots 1 and 2 and only the disk in slot 2 was seen by the BIOS.

I have since discovered that if I set the BIOS to, "ATA/IDE Configuration [Enhanced]", then I can get the disks in slots 1 and 2 to be seen by the BIOS at the same time (and presumably I can also get the disks in slots 3 and 4 to be seen by the BIOS at the same time).

With ATA/IDE Configuration = Enhanced, I also needed to choose "Configure SATA as [AHCI]" or "Configure SATA as [RAID]".

I've presently got it set as "Configure SATA as [AHCI]", since the other configuration would seem to be for hardware-based RAID and presumably freeNAS does its own RAID in software.

So I am back to having

Code:

Volume  Path       Used           Available  Size     Status
vol1    /mnt/vol1  1.3 TiB (99%)  13.6 GiB   1.3 TiB  HEALTHY

Although freeNAS is still only using 1 disk (as far as I can tell).

Here is what "zpool status -v" now reports:

[root@freenas] ~# zpool status -v
pool: vol1
state: ONLINE
scrub: none requested
config:

NAME STATE READ WRITE CKSUM
vol1 ONLINE 0 0 0
gptid/7a123633-7850-11e0-8261-00151731efd9 ONLINE 0 0 0

errors: No known data errors​

Note that instead of stating, "gpt/ada0 ONLINE" like it did before, now it states, "gptid/7a123633-7850-11e0-8261-00151731efd9 ONLINE"

I don't know the significance of that.

Any suggestions on how to get freeNAS into a mirrored configuration from here?

 

[rico]

Cool! Someone has recently posted a titled,

"Built system with 1 drive, now want to add second drive and set to raid 1"

http://forums.freenas.org/showthrea...ow-want-to-add-second-drive-and-set-to-raid-1

Sounds like just what I need.

 

[StephenFry]

Thank you for reporting back what the problem was. Good thing the drive didn't fail, too :)