ZFS volume unavailable after drive change and resilvering

[MoR]

HELP!!! I had a ZFS volume with 2 drives that were mirrored. One drive died and the volume became degraded. I bought a new drive to replace it with and followed the instructions provided at http://doc.freenas.org/index.php/Volumes#Replacing_a_Failed_Drive . Once that was done I restarted it and it began resilvering. It got stuck at 19.5% percent for hours. I restarted the NAS and removed the old drive that wasn't working out of the volume to leave only to drives. This seemed to worked and both drives were online this time. It began to resilver again and finished how ever the new drive is now unavailable again and the drive is degraded and I cannot access it through windows even though none of the shares or permissions have changed. I have a lot of work stuff on there and need this working ASAP. I've trawled through heaps of documents with no luck. I now look to who ever reads this to hopefully help me out please.

 

[joeschmuck]

First, backup all your data since you have access to it before you screw something up by accident. I would think you have that done already since your occupation is IT, but...

Second, if the resilvering really did stop, you may have other issues. This could be a SATA cable, bad RAM, Mother board issue, or Power Supply. You should investigate and figure it out. Maybe you had a power glitch, or something else off the wall happened.

And lastly, what version of FreeNAS are you running and list your hardware.

 

[MoR]

Well I actually didn't get to back it up because I restarted it to see if it would fix the problem and then tried a few other things. Long story short I can't remember exactly what I tried but it stopped letting me access it. When it was degraded it was working and then after doing what I did still degraded and getting access denied messages from windows. Changed the drive, replaced it in the pool. First resilver failed for reasons unknown. When I restarted it and removed the damged drive from the pool leaving only 2 drives the resilver completed successfully but had a lot of errors.....in the 8000's.

I have a raid 0 volume as well with 2 drives and that is working fine and accessible from windows.
Also when I try to create a new share, I select the volume and it just thinks....doesnt actually drop the tree down to select a folder.

Version is FreeNAS-9.2.0-RELEASE-x64 (ab098f4)

 

[MoR]

Is there recovery software like GetDataBack for Oracle/ZFS volumes? I'm still new to the oracle/freenas stuff.

 

[danb35]

No, there is no recovery software. Your problem has nothing to do with the shares, your problem is that the volume itself is unavailable. Try dropping into the shell and running (without the quotes) "zpool import", and post the results here.

 

[MoR]

I actually tried UFS explorer which is a recovery program. Picked up the volume but didnt seem to do anything for file recovery stage. Strange. Tried zpool import. It did nothing. Thought about it for 5 seconds and then brought back the prompt line.

[root@NAS ~]# zpool import
[root@NAS ~]#
[root@NAS ~]# zpool import [root@NA ~]#

 

[danb35]

How about the output of "zpool status" and "camcontrol devlist"?

 

[MoR]

Thanks for the input. Really appreciate this.

[root@NAS ~]# zpool status
pool: Safe
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 0h23m with 904 errors on Sun Jun 29 00:23:10 2014
config:

NAME STATE READ WRITE CKSUM
Safe DEGRADED 12.8K 0 2.46K
mirror-0 DEGRADED 48.6K 0 9.40K
769634960547955908 UNAVAIL 2 145 8.28K was /dev/gptid/c1db9440-7e45-11e3-80d1-10604b92b
b80
gptid/c57fdc05-fc09-11e3-b5f7-10604b92bb80 ONLINE 0 0 58.0K

errors: 904 data errors, use '-v' for a list

pool: Volume1
state: ONLINE
scan: scrub repaired 0 in 1h14m with 0 errors on Sun Jun 22 01:14:53 2014
config:

NAME STATE READ WRITE CKSUM
Volume1 ONLINE 0 0 0
gptid/653ad00c-7ce7-11e3-a42d-10604b92bb80 ONLINE 0 0 0
gptid/65a10618-7ce7-11e3-a42d-10604b92bb80 ONLINE 0 0 0

errors: No known data errors

[root@NAS ~]#
[root@NAS ~]# camcontrol devlist
<WDC WD20EFRX-68AX9N0 80.00A80> at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD20EFRX-68AX9N0 80.00A80> at scbus1 target 0 lun 0 (ada1,pass1)
<ST31000528AS CC35> at scbus2 target 0 lun 0 (ada2,pass2)
<ST31000528AS CC38> at scbus3 target 0 lun 0 (ada3,pass3)
<TOSHIBA TransMemory PMAP> at scbus7 target 0 lun 0 (pass4,da0)
[root@NAS ~]#

 

[danb35]

It appears that both of your mirrored drives have experienced errors, resulting in data corruption in that pool. zpool status -v Safe should give you a complete list of the corrupted files; the remainder of the contents of the volume should be safe. However, you're in a very bad way with respect to that volume at this time. Since it does appear to be online, I'd try to copy as much as you can to Volume1. It looks, though, like Volume1 is just a striped pool, which is very dangerous.

 

[MoR]

Yeh volume 1 is striped and thats ok. Its just data that i can loose, The Safe drive, which is mirrored and the one I;m having trouble with is still online in a degraded state. Its not acceesible though. Thats my problem. It tells me that access is denied.

 

[danb35]

Can you access files on Safe from the command line?

 

[MoR]

If you mean from windows cmd prompt no. If you mean from freenas.......how do you do that?

 

[danb35]

I do mean from the FreeNAS command prompt. Start with

# cd /mnt/Safe
# ls

Does it list your files/directories? If so, that's a good start. Try to copy a file:

# cp somefile /mnt/Volume1

You might want to make a directory under Volume1 for this stuff:

# mkdir /mnt/Volume1/SafeBackup

Then, to copy everything there:

# cp -r * /mnt/Volume1/SafeBackup

There are likely to be some errors, but this should get at least some of your data copied to a healthy pool.

 

[MoR]

Hmmmmmm it didn't work. This is starting to worry me.

[root@NAS ~]# cd /mnt/Safe
[root@NAS /mnt/Safe]# ls
[root@NAS /mnt/Safe]# cp -r 8/mnt/volume1/Safebackup
usage: cp [-R [-H | -L | -P]] [-f | -i | -n] [-alpvx] source_file target_file
cp [-R [-H | -L | -P]] [-f | -i | -n] [-alpvx] source_file ... target_directory
[root@NAS /mnt/Safe]# cp -R 8/mnt/volume1/Safebackup
usage: cp [-R [-H | -L | -P]] [-f | -i | -n] [-alpvx] source_file target_file
cp [-R [-H | -L | -P]] [-f | -i | -n] [-alpvx] source_file ... target_directory
[root@NAS /mnt/Safe]# cp -r * /mnt/Volume1/SafeBackup
cp: *: Device not configured
[root@NAS /mnt/Safe]#

 

[danb35]

Yes, this is not looking good. Do I read your post correctly that "ls" returned nothing? That's not a good sign at all.

 

[joeschmuck]

I don't think he put in all the stuff he typed, well I hope not. Yea, if ls does not show anything in the /mnt/Safe/ location, well you may not have any data at all.

 

[cyberjock]

Oh boy.. it sounds like your data is gone. :(

 

[MoR]

Ah crap! Hmmmmmm this is going to cost me. Time to find some data recovery nerds in Melbourne.

 

[rs225]

Since you have a lot of errors on two drives, there is a chance that what you really need to try is connecting the drives to an entirely new piece of hardware. Your problem might vanish if you have a flaky power supply, controller, or cables.

 

[gpsguy]

You never really told us what the results of running ls were. Did you see any files/directories? If so, could you cd into them and look around?

[root@NAS ~]# cd /mnt/Safe
[root@NAS /mnt/Safe]# ls