Pool not detected anymore Error getting used space (Error)

[o0larry0o]

Hi everyone,
I'm new here !! :)

I've been using Freenas 8 RC4 (I know i'm a little late .. but I never had the guts to upgrade since the disk where full of data and i really needed it to be up and running quickly) for quite a while, with no problems, It's build on an ESXi server
Lately I could not reach my CIFS share .. tried to reach the webGUI ...unreachable, tried to reboot the system within vSphereClient no luck. So I did a reset in vSphere ....(the hard way then) and the system went back to normal :)

Today same symptoms, tried the first 2 solutions same results so I reseted the VM, like last time.

and I could still not reach my CIFS share....so I logged back into the webGUI and this I saw :


additionaly I saw that I have the GEOM: da0s1: geometry does not match label da0s1 being my virtualdisk where Freenas is installed (dunno if it can be related, from what i've seen on other thread maybe not)

Well starting from there I read this thread :
http://forums.freenas.org/showthread.php?1764-Error-getting-used-space-(Error)

Trying an auto import on the webGUI do nothing, there is a drop down menu in the popup that is empty (i guess it should show the pool I'm looking for)
when I looked i /var/log/messages, here is what i see just after hitting the auto import button:


any zpool history or list gives me "no pools available"

a Zpool import gives me no error, but nothing happens in the GUI I still have my errors.

at the end of this thread there is a link to another one
http://forums.freenas.org/showthrea...Issues-Space-Error&p=8448&viewfull=1#post8448

I first check if i I had a "RAIDZ" file in /mnt and there were none.
So I gave it a try too, and i'm having the same issue as the last post of the thread being
doing mount -uw / works
but then the zfs set mountpoint=/mnt RAIDZ (RAIDZ being my pool) gives me 'cannot open RAIDZ: dataset does not exist'

last thing, My pool is mabe of 3 1TB HDD, all are detected by freenas at boot (dmesg) and all three are seen when I click on the "view disk" button of my primary volume in the GUI.

My whole life is in this box please gimme some help, I can't afford to lose everything.

Thanks in advance.

 

[ProtoSD]

Hi Larry,

Don't panic, your data should still be there, it's just a GUI/database sync error.

Have a look a this blog post, specifically the section on "Commands To Reset The Database":

http://protosd.blogspot.com/2011/12/useful-commands-for-diagnosingmanaging.html


If you're still lost, I'll try to point you in the right direction some more, but I'm about to leave town in a few days and things are a bit hectic.

 

[o0larry0o]

ahh thanks i'll "try" to read your blog on my phone... blogs are somehow "websensed" at work as I like to call it.

I've been reading all morning and seen your post everywhere I knew you would come to rescue me :)

 

[o0larry0o]

Okay so I read your blog, and I will keep you posted later today when i'll get my hand on my machine.

Basicaly what in your experience could cause that ? and how to resolve it ? as I didn't change any hardware or nothing ?
could it be a mountpoint problem ?

 

[ProtoSD]

There are so many possible things that could have caused it, it would be hard to guess which one.

Possibly the easiest thing to try would be to create a completely new USB installation of FreeNAS and boot from that, then try the Auto Import. If that solves your problem and you can access your data again, then it's a matter of repairing the database or re-entering your settings manually.

I'm off to sleep, I'll check back and see what you've discovered later.

Cheers!

 

[o0larry0o]

Good night then ^^

Last question that your probably will answer tomorow,
If I build a new install and the auto import work, what will I loose from the old database ? Users ? quota ? datasets ?

thx

 

[ProtoSD]

Good night then ^^

Last question that your probably will answer tomorow,
If I build a new install and the auto import work, what will I loose from the old database ? Users ? quota ? datasets ?

thx

Good morning! ;)

Yes, you will lose the settings with a new install, but the idea is test that your data is ok and that Auto Import will work. You can still switch back to your other USB device and fix the data base after testing this.

 

[o0larry0o]

Okay here I am at home again.

So as i'm using VMware i created another vm for a new instance of the lastest freenas x64 version
I'm using my 3 1T HDD as RDM(raw direct mapped) so freenas as direct access to the disks.

Annnndd ...using the auto import from the GUI shows me nothing in the drop down list ...
so I tried a couple of command from your blog and here are the output:

[root@freenas] ~# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (da0,pass1)
<ATA SAMSUNG HD103SI 1AG0> at scbus2 target 1 lun 0 (da1,pass2)
<ATA SAMSUNG HD103SI 1AG0> at scbus2 target 2 lun 0 (da2,pass3)
<ATA SAMSUNG HD103SI 1AG0> at scbus2 target 3 lun 0 (da3,pass4)

Looks ok to me

[root@freenas] ~# sysctl kern.disks
kern.disks: da3 da2 da1 da0 cd0

also ok

[root@freenas] ~# gpart show
=> 63 16777152 da0 MBR (8.0G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 12872223 - free - (6.1G)

=> 0 1930257 da0s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)

and not very ok to me and I don't really understand what I see here
appart from :
63 16777152 da0 MBR (8.0G) is my VM disk

the rest is very blured to me ... and i don't understand the => 0 1930257 da0s1 BSD (943M)
as 943M is one on my 1TB HDD but da0s1 sounds like the VMdisk which is not 943M obvisouly

before going any further can you give me your opinion ? and what could I do next ? are there other command line I could check to gather more informations?

many thx

 

[ProtoSD]

Hmmm, you're right, things don't look too good at the moment. The da0, da0s1 isn't a problem, da0s1 is a partition on your 8GB virtual system disk.

The part that concerns me is that there is no partition information on your 3 Samsung disks, which is why Auto Import isn't working. At the moment I don't have any suggestions.

Hopefully someone else here will have some idea. I'm sorry, but I would probably start to panic... :(

 

[o0larry0o]

I never loose hope, the first time I had a RAID issue it took me 6 month to recover but I did it lol !!! or mostly :)

I read some very interesting blog and send emails to their authors, 3 seems to be very advanced in ZFS. So I'll see.

I still cannot understand why the partition table are missing, but tools exists for FAT and NTFS to recover the partition table so i bet it must be possible to do so on a GTP partition table.

In the meantime i'm trying to play around with PC BSD (that you mention in another thread) and i'll see if ZFSv28 could help me more.

If any idea pops up in your mind please let me know:)

 

[o0larry0o]

Hi I hope some other people will have some insight on this matter.
As i need to understand thing before getting started, I started to read a bit of real technical stuff about GPT and ZFS, and googling a bit lead me here :
http://forums.freebsd.org/showthread.php?t=20941

This guy knows his homework about ZFS and GPT, and had the EXACT same problem.
Before trying to get in touch with him, I would to know how exactly how FreeNas 8 proceed to start working with the drives, from the start !!!

lets say I just plug 3 new drives, and go create a RAIDZ.
when and how is the GPT written ?
I've read on the thread I'm mentioning that it could have 2 GPT structure, one on each HDD, and one for the ZFS filesystem, is that true ? if yes what does contain the "ZFS GTP" ?

Thanks for your help


If I could have a very detailled answer maybe it could help me to recover my GTP partition and get back my pool !!!!

 

[paleoN]

This guy knows his homework about ZFS and GPT, and had the EXACT same problem.
Before trying to get in touch with him, I would to know how exactly how FreeNas 8 proceed to start working with the drives, from the start !!!

Really? From what I read sub_mesa had a corrupt GPT label plus what looked to be a corrupt ZFS label inside the partition. ZFS v28 was able to work around that for him though.

What do you see when you run the following on the original VM:

Code:

camcontrol devlist
sysctl kern.disks
gpart show

Also, try running

Code:

diskinfo daX

For each drive in both new VM & old one.


See this [post=28362]post[/post] for some ZFS v28 live CDs.

 

[o0larry0o]

thx for your reply here what i got

Code:

[root@freenas] ~# camcontrol devlist
<NECVMWar VMware IDE CDR10 1.00> at scbus1 target 0 lun 0 (cd0,pass0)
<VMware Virtual disk 1.0> at scbus2 target 0 lun 0 (da0,pass1)
<ATA SAMSUNG HD103SI 1AG0> at scbus2 target 1 lun 0 (da1,pass2)
<ATA SAMSUNG HD103SI 1AG0> at scbus2 target 2 lun 0 (da2,pass3)
<ATA SAMSUNG HD103SI 1AG0> at scbus2 target 3 lun 0 (da3,pass4)

Code:

[root@freenas] ~# sysctl kern.disks
kern.disks: da3 da2 da1 da0 cd0

Code:

[root@freenas] ~# gpart show
=> 63 16777152 da0 MBR (8.0G)
63 1930257 1 freebsd [active] (943M)
1930320 63 - free - (32K)
1930383 1930257 2 freebsd (943M)
3860640 3024 3 freebsd (1.5M)
3863664 41328 4 freebsd (20M)
3904992 12872223 - free - (6.1G)

=> 0 1930257 da0s1 BSD (943M)
0 16 - free - (8.0K)
16 1930241 1 !0 (943M)

This is only from the old vm, i wasn't able to play along with another version of FreeBSD, I tried PCBSD but i took forever to install ...well it never finish basicaly.
But I had the same output with FreeNAS 8.0.2

I have some doubt with my VM and RDM technic I think my next step is to bypass VmWare and use FreeBSD as primary OS so it can have a real direct access to the disk.

What you are describing from the other forum looks exactly the same as my problem (look at my output from gpart show). I didn't have the time to see in dmesg if I have the same message as he does

Code:

 GEOM: ada5: corrupt or invalid GPT detected.
GEOM: ada5: GPT rejected -- may not be recoverable.

didn't have the time to run diskinfo either.

Could you give me some answer on the question I asked ?

----------

lets say I just plug 3 new drives, and go create a RAIDZ.
when and how is the GPT written ?
I've read on the thread I'm mentioning that it could have 2 GPT structure, one on each HDD, and one for the ZFS filesystem, is that true ? if yes what does contain the "ZFS GTP" ?

Thanks

 

[o0larry0o]

So I'm back in this business !!
Now I have done a full raw copy of my 3 old HDD on my 3 new ones (same model)

As I understand from this post : http://forums.freebsd.org/showpost.p...8&postcount=12

I would need to:
- writes zeroes to the first 1MiB of the disk
- creates a new GPT partition scheme
- creates a freebsd-boot boot partition
- creates a data partition spanning the rest of the drive minus 1MB of the GPT partition table and freebsd boot partition

After that, ZFSv28 would be able to see my ZFS file system and import my ZFS pool.

According to this post here : http://forums.freebsd.org/showpost.p...7&postcount=12
To do so I would need to do :

- gpart create -s gpt dax
On each drive ?
- gpart add -t freebsd-boot -s 64k dax
On each drive again ?
- gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 dax
I don't understand this command, do I need it ? can you explain it to me ? I have the feeling that it's to make the disk bootable, if so I guess I don't need it cause I don't boot on them, they are only used for storage
- gpart add -t freebsd-zfs -l disk0 -b 2048 -a 4k dax
On each drive again ? Is the label important ? I mean, if my old disk didn't have any would it make a different in the process ? and I couldn't find any entries in the gpart man page about the -a option, what is it used for ?

I just thought that I might have missed a step in my process.
Maybe I could use my 3 new HDD to start a new configuration in FreeNAS to check the partitionning scheme and to have a reference to copy from. To be sure of the lenght of each partition etc. Because maybe FreeNAS does thing differently than a manual configuration on a basic FreeBSD system. I'm wondering if the size of the freebsd-boot is the same (64k), and if the freebsd-zfs starts at the same position ?
Does it make any sense ?

Thanks in advance for your replies.

Ludovic