Key drive went bad, I think? no pools found RAIDZ1

[Chris5033]

I should have an existing RAIDZ1 pool 3x3tb, but I believe that my key drive may have gone bad, and now I cannot get it to show up on a fresh install of Freenas after the server would not reboot with that drive.

I started receiving errors to my email last week:

Code:

Cron <root@AmpEnergy> /bin/sh /root/save_rrds.sh

Segmentation fault (core dumped)

They repeated every hour for 29 hours before I attempted to reboot the machine by pressing the power button on the front of the PC assuming it would initiate proper spin down, but this may be my flaw? Would not like to think that. The machine did not boot, the config seemed to be corrupt and threw an error, and now the flash drive just boots to a very genric freenas boot screen that says it cannot boot.

So I did some research, got a new flash drive and installed the same version of freenas. Booted to that, but now cannot find the zpool using the autoimport tool. Nothing on the list. Nothing for zfs import in command line, or any of the variants I could find to try and import it.

I only setup the configuration 2 months ago, and have made the ill fated mistake of not having a proper backup yet, as I was finishing up the process of migrating things TOO it, and severely underestimated the power of data loss, even while trying to mitigate that very problem. Which is irony I suppose. but mostly stupidity.

I found an old topic that asked for a gpart show, which does show all three drives.

Code:

[root@freenas ~]# gpart show                                                                                                       
=>        34  5860533101  ada0  GPT  (2.7T)                                                                                        
          34          94        - free -  (47k)                                                                                    
         128     4194304     1  freebsd-swap  (2.0G)                                                                               
     4194432  5856338696     2  freebsd-zfs  (2.7T)                                                                                
  5860533128           7        - free -  (3.5k)                                                                                   
                                                                                                                                   
=>        34  5860533101  ada1  GPT  (2.7T)                                                                                        
          34          94        - free -  (47k)                                                                                    
         128     4194304     1  freebsd-swap  (2.0G)                                                                               
     4194432  5856338696     2  freebsd-zfs  (2.7T)                                                                                
  5860533128           7        - free -  (3.5k)                                                                                   
                                                                                                                                   
=>        34  5860533101  ada2  GPT  (2.7T)                                                                                        
          34          94        - free -  (47k)                                                                                    
         128     4194304     1  freebsd-swap  (2.0G)                                                                               
     4194432  5856338696     2  freebsd-zfs  (2.7T)                                                                                
  5860533128           7        - free -  (3.5k)                                                                                   
                                                                                                                                   
=>      63  15633345  da0  MBR  (7.5G)                                                                                             
        63   1930257    1  freebsd  [active]  (942M)                                                                               
   1930320        63       - free -  (31k)                                                                                         
   1930383   1930257    2  freebsd  (942M)                                                                                         
   3860640      3024    3  freebsd  (1.5M)                                                                                         
   3863664     41328    4  freebsd  (20M)                                                                                          
   3904992  11728416       - free -  (5.6G)                                                                                        
                                                                                                                                   
=>      0  1930257  da0s1  BSD  (942M)                                                                                             
        0       16         - free -  (8.0k)                                                                                        
       16  1930241      1  !0  (942M)     

At this point I'm at a loss, and I'm just not sure why they don't appear, and just suddenly dropped off the planet yet the data still exists! I am at a wall, so I'm desperately asking for some guidance.

 

[DrKK]

Calling @cyberjock.

 

[cyberjock]

Can you post the output of "zpool status", "zpool import" and a debug file on your system?

 

[Ericloewe]

Also important: Please post your FreeNAS version!

 

[cyberjock]

Also important: Please post your FreeNAS version!

You silly guy.... the debug will have that in it. ;)

 

[Chris5033]

First, thanks so much for the quick response. I have been traveling for work the past few days so apologies for the delayed logs.

Second, more details surrounding what happened immediately prior to the issues. One day before I started receiving the error log to my email, I had been trying to move some files around jails, and almost certainly jacked up permissions in the process. I did not think that file permissions would wreck a zfs pool though. I recall making a recursive chmod to a pretty high directory, could that be the culprit and not a dud flash drive?

[root@freenas ~]# zpool import
[root@freenas ~]# zpool import AmpServer
cannot import 'AmpServer': no such pool available
[root@freenas ~]# zpool import AmpEnergy
cannot import 'AmpEnergy': no such pool available
[root@freenas ~]# zpool status
no pools available

(I can't recall the exact name of the pool)

Debug for osinfo, hardware, and system.
https://dl.dropboxusercontent.com/u/1394954/Fiberlink/fndebug.chris5033.11-6.tgz

 

[cyberjock]

File permissions wouldn't be responsible for your pool. A "zpool import" without a poolname should list all available pool on all hard drives and it's not. So there isn't even a hint that there's a zpool on your drives.

I do see three 3TB hard drives on the system, so the drives are clearly there.

Since you have no .system dataset 90%+ of the data I'd normally look at isn't there. Can you provide the output of "gpart list". It could be quite long so you should probably do it over SSH with a very large buffer so you can scroll back and copy/paste the text.

Basically you have just a few possible outcomes:

1. You used UFS (in which case I don't know how to help you because I know nothing about UFS support on FreeNAS... maybe someone else could help???)
2. You destroyed your pool via the WebGUI (we've seen people do some really big mistakes in the past not realizing the consequences)
3. There's some hardware failure resulting in your problem.

What you can definitely rule out right now is the possibility that a broken USB stick is responsible for your pool's disappearance in a direct fashion. But, it could be indirectly responsible if you did an encrypted pool and didn't save your encryption keys. But let's get the output of "gpart list" first.