Lost ZFS - How can I import a "ghost" pool

[TonyNAS]

Hello everybody :)

PREFACE (optional)

Firstly I want to thank you for your cooperation in solving the problems of our community. Using an open-source OS transform us from users to developers with the inherently responsibilities

INTRODUCTION

After the upgrade to FreeNAS 9.3 I've experienced several problems on my system, after several time-consuming tests I'm in a almost stable functioning. Unfortunately during my experiments I lost an HD a WD 6TB. To be more precise I cannot access to the zpool on that hard drive anymore. If i try to execute the GUI Auto-import function the drive is not present in the list and therefore I cannot mount it.

THE CONTINGENT SITUATION

A clarifying screen-shoot...

I will post some of my outputs to let you know about my problem...

LIST OF PARTITIONS ​

Code:

[root@freenas] /# gpart show
=>      34  15826877  da0  GPT  (7.6G)
        34      1024    1  bios-boot  (512k)
      1058         6       - free -  (3.0k)
      1064  15825840    2  freebsd-zfs  (7.6G)
  15826904         7       - free -  (3.5k)

=>       34  488397101  ada0  GPT  (232G)
         34         94        - free -  (47k)
        128    4194304     1  freebsd-swap  (2.0G)
    4194432  484202703     2  freebsd-zfs  (230G)

=>        34  5860533101  ada1  GPT  (2.7T)
          34          94        - free -  (47k)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5k)

=>         34  11721045101  ada2  GPT  (5.5T)
           34           94        - free -  (47k)
          128      4194304     1  freebsd-swap  (2.0G)
      4194432  11716850696     2  freebsd-zfs  (5.5T)
  11721045128            7        - free -  (3.5k)

=>        63  1250263665  ada3  MBR  (596G)
          63        1985        - free -  (992k)
        2048   641087842     2  vmware-vmfs  (305G)
   641089890       13982        - free -  (6.8M)
   641103872   609157120     1  ebr  (290G)
  1250260992        2736        - free -  (1.3M)

=>        0  609157120  ada3s1  EBR  (290G)
          0  609157120       1  ntfs  (290G)

In particular I want to focus on the ada2

LIST OF ZPOOLS
As you will see WDBigger pool (the 6TB hd) is not present
​

Code:

 pool: Small
state: ONLINE
status: The pool is formatted using a legacy on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on software that does not support feature
        flags.
  scan: scrub repaired 0 in 1h7m with 0 errors on Sun Feb  8 01:07:49 2015
config:

        NAME                                          STATE     READ WRITE CKSUM
        Small                                         ONLINE       0     0     0
          gptid/a9f878b5-e8ac-11e2-9c4f-28924a3077fa  ONLINE       0     0     0

errors: No known data errors

  pool: WDBig
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 11h53m with 0 errors on Sun Mar 15 11:53:38 2015
config:

        NAME                                          STATE     READ WRITE CKSUM
        WDBig                                         ONLINE       0     0     0
          gptid/3280fded-00ec-11e3-8e03-28924a3077fa  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        freenas-boot                                  ONLINE       0     0     0
          gptid/1a72aec6-cf46-11e4-a6a1-28924a3077fa  ONLINE       0     0     0

HOW TO SOLVE THE PROBLEM

My question is, how can I let the system recognize the 6TB partition as a ZFS pool, or alternatively how can I recover some very important files and documents from that hard drive.

I remain available for other info you may need. Thank you to all :D

 

[joeschmuck]

First please follow the forum rules and give us some basic information about your system. What did you upgrade from and to (be specific) for the software, how much RAM do you have, list your system hardware such as if you have a SATA controller, MB, etc... The more we know then it will be easier to give you good advice. If you have tried some command line options, specify them (Do not use the command line interface unless someone specifically directs that action because it can easily cause more harm than not if done incorrectly). Also, ada2 is the 6TB drive, not ada1.

 

[TonyNAS]

Of course joeschmuck. You found a "bug" :D (i corrected it in my previous post) in fact I was referring to ada2

I upgraded from 9.2.1.4 to 9.3 through the GUI interface. I upgraded one step for each version, as example from 9.2.1.4 to 9.2.1.5 to 9.2.1.6 to 9.2.1.6 etc... until 9.3

Hardware details -> HP Proliant N54L - dual core 2GB RAM. No SATA or more complicated stuff. It's a simple network storage.Every hard drive is working by itself and there are no striping or RAID complications that should simplify the resolution of this problem... I hope

 

[enemy85]

Well the problem is that u are using the 9.3 with ZFS on just 2GB ram...

 

[TonyNAS]

Ok, I have bought some more RAM... I will let you keep up to date as soon as I upgraded the system.... :D

 

[joeschmuck]

So you have a few options here to recover your data...
1) FreeNAS 9.3 requires 8GB of RAM (4GB might be doable for a single drive just to grab your data but it's a stretch and not recommended).
2) Create a new 9.3 bootable USB flash drive, use any computer with 8GB RAM and that you can install only your 6TB drive into, boot from the USB, auto-mount the drive and save off your data.

And like I always say, backup your important data to another device or storage media, I prefer DVD/CD-R media.

Good luck.

 

[cyberjock]

Hate to break it to you, but you probably killed that pool by using just 2GB of RAM. If this is correct, you'll find that adding RAM won't fix the problem and your data is gone for good. :(

 

[TonyNAS]

Hate to break it to you, but you probably killed that pool by using just 2GB of RAM. If this is correct, you'll find that adding RAM won't fix the problem and your data is gone for good. :(

I hope to resolve the problem by increasing RAM, however if the test doesn't work I have to recover some documents in any way. You are welcome to let me know about the "hard procedure of recovering" whatever it takes

 

[cyberjock]

I hope to resolve the problem by increasing RAM, however if the test doesn't work I have to recover some documents in any way. You are welcome to let me know about the "hard procedure of recovering" whatever it takes

Sorry, but if the "zpool import" doesn't even list the pool, then the uberblocks are not being found, and you need far far more than just "hard procedure of recovering". You need "an act of God" or "5-figures in cash to pay to recovery companies".

There's a reason why I spent my time to write this in my noobie presentation:

• For maximum performance, more RAM is always better. Maximum allowed motherboard RAM is recommended. The resultant improper shutdowns of your system can result in a corrupted zpool that will no longer mount. Don’t be a statistic! Use 8GB+ of RAM with ZFS or use a different NAS solution. I don’t even bother responding to posters that can’t have the minimum hardware and follow recommendations in the stickies.

• This is a fairly common occurrence for many forum users that are new to FreeBSD/FreeNAS and think they can “get by” with less RAM because they are using hardware that doesn’t support 8GB of RAM, 8GB of RAM costs too much, or they are convinced they know better than the manual. Some people get lucky and have no problems, others have lost their zpools. I’d never risk my data by using less than 8GB of RAM and I recommend the same to you.

• If you want to use ZFS with less than FreeNAS’ recommended RAM you should consider the NAS4Free project. NAS4Free recommends 1GB per TB of storage without the baseline 8GB. So if you have 5TB of storage you’d need 13GB of RAM for FreeNAS, but only 5GB for NAS4Free. Note that using NAS4Free does not alleviate the risk of improper shutdowns crashing your system. So don’t lowball your RAM needs.

So as you can see, I don't think too highly of people that can't follow minimum specs, and I definitely don't spend much of my own time trying to help people with problems they created themselves. It's just not worth my time.

 

[TonyNAS]

Sorry, but if the "zpool import" doesn't even list the pool, then the uberblocks are not being found, and you need far far more than just "hard procedure of recovering". You need "an act of God" or "5-figures in cash to pay to recovery companies".

There's a reason why I spent my time to write this in my noobie presentation:

So as you can see, I don't think too highly of people that can't follow minimum specs, and I definitely don't spend much of my own time trying to help people with problems they created themselves. It's just not worth my time.

I can understand your position about this issue, you have seen a multitude of these cases and you are truly right when you say "DON'T BE A STATISTIC". If I can say something in my defense I would say that I didn't read the document before this problem and the NAS worked very well before the upgrade to the version 9.3 even with 2GB of RAM.

As you describe in this post ZFS, it seems to be an unreliable file system because it suddenly stops working. I can understand a working limitation for what concern RAM but I cannot be flexible on data security. A simple NTFS usb HD would be more reliable and this is unacceptable discrepancy for a NAS server.

:( Now that I released my anxiousness about data loss, I'm still hopeful about my test with new RAMs. Cyberjock I ask your help in case I cannot reach a solution that way. You can help not only me but a huge mass of "STATISTICS" as you said. So please be kind at least for this time....

 

[joeschmuck]

This is another statistic in why we need the FreeNAS software to do a RAM capacity check before an install/upgrade but it was shot down many times. I'm not sure that I agree that before a user upgrades to a newer version of FreeNAS, that they must check the minimum specs again. But if there is a remote chance that the upgrade could cause data loss, there should be some strong warning before the file system is mounted. I'm sure there are many ways to implement this but it should be implemented but that is just my opinion and you know that every one has one :p.

 

[Bidule0hm]

2 GB with ZFS is like using bicycle tires on a car and then saying "this car is unreliable"...

ZFS is very reliable (it's the whole point of ZFS...), but only if you follow the minimum requirements.

 

[TonyNAS]

Guys, I decided to sacrifice my data for this time. This was a good lesson to motivate me to backup what is very important to me and not to rely too much on network storage. I think I will return to the old usb key to make back-ups.

However I have another problem now... I installed 16GB of RAM for the happiness of those guys who love RAM and the happiness of ZFS and I though that increasing the RAM would kick out all my problems of storage but unfortunately I was wrong. After the update to the 9.3 version my CIFS share doesn't work anymore. Before I changed the RAM I though that the cause was too little RAM but now I still have the same problem even though I've got 16GB of RAM. What is the problem ? I will give whatever info can help you to resolve this problem. Thank you

 

[SweetAndLow]

Your cifs share not showing up is a configuration issue. Make sure your permissions are correct the the correct users can access the data then your sharing should work.

 

[TonyNAS]

I tried to change the permissions, I double checked them... CIFS still isn't working therefore the problem doesn't seem to be explained by those ones

 

[TonyNAS]

I have installed the latest .ISO of 9.3. Mounted one only hard drive to test the CIFS share... It still doesn't work... Have a look to the following screenshots:

HERE YOU CAN SEE THAT THE CIFS SHARE HAS NO OPEN PORTS

Please note that other CIFS shares of another NAS server on the same subnet are working very good so it's not a problem of router... I followed the procedure written on the FreeNAS guide but I was unsuccessful, please let me know where I can look for to find the problem. Thank you

 

[TonyNAS]

I resolved the problem, after a lot of hours changing settings I have resolved this very dramatic event.... I still don't know why I had that problem...

 

[Ericloewe]

Could be that the config file was itself borked by the insufficient RAM. I've begun recommending a new config whenever people have mysterious issues that sound like low RAM, after they add more RAM. If that doesn't help, the old one is still backed-up somewhere (right...? /wishful thinking), so nothing was really lost.

 

[david kennedy]

2 GB with ZFS is like using bicycle tires on a car and then saying "this car is unreliable"...

ZFS is very reliable (it's the whole point of ZFS...), but only if you follow the minimum requirements.

Similar to joeschmuck's post. If there are minimums that will cause "bad things" like total data loss if you go below them, why does the OS not enforce them?
Shouldn't be hard to do a quick mem check and throw a message if there is not enough memory to proceed. Many OS installers do this check and wont move forward if the minimum isnt met.

There are all sorts of checks in Freenas, like making sure you have the often quoted "optimal drive" config for a pool (subsequently removed)

Sometimes it is the users fault, sometimes the systems can do more to prevent it.

 

[Bidule0hm]

There's already a thread on this point ;) Note that in theory the administrator should knows what a system need and do the right choices to maintain it healthy.

The problem with adding system checks is when the administrator knows what he is doing and wants to do something the system doesn't allow. I think the best is to add the check but let the possibility to bypass it :)