Need help on zpool import

[drucker10]

Hi all,

i use Freenas 9.2.1.5 for a home NAS solution. Since the storage ran out of space (due to my VMware labs) i decided to grow the raidz2 pool (six disks in one vdev) from 2GB to 6 GB each.

I raised the ZFS Version from 28 to 5.00

I took the first disk offline, replaced it and resilvered the new one. I did with the following four disks the very same.
One thin to mention: Every resivering lasted round about 10 to 11 hours. Except no. 5. This on took nearly 17 hours.

I do not have any SMART or similiar Jobs configured. I start them manually on an not regular schedule.
So there should be nothing that interfere with the resilvering.
I took the last disk offline. Shutdown the system. Replaced the drive and fired up the system.
It lasted a long time for reboot (6 or 7 Minutes).
Now the pool has gone!

I tried different things:
First i assummed some cablein issues. So i replaced the sata cables.
I changed plugged the sata cables in different ports on the mainboard.
Then i used an SATA expansion card.
Then i replaced the mainboard with the very same one from my ESXi Host.
Nothing changed.

Then i unplugged the fifth disc which needed the extra long resilvering.
Now th system boots as expected. But still the zpool is not visible.

Code:

[root@freenas] ~# zpool status
no pools available
[root@freenas] ~# zpool import
   pool: Vol_1
     id: 17651771559917634832
  state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

        Vol_1                                           UNAVAIL  unsupported feature(s)
          raidz2-0                                      DEGRADED
            gptid/06b40e3f-517c-11e4-b52e-bc5ff4fdaaba  ONLINE
            gptid/af270422-524f-11e4-84ca-bc5ff4fdaaba  ONLINE
            gptid/2c3f775b-51e1-11e4-aba0-bc5ff4fdaaba  ONLINE
            13925070240241922921                        OFFLINE
            gptid/d69e9b02-53ce-11e4-9f14-bc5ff4fdaaba  ONLINE
            16997931041313570629                        UNAVAIL  cannot open

The OFFLINE Disc might be the sixth one i took offline.
The UNAVAIL the fifth, which had the resiverling issues.

While booting i saw the following messages while freenas tried to import Vol_1

Code:

unsupported feature: com.delphix:hole_birth

Shall i use the

zpool import -f Vol_1

Is this the correct command?


Can you please help me to get the pool operational again?

Many thanks in advance and special thank for all others who posted their questions and those sharing their valuable knowledge. I have learned some things in here. But still enough to learn.

 

[SweetAndLow]

Don't use that command yet! Wait till someone who knows what they are doing replies to this thread.

 

[drucker10]

Hi SweetAndLoud,

i will wait. It was 2 o´clock AM when i wrote the message. Now i am on my way to work.

Hopefully someone can ghelp me.

Cheers,

 

[cyberjock]

No, the problem is that your pool has feature flags enabled that are not compatible with the FreeNAS version you are using. In essence, the zpool was upgraded to support a newer version of FreeNAS, then you installed an old version.

You won't be able to mount the pool with the version of FreeNAS you have.

And what you are saying seems nonsensical to me. You had 2GB drives and you increased them to 6GB. Too bad you need drives bigger than 2GB just for swap space to be created.

So either you are very confused, your saying things that are incorrect, I'm not understanding what you are writing, or a combination.

Read the FreeNAS manual warning about "zpool upgrade" as that's where things went bad for you. No clue what you are trying to do after that because you're clearly doing some kind of experimenting (or something) and not giving all of the information.

 

[drucker10]

Hi Cyberjock,

thanks for the reply. You are right, some of my infos may be misleading or confusing. (It was really late. Thats why i stopped at this point)
And you are right too, i experimented to much.

I started with the raidz2 (6 x 2 GB WD Red Drives) on Freenas 9.2.1.5. The raidz2 had the ZFS Version v28.
Then i planned the drive upgrade to 6 x 6 GB Red Drives.
I read the Freenas Docs and tried to get the first Disk offline through the WEB GUI.
This failed and i searched around the forum and found the workaround using the commandline.
There i found some Information in the zpool commandline stating that i should upgrade the ZFS Version. So i followed this, cause i did not intend to switch back the FreeNAS Version.

Then i started with the disc replacement.
- Offline the disc from command line.
- Powered off the Server.
- Replaced the disc
- Started the Server
- Replaced the Disc in the Pool via Web GUI.

I did this for ever disc and everything went fine until i changed the fifth one, which neede much more time to resilver. (round about 17 instead of 10 to 11 hours)
As usual i checked the resilvering process with the "zpool status" command and it stated successfully finished.

Then i took the last disc offline. Shutdown the system and replaced the last 2.0 GB disc with the 6.0
I rebooted the system. In The Web GUI i have seen the status: UNKNOWN

The zpool command showed: No zpool availiable.

So i started diggin for the error.

First i detached the lastly insterted Disc from the power and data cable.
No Luck.

In Short i also tried to:
- switch the sata connection of the Harddisks to six differnt on the mainboard.
- Insterted an PCI Express four Port sata Controller

To make the long story short i went more in the direction of a hardware issue.
So i changed the Mainboard to ther very identical one: The Same Board, the Same BIOS Verision.

The FreeNAS boots from an USB Stick and the FreeNAS version was not changed.

Actually i "only" replaced the Mainboard. An Asrock D2750 with an Asock D2750.

Then is started looking into the commandline tools. Read many forum posts and finally got an answer from the zpool import command.

Code:

[root@freenas] ~# zpool import
   pool: Vol_1
     id: 17651771559917634832
  state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

        Vol_1                                           UNAVAIL  insufficient replicas
          raidz2-0                                      UNAVAIL  insufficient replicas
            9444362728414954018                         UNAVAIL  cannot open
            gptid/af270422-524f-11e4-84ca-bc5ff4fdaaba  ONLINE
            gptid/2c3f775b-51e1-11e4-aba0-bc5ff4fdaaba  ONLINE
            13925070240241922921                        OFFLINE
            gptid/d69e9b02-53ce-11e4-9f14-bc5ff4fdaaba  ONLINE
            16997931041313570629                        UNAVAIL  cannot open

So i started to gather the information of the availiable gptid discs with the camcontrol command and could verify that these are ada0, ada1 and ada2
The offline i know is ada4 (the 2.0 GB disc i wanted to replace)
The other in the status UNAVAIL must be ada5 and ada3

To get the last peice of information: The replacement of the discs was done in the following order:
ada5
ada0
ada1
ada2
ada3
ada4

So i unmounted the fifth disk (6.0 GB - resilvering successfully finished) an dreplaced with the 2.0 GB (replaced) disk.

Now i got:

Code:

[root@freenas] ~# zpool import
   pool: Vol_1
     id: 17651771559917634832
  state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
   see: http://illumos.org/msg/ZFS-8000-EY
config:

        Vol_1                                           UNAVAIL  unsupported feature(s)
          raidz2-0                                      DEGRADED
            gptid/06b40e3f-517c-11e4-b52e-bc5ff4fdaaba  ONLINE
            gptid/af270422-524f-11e4-84ca-bc5ff4fdaaba  ONLINE
            gptid/2c3f775b-51e1-11e4-aba0-bc5ff4fdaaba  ONLINE
            13925070240241922921                        OFFLINE
            gptid/d69e9b02-53ce-11e4-9f14-bc5ff4fdaaba  ONLINE
            16997931041313570629                        UNAVAIL  cannot open

Something went wrong with the resilvering of the fifth disk. The "old" guid of the 2.0 Gb Harddrive was still listed.

The actual status of the sysem:

The very same FreeNAS version is used all the time.
The Mainboard is actually replaced with a board from the same manufactuer, the same model and the same BIOS Version.
(Maybe this causes some errors now)
The pool Vol_1 replies with four online and one Offline disc. Which cannot be changed to "Online" since the zpool command does not know the pool.

Maybe i should revert the Mainboard swap and check again.

Is there any command i could issue to get more and reliable information?
Actually i am on work and cannot revert the Mainboard Swap. But i am connected with my environment and i can dig for information.

I hope this is a better descripton.

@ Cybejock
Many thanks for your efforts.

 

[cyberjock]

You've done so many steps that I'm not sure I can even help you.

Few things:

1. You said you swapped 2GB with 6GB. Again (and I said this earlier), I have no doubt in my mind that you do NOT have 2GB and 6GB disks. You might have 2TB and 6TB drives, but you *definitely* do NOT have 2GB and 6GB drives. Errors like this, while seemingly minor, can really derail this process. So you need to be 100% sure that what you want to say is what you are saying. If you don't you may find you will never see your pool again.
2. The error you got "unsupported feature: com.delphix:hole_birth" is NOT going to happen on any 9.2.1 release. So I *know* you were *not* using any 9.2.1 release at some point. If you are going to tell me that for 100% certainty that you were using 9.2.1.x, then your/my work is done because your pool is so corrupted there is no way you're going to see your data again.
3. You offlined the disk from the command line. This is a major no-no. If you follow the FreeNAS manual, there is no step anywhere in disk replacement that involves you doing anything from the CLI. So this was a major failure on your part.
4. You said:

Something went wrong with the resilvering of the fifth disk. The "old" guid of the 2.0 Gb Harddrive was still listed.

That's not a failure with resilvering. That's improperly removing and replacing disks (see #3... this is an error that is likely caused by using the CLI).

Now, to sum this all up, I am still totally baffled at what you did (or didn't do). You've clearly made some serious mistakes, mistakes that may be uncorrectable. I have zero advice because (and forgive me for being honest) you seem to be doing multiple things wrong and I have no doubt you probably did more things wrong than you realize which is adding to my confusion with trying to put the whole story together.

At this point though, I'm going to have to say "I have no advice" because I can't assemble a series of steps in my head to get to where you might be. Unfortunately, as soon as you start doing CLI, there's an almost unlimited number of ways to break a pool. This is why you shouldn't be doing zpool manipulations from the CLI, ever. FreeNAS is usually easy to diagnose because the WebGUI has very specific steps and behavior for various conditions. You've broken that logic and, despite having FreeBSD CLI experience, I'm not getting how you could start with a pool with 6 working disks and, through a series of missteps from the CLI, somehow end up where you are. There's missing steps and information that is critical to identifying the exact problem and how to undo the mess (assuming you can).

If someone else has a clue, they are welcome to pipe in and give it a go. But as it sounds right now, not only is your pool unrecoverable, it seems to be the result of either multiple hardware failures, use of the CLI when it was inappropriate, or a combination of both. I have nothing to offer except to take all of your disks, attach them *all* to your system at the same time, and see if a zpool import is possible. If not, I think it's game over.

Sorry.

 

[drucker10]

Hi Cyberjock,

many thanks for the honest words. Partially i can agree with you.

1. You are right. I am refferring to 2 TB and 6TB (not GB). I should better focus on my words.

2. As i wrote my answer and tried to validate the versions i have seen the in my Browser Hostory i see:
FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
Now in the Browser is see: FreeNAS-9.1.1-RELEASE-x64 (a752d35)
So i was using 9.2.1.5 but now it shows 9.1.1.
In conclusion you are right. I now use (however) an earlier version of FreeNAS.
But i use the very same USB Stick to boot the FreeNAS. I did not change anything

3. I ran into this issue: https://bugs.freenas.org/issues/5035
There the workaround was to offline the disc via commandline.

4. I agree that i made some mistakes due to less knowledge and misbehaviour in troubleshooting.
But nevertheless i am willing to learn something and try to get the volume back again.

Thanks for your time and long answer. This is really highly appreciated. Since i know what you are talking about. I did a long time support for network equipment.
but in these areas i feel home. In the Storage area i am a newbie.

So due to your persist questions i got some valubale hints.

Can the next step be an upgrade of the FreeNAS? Or better prepare a new USB Stick and import the volume?
Hmm, what caused the system to boot with an older Image?

I am also confused now.

Thank you very much so far.

Cheers,
Frank

Again, many thanks for your honest words and open mind.

 

[drucker10]

Dear all,

thanks to cyberjocks valuable hints i got it.

What have i done?

I made a RAW copy of my USB-Stick (FreeNAS Bootdev).
Then i detached all SATA Cables from the Mainboard
Booted the System with an 9.2.1.5 CD Image and made an upgrade.
After checking that all boot errors i had before were gone i shutdown the system and
attached all drives (except faulty ada3) again.
Now the zpool is operational with four disks.

@cyberjock
Many thanks for your valuable hints.

Please mark this as solved.

Cheers,
Frank

 

[cyberjock]

1. Glad I got that right. I was very confused because nobody should be using 2GB drives on FreeNAS. ;)

2. I have no clue how you ended up with an older version. Nobody has ever "suddenly" been on an older version without installing an older version of FreeNAS on the USB stick. This is one of the inconsistencies I'm getting with your story. Not that I think you are lying or anything, but it makes the story weird and nobody has ever "suddenly" been on an old version of FreeNAS. To make matters worse, your config file is probably messed up because you've gone "backwards" in versions. A bad config file wouldn't cause your import problems though. I'd recommend installing 9.2.1.8 to a USB stick and see if you can import it. If you can't get enough devices to be available to mount through the auto-mount feature, you still won't be able to access your files.

3. I've seen that issue once or twice. Not something that we've been able to reproduce and not fixed according to the bug.

4. I have no advice to give you to try to get the pool back. If you can't import the pool with 9.2.1.8 with all of your disks attached there isn't much hope, and I wouldn't even know where you'd go from there.

 

[cyberjock]

I see you and I posted within seconds of each other. Glad you got it sorted out.

 

[drucker10]

Dear all,

i tried to get some of the issues sorted after this disaster.
Here are my (small) results:

I do not have any clue why the FreeNAS Version reverted to 9.1.1.
I started my FreeNAS experiences with 8.3.1. Then upgraded to 9.1.1, which was finally followed by 9.2.1.5.
As you see i am not a version-number-junkie.
All i know is the following: I used to work with 9.2.1.5 and then it reverted to 9.1.1.
After all the trouble fade away and i upgraded to 9.2.1.5 again i recognized tht ligin user changed from admin to root again.
And then i realized that in the night the trouble began i was not able to login to FreeNAS. Because i tried the user root as usual and failed - until i changed to the user admin.
Really strange.

Regarding the harddisk failure. I connected the disk to a different PC and boot this with Knoppix from DVD.
I started gparted and it stated a failure in the GPT backup table.
Every change i made on the disk it nagged my ear of with this error message.
Now i rewrote the Partition Table and the disk is fine again.
Now i run a long SMART Test on this disk. Just to be sure.

Whatever happened to this disk i am not sure.

For now everything runs fine again and i need to replace the last 2.0 TB Drive in the zpool with the last 6.0 TB.

Thanks to Cyberjock to lead me in the right direction.
And many thanks to all FreeNAS Devs for assembling and maintaining this distribution.
Keep on running.

Cheers,
Frank