SOLVED Zpools online, kernel panics when I try to zpool import

Status
Not open for further replies.

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
Hi all,

I needed some advise on what the best solution would be for me to move some or all of my data out of my now corrupted FreeNAS so that I can wipe/format the OS and data drives and build a new clean system.

Here is the hardware we have available to move data to/from:
  1. FreeNAS-main -- 40TB RAW (using RaidZ2 it's about 94% full) - FreeNAS (FreeNAS-11.1-U4) installed
  2. FreeNAS-sandbox01 -- 32TB RAW empty - FreeNAS (FreeNAS-11.1-U4) installed
  3. FreeNAS-sandbox02 -- 32TB RAW empty - No OS installed
Current status on FreeNAS-main system:
  • After multiple hard resets and random shutdowns, the system is now unable to boot up to the main page with 11 options and a webURL to access to GUI.
  • Bootup is halted at some point where I am left with db> prompt.
Here is what I have tried so far:
  • I tried continue and reset options at the db> prompt but it didn't help.
  • Then I tried to boot up the installer USB so that I can select shell from it. Was stuck on "GRUB" screen but was able to resolve the issue with the help of THIS thread
  • Once loaded, I selected #3 "SHELL" and according to THIS thread, I typed in #zpool import and was able to see my raidz2 pool called media with all the drives ONLINE
Here is what I need help with:
  • I would like to import this pool then copy the data onto my new 80TB drives that I have yet to install
  • Is it possible to take my 40TB drives (10 x 4TB) and move the HDDs from FreeNAS-main to FreeNAS-sandbox01 and get it up and running on FreeNAS-sandbox01?
  • This way, I would have FreeNAS-sandbox01 with my old data and I can format/wipe the OS on FreeNAS-main, install (10 x 8TB) and copy/rearrange the data back from FreeNAS-sandbox01 to FreeNAS-main

I tried my best to be as detailed as possible here, but if I missed something or if anyone needs more info, please let me know and I can get that right away.
Any help would be appreciated.

Thanks,
Mehul
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Your data should be fine as you pool is separate from the OS. Save your config and reinstall FreeNAS to your boot device and upload your saved config and you should be back in business.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
Your data should be fine as you pool is separate from the OS. Save your config and reinstall FreeNAS to your boot device and upload your saved config and you should be back in business.
Unfortunately I am not able to access to GUI to backup the latest config file. Is there a different way to back it up?
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
It is backed up if I'm not mistaken and should be located in your pool at /data/freenas-v1.db
 

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
It is backed up if I'm not mistaken and should be located in your pool at /data/freenas-v1.db
freenas-v1.db is the live copy on your boot device (which is also a pool, of course). The backups are at /var/db/system/configs-longhexnumber/.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
I just tried to import the pool using #zpool import -f media and I got stuck at db > prompt again.
Attached is a screenshot of the error.
 

Attachments

  • Error01.jpg
    Error01.jpg
    426.8 KB · Views: 607

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Reinstall the same version of FreeNAS to another boot device and import your pool so you can recover your config from there.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
Reinstall the same version of FreeNAS to another boot device and import your pool so you can recover your config from there.
Installed same FreeNAS on another device and I was able to get to the UI. Logged into the web interface and there were no volumes under storage tab. I click Import volume and selected my volume from dropdown. As soon as I hit next, the UI said "Some error occured" and the console got stuck at attached screenshot.
The system then rebooted and I am back at the db > prompt on the console.
 

Attachments

  • Error2.jpg
    Error2.jpg
    410.7 KB · Views: 585

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
I moved all 10 HDDs to a different system, installed clean FreeNAS on a separate boot SSD and when I boot into it, the system can see the 10 HDDs during boot up but using the "View Disks" options in the FreeNAS GUI does not see any of the 10 HDDs. What am I doing wrong here? Instead of progressing, it seems I am going in reverse :(


Update: Changed the raid controller to JBOD and disks are now showing. The clean install works fine until I try to import the volume, then kernel panic happens and I am back to db > prompt.
Last thing I tried was zpool import -fFn -R /mnt media and panic happened and I am back at db > prompt.

It has now been 14 hours since I started troubleshooting and I have gotten much progress. I am starting Memtest86 and going to bed. Hopefully someone here can provide some advise on what I should try next.

-Mehul
 
Last edited:

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
Changed the raid controller to JBOD
Well this would have been good to know from the start and is one of the reasons the forum rules state to post complete hardware specs when asking for help. If this pool was created using a RAID card you're pretty much out of luck. FreeNAS requires direct access to the disks and a RAID card does not provide that.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
Well this would have been good to know from the start and is one of the reasons the forum rules state to post complete hardware specs when asking for help. If this pool was created using a RAID card you're pretty much out of luck. FreeNAS requires direct access to the disks and a RAID card does not provide that.
I apologize, I should have been more clear about what I did. The main system, we're calling it FreeNAS-main originally had the 4TB x 10 in it. In order to rule out any bad hardware or bad boot device, I physically removed the drives from FreeNAS-main and installed them into FreeNAS-sandbox01. This new machine is the one that I had to change the controller to JBOD to make it work similarly as the main FreeNAS machine.
Theoretically, I should be able to install FreeNAS on FreeNAS-sandbox01 and just import the volume from the old drives, right?
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
What is your system specs?
Specs for FreeNAS-main:
  • 24 Bay FreeNAS Chenbro RM41824 4u Server
  • 2 x Intel Xeon E5-2650 2.0GHz 8 Core CPU
  • 3 x LSI SAS9211-8i Raid Cards
  • 1 x 1200W Power Supply
  • 8 x 8GB Ram DDR3 ECC + 4 x 16GB DDR3 ECC

Specs for FreeNAS-sandbox01:
  • Ci Design NSR316 chassis (3RU rackmount)
  • Supermicro X8DTHiF motherboard
  • 2 x Intel Xeon E5520 2.26GHz 4 Core CPU
  • 6 x Hynix 8GB DIMMs (1066) (48GB total)
  • 3Ware 9650SE16ML SATA RAID controller (currently in JBOD) w/ battery backup option installed
  • SATA DVD drive
  • OCZ ZDrive PCIe SSD

Here is a video link that shows the pool is online but I get kernel panic when I try to import it as readonly - https://www.youtube.com/watch?v=UTaSOrA4aq0
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
What’s the firmware of the LSI cards?
Not sure, but I will check and find out. Currently FreeNAS-main is turned off and I am working on FreeNAS-sandbox01 with the 10 drives installed. I would like to get this working on FreeNAS-sandbox01 so that I can build a new system on FreeNAS-main and migrate the data over there.


I installed live version of mfsbsd on USB and booted into it.
zpool import still shows the "media" pool online and all 10 HDDs available, but when I try to import and mount it on /mnt/media, kernel panics and system reboots.

While going over THIS thread, I ran into a helpful command that shows what might be causing the kernel panic.

After running zdb -e -F media, I get the following:
Code:
error: blkptr at 0x837f42d00 has invalid CHECKSUM 0
pid 39 9=(zdb), uid 0: exited on signal 6 (core dumped)
Abort trap (core dumped)


Attached is the screenshot of the error.
 

Attachments

  • IMG_20180512_154254.jpg
    IMG_20180512_154254.jpg
    325.1 KB · Views: 609

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
Updated title to reflect the issue now that we know a little more about the situation.
 

garm

Wizard
Joined
Aug 19, 2017
Messages
1,556
Can’t help you, you are running an old and unsupported firmware for those LSI cards. To be clear, you haven’t given ZFS full access to the drives. You are in a very bad spot and anything I say would jeopardize data. My two cents are to find a data recovery firm specializing in ZFS. They might be able to help you rebuild the array you based the pool on.
 

Mehul V

Explorer
Joined
Oct 17, 2015
Messages
53
Can’t help you, you are running an old and unsupported firmware for those LSI cards. To be clear, you haven’t given ZFS full access to the drives. You are in a very bad spot and anything I say would jeopardize data. My two cents are to find a data recovery firm specializing in ZFS. They might be able to help you rebuild the array you based the pool on.
Is this because the raid controller is in IR mode? Can I install the drives back in it's original system, change the controller to be in IT mode and then try to import the pool?

Going to a data recovery firm would be too expensive and is not an option for me.
 

Jailer

Not strong, but bad
Joined
Sep 12, 2014
Messages
4,977
It depends on how your pool was created. But with it panicking at this point i dont have much confidence that it will work.

Having said that it would be good to get the opinion of some more knowledgable members such as @Ericloewe or @Stux or @Spearfoot and see what they think.
 
Status
Not open for further replies.
Top