Heeelp: Afraid to loose all my data

[ThomasJ]

Dear community,

I am running FreeNAS for multiple years already without any major problems. Now just recently it appears to me that my USB Stick as well as my raid controller?! crashed at the same time. Maybe my power supply had a failure.

System setup: I am running an encrypted 0+1 RAID with 4x 4TB and the discs had no sign of SMART errors recently. I am booting from a Scandisc USB and was running the latest version of FreeNAS (not TrueNAS yet).

From one day to the other my network drives where not reachable any more and the system didn't respond to the UI any more. After restart I saw that there was no Raid Array configured for the raid controller (0 logical volumes). When I tried to run HP Smart Storage Administrator the system bar loads to 100% and then crashed with a Kernal panic: LZMA data is corrupt; Kernel panic: not syncing: VFS: Unable to mount; Pid 1 comm: Swapper Not tainted 3.0.76-0.11-default #1

Since I count create a logical volume with that machine, I decided to buy an identical server (HP Proliant Miniserver Gen 8) and identical drives.

At first I cloned all 4 HDDs to make sure I only work with the cloned copies to not risk any data loss.
Secondly I tried to restore the Raid Array with the new Server (and the cloned HDDs).

Once the Array was in created (via HP Smart Storage Administrator) I tried to boot via the old Scandisc Stick in the new Server. The system still cant boot even if I pick old Kernel versions. The bootup stops when the system was trying to mount the devices.

Next I installed a clean trueNAS on a new SSD and used that SSD for Bootup (BIOS settings to support legacy RAID). TrueNAS boots successfully and I can see the 4 discs in TrueNAS. Now I did a mistake, I guess - I tried to create new Pools using the Array with the 4 discs, which started a formating process.

So now I am creating new clones and I need to try to not create new pools but import existing ones. The problem is I have no settings backups or anything. Of course I have the HDD encryption password, but apparently I also need a "file" to import pools.

My questions:
- Any chance to import the pools without having done a backup accordingly?
- Any chance that I get the pool information from the SSD stick that I still have access to, but apparently it wont boot again?

PS: Sorry if some of the terminology is wrong. I am not super specialist but a self thought beginner.

Thanks for your support

 

[Chris Moore]

When I tried to run HP Smart Storage Administrator the system bar loads to 100% and then crashed with a Kernal panic: LZMA data is corrupt; Kernel panic: not syncing: VFS: Unable to mount; Pid 1 comm: Swapper Not tainted 3.0.76-0.11-default #1

This is one of many reasons why we DO NOT use hardware raid with ZFS / FreeNAS.

 

[ThomasJ]

@Chris Moore: Thanks. Understand. But as I said my configuration is existing for a very long time and I am / was a beginner at that time.

So my questions remains: What can I do now?

 

[Arwen]

First, using hardware based RAID controllers with ZFS on FreeNAS / TrueNAS is a recipe for disaster. It can be done, and work reliably, (in a production Datacenter, with EMC or Hitachi SAN storage...). But, for FreeNAS / TrueNAS, avoid like plague, (unless you really know what you are doing).

Second, some hardware RAID controllers keep the configuration in on-board flash memory. This sounds like what got corrupted. If you have a total hardware RAID controller failure and replace it completely, their is generally a method to "recover" a RAID set from the disks. Not create a new one. Can't tell from your post if you tried the "recover" option with the new server. You may even be able to "recover" the RAID set with the original controller, (assuming the on-board flash simply faded, but is still good).

Third, using encryption without backups is a serious mistake. You need backups of the encryption key file at a BARE MINIMUM. Preferably multiple backups, like a bank safety deposit box if you have one. If you don't have that file, you may be out of luck. I don't use FreeBSD's GELI encryption, and it sounds like you are not using ZFS native encryption, so I can't help here.

Last, some hardware RAID controllers use serial numbers on the disks to determine what is in a RAID set. By cloning the disks, and then using the clones,you may not be able to "recover" the RAID set. It can still be done. Just instead of using the cloned disks, you may have to use the originals. And if you try again, use the clones as the source for the copies. Use at your own risk.

 

[ThomasJ]

thank you Arwen for your extensive answer. Even if the news does not read too good.

Today, I am aware that I made these 2 major mistakes in the setup but I cant change the past. I have done a lot of reading in the meantime but I really want to try everything to keep my data intact.

I am using a HP Proliant Microserver GEN 8 server with the following onboard RAID controller: HP Dynamic Smart Array B120i Raid Controller.

I have not configured any special encrypted but I am using the default encryption of FreeNAS. I cant recall if it was GELI or ZFS native. I think all I ever had was a password.

 

[Heracles]

Well... How many No-Go do we have here :
--USB Boot drive without redundancy or backups (minor)
--Hardware Raid Controler (major)
--GELI encryption without the extra-rigorous procedures and protocols it requires not to turn to a self-inflicted ransomware (major)
--Creation of a new pool over disks containing an existing one (major)
--No Data backups (major)

I only work with the cloned copies to not risk any data loss.

It may very be already too late...

Secondly I tried to restore the Raid Array with the new Server (and the cloned HDDs).

Once the Array was in created (via HP Smart Storage Administrator) I tried to boot via the old Scandisc Stick in the new Server. The system still cant boot even if I pick old Kernel versions. The bootup stops when the system was trying to mount the devices.

Next I installed a clean trueNAS on a new SSD and used that SSD for Bootup (BIOS settings to support legacy RAID). TrueNAS boots successfully and I can see the 4 discs in TrueNAS.

Something is wrong here : if you created a virtual drive with your hardware Raid controller, TrueNAS can not see the 4 drives. It will see the single virtual drive made out of these 4 drives. So what do you actually see ? The virtual drive or the 4 physical drives ??

Now I did a mistake, I guess - I tried to create new Pools using the Array with the 4 discs, which started a formatting process.

Indeed, creating a new pool will wipe the drives...

The problem is I have no settings backups or anything. Of course I have the HDD encryption password, but apparently I also need a "file" to import pools.

Indeed, you need the crypto key that is protected by the password you said you know.

- Any chance to import the pools without having done a backup accordingly?

Not much... I avoided pool level encryption and will never touch it because it is a million times too sensitive and fragile. It will loose you your own data way more often than protecting it from unwanted access. I do my encryption at file level with tools that are much safer to handle. It looks like you will be another example of this principle...

- Any chance that I get the pool information from the SSD stick that I still have access to, but apparently it wont boot again?

If that one still holds a copy of the crypto keys you need, that can be your last hope. But again, I can not tell you the exact procedure for it because as for me, pool level encryption is as bad and risky as hardware raid controllers.

With 4 major No-Go in your setup, maybe the most important thing that will come out of this is learning how and why to do things the right way...

About Pool level encryption...

 

[Arwen]

@ThomasJ - Sorry we have not been helpful. It's just that you are not the first, second or hundredth person with either the encryption problem. Or the hardware RAID controller problem. Having both was in essence a disaster waiting to happen, (unless you knew the risks and mitigated them with backups, backups...).

Okay, some steps to try:

- Learn if your hardware RAID controller has a recovery mode. This is almost certainly not the same step used to create a new RAID set. Most of us can't help because we don't use that particular hardware RAID controller.

- Check to make sure your cloned disks are the exact same size. Some hardware RAID controllers put the RAID set information at the end of the disks. So if the cloned disks are larger, even just slightly larger, that could confuse the hardware RAID controller's recovery mode. SATA does include a method to shrink disks, see manual page for "hdparm" in Linux, option "Host Protected Area". I don't know the FreeBSD / FreeNAS option for that function, ("camcontrol"?).

- As I said, it's possible that you can't use the cloned disks to recover the RAID set. You may need to use the cloned set as the masters, (after making a new clean copy), and experiment with the originals, (because of WWN / serial # of the drive). But, don't quote me, or blame me for any problems. I'd carefully get step 1 working first, and if it just won't recover but you think you have the correct method, then you might have to try the original disks...

- Check your USB flash drive. You need to see if you can recover the GELI encryption key file, (which you need even if you have a passphrase, if I understand it correctly). I can't help you here, because I don't know GELI encryption on FreeBSD. But the location of the GELI key file might be in the FreeNAS documentation. So read the docs on encryption. Understand how GELI encryption works, which may lead to things like where the GELI encryption key file is located on the boot media.

- Next, it sounds like your SSD OS was created AFTER your disaster. So, unless you were able to copy the GELI key file to it, that SSD FreenNAS load is likely not useful. (Well, you can use that SSD as a starting point for a new FreeNAS load, if you can find your GELI key file.)


One other note. FreeNAS was designed as an appliance. Because of that, their is a configuration file that stores all the share configuration, permissions for those shares, network info, etc... If you don't have that backed up too, you may have to start that part over. This is generally not fatal, just annoying and time consuming. (Unless you have critical details configured...)

Most of us use an automated script to either store the configuration on our, (un-encrypted), data pool. Or somewhere remotely. Or as part of our backup scheme. Mine saves to my un-encrypted data pool, which during backups gets saved.


Good luck, sorry you had this bad experience.

 

[ThomasJ]

@Arwen , @Heracles thanks for taking the time to provide me such a long explanation and entry points for further research.

After a lot of back and forth, I was able to restore my raid and import the pools into the new TrueNAS version.

I have access to all data again. I guess it was a little luck and I have learned from this. After making sure I have backed up everything I I’ll try to change the setup (recovery keys, cloud backup, no hardware raid, no encryption).

thanks for your support guys

 

[Arwen]

Glad you have your data back.

One reason FreeNAS / BSD uses ZFS, is that ZFS can protect your data better than hardware RAID controllers we see on the low end. Some of the low end, end-user NASes can and do loose data because of their RAID or file systems. In some respects, FreeNAS is the top of the list for free, & open source NAS software. But, some of the advanced features can bite people if their are not familiar with the features.

 

[KrisBee]

@ThomasJ Congrats on getting all your data back. I'm impressed your data recovery was successful, which I guess was dependent on re-creating your original (incorrect) setup on the second microserver. Just remember ALWAYS to use the BIOS to set the embedded raid controller to work in AHCI mode when using FreeNAS, or any other OS, with zfs support on the microserver.

 

[Heracles]

After a lot of back and forth, I was able to restore my raid and import the pools into the new TrueNAS version.

Great news! Good that you saved yourself with the lesson learned and no data lost. So many learned it only after loosing it all.

Have fun designing your new config and your new backup solution. Do not hesitate to ask questions on how to get it right.