FreeNas failed to boot with KDB: enter: panic

[nate-wilkins]

Hi! I'm having some trouble with my FeeNas server and was hoping someone(s) could help me get it back up and running.

My server recently, a few days ago, stopped working and on a reboot the server would crash on boot.

I get the following error:

Code:

KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe04561348e0
vpanic() at vpanic+0x177/frame 0xfffffe04561349040
panic() at panic+0x43/frame 0xfffffe04561349a0
assfail3() at assfail3+0x2c/frame 0xfffffe04561349c0
zfs_mknode() at zfs_mknode+0x178/rame 0xfffffe0456134e20
zfs_freebsd_create() at zfs_freebsd_create+0x50f/frame 0xfffffe0456134ef0
VOP_CREATE_APV() at VOP_CREATE_APV+0x7c/frame 0xfffffe0456134f20
zfs_replay_create() at zfs_replay_create+0x5c8/frame0xfffffe0456135130
zil_replay_log_record() at zil_replay_log_record+0x21b/frame 0xfffffe0456135280
zil_parse() at zil_parse+0x4ca/frame 0xfffffe0456135470
zil_replay() at zil_replay+0xca/frame 0xfffffe04561354d0
zfsvfs_setup() at zfsvfs_setup+0xb3/frame 0xfffffe04561354f0
zfs_mount() at zfs_mount+0x6b2/frame 0xfffffe0456135680
vfs_domount() at vfs_domount+0x598/frame 0xfffffe04561358a0
vfs_donmount() at vfs_donmount+0x92d/frame 0xfffffe0456135940
sys_nmount() at sys_nmount+0x72/frame 0xfffffe0456135980
amd64_syscall() at amd64_syscall+0xa38/frame 0xfffffe0456135ab0
fast_syscall_common()at fast_syscall_common+0x101/frame 0xfffffe0456135ab0
--- syscall (378, FreeBSD ELF64, sys_nmount), rip = 0x8010f55ba, rsp = 0x7fffffffcd68, rbp = 0x7fffffffcde0 ---
KDB: enter: panic
[ thread pid 1639 tid 100492 ]
Stopped at       kdb_enter+0x3b: movq    $0,kdb_why

I have no idea where to go from here though because I've never had to deal with a kernel level crash before and wasn't sure the best approach for getting back up and running. I'm assuming that one of my drives is busted and can no longer mount but I haven't confirmed this because I'm not sure how.
I did, however, get a bootable Manjaro thumbdrive to boot so I don't think its the MB, PSU, RAM, or CPU.

Any help would be greatly appreciated!

Server Specs:

Code:

Case: Corsair Carbide Series 200R Black Steel
Processor: Intel Core i7-2600k
Memory: 16GB XM 1600MHz 9-9-9-24 1.50v
GPU: N/A
PSU: COSAIR 1050
Motherboard: Asus PBZ77-V LX LGA 1155
Storage: 3x WD Red Plus 2TB NAS Hard Disk Drive - 5400 RPM Class SATA 6Gb/s, CMR, 64MB Cache, 3.5 Inch - WD20EFRX - OEM

 

[HoneyBadger]

Welcome.

It is very likely that you're seeing a kernel panic on boot as a result of a corrupted or damaged boot device. Were you using a USB thumbdrive by any chance?

Try a fresh install of TrueNAS (optionally, to a new device) and either restore your configuration backup, or import your pool and manually recreate the shares.

Your configuration is automatically backed up to the system dataset [1] (which is migrated to your first created pool) - you may be able to recover it from there if you had a complex configuration, but you should be able to reinstall and choose "Import Pool" as a first step.

[1] https://www.truenas.com/docs/core/c...n/usingconfigurationbackups/#automatic-backup

 

[nate-wilkins]

That could make sense as well.
I was using my Samsung 500gb SSD.

I also had a fairly complex configuration... Is this the relevant part about uploading a backed up configuration?

I'm not sure which version of FreeNas I was running which is why I'm a little concerned about reinstalling it.
Do you know of any way to figure out which version of FreeNas I was running?
And if not do you think its alright to import an old FreeNas version of the configuration file?

Are there issues if I rearrange my drives on my motherboard? I was going to unplug them while I installed the new TrueNAS installation but don't want to mess anything up.

Sorry for all the questions, I'm so new to this!

 

[HoneyBadger]

That could make sense as well.
I was using my Samsung 500gb SSD.

I also had a fairly complex configuration... Is this the relevant part about uploading a backed up configuration?

URL corrected: https://www.truenas.com/docs/core/coretutorials/systemconfiguration/usingconfigurationbackups/

You should be able to install a fresh copy of TrueNAS, import your existing pool (don't create a new one!) and then explore the "system" dataset. If you open a command line, you should be able to see it mounted as /var/db/system/configs-[very long GUID here] - you can use a command-line or GUI-based SCP or SFTP program to connect to the server and copy a recent backup file, which you can then restore through the web interface.

I'm not sure which version of FreeNas I was running which is why I'm a little concerned about reinstalling it.
Do you know of any way to figure out which version of FreeNas I was running?
And if not do you think its alright to import an old FreeNas version of the configuration file?

It should tell you during the initial boot menu and process before it crashes out; but if not, it will be named in the directory under the configs-GUIDVALUE path. And yes, an older configuration should import into a newer TrueNAS; however, I would suggest not performing the step of "Update pool version" until you're confident everything has been restored, and perhaps even after a scrub has run.

Are there issues if I rearrange my drives on my motherboard? I was going to unplug them while I installed the new TrueNAS installation but don't want to mess anything up.

Provided that you created your pool from the GUI, there will be no issues. When creating the pool this way, TrueNAS uses drive-specific identifiers that remain constant regardless of the physical port or even drive controller (SATA vs SAS HBA) - if you created a pool manually from a command prompt using device names such as "/dev/ada0" then it might cause your pool to not auto-import, and require manually specifying the member devices.

But it's definitely a good idea for safety to unplug your data drives so you don't accidentally overwrite one. You

Sorry for all the questions, I'm so new to this!

Sorry it took so long to answer them! :)

 

[nate-wilkins]

This is invaluable, thank you! I really appreciate the help.

You should be able to install a fresh copy of TrueNAS, import your existing pool (don't create a new one!) and then explore the "system" dataset. If you open a command line, you should be able to see it mounted as /var/db/system/configs-[very long GUID here] - you can use a command-line or GUI-based SCP or SFTP program to connect to the server and copy a recent backup file, which you can then restore through the web interface.

Okay so I downloaded and booted up the latest version of the TrueNAS installer but I'm running into my first issue which is I really don't know where the config backup is... Which probably means that I don't have one unless it was automatically backed up somewhere? The docs do say that the config is backed up automatically to the system dataset but I don't know where that lives, if that makes sense?

I booted up the TrueNAS installer shell to try to get an idea about what's going on but I'm thinking that this actually doesn't help because the SSD that I was booting from isn't mounted or anything so I can't look at it. Which explains why `/var/db/system` doesn't exist when I check it in the shell.

Do I need to mount my bootable SSD in the installer shell and pull out my system dataset config? Or am I getting the steps wrong and the dataset really exists on my terrabyte drives?

 

[nate-wilkins]

Just ran the installer and accidentally booted into my SSD (old FreeNAS boot) and it works. (No pool storage attached at all)

I don't know if that changes things but I'm able to export the "System" -> "General" -> "Save Config" backup config now. Which I think is a good thing.

Do you think there is anything else I should pull out before installing TrueNAS?

If not I'm going to:

1. Install latest TrueNAS
2. Import my, just saved, backup config through the TrueNAS GUI
3. Plug in all my storage/data drives
4. Import pool through the TrueNAS GUI

In summary so far I think this means that one of the storage/data drives is broken/corrupted.

 

[nate-wilkins]

This has been a learning experience.

I've completed the following:

1. Install latest TrueNAS
2. Import my, just saved, backup config through the TrueNAS GUI
3. Plug in all my storage/data drives

However, I'm stuck on the last step "Import pool through the TrueNAS GUI".

It does seem to have partially worked though because I can now boot with all the drives connected but the pool is encrypted and it seems that it can't be decrypted?

Here's what the GUI shows:

This other thread seems like the right approach but I don't want to disconnect the pool in case it breaks something. Ha.

When I do try to import, without disconnecting the `red_pool`, it doesn't pick up any devices. As shown below:

They are listed in devices though. The strange thing is I don't remember the drives with those names, but I could be wrong about that.

 

[HoneyBadger]

the pool is encrypted

Did you make a backup of the encryption and/or recovery keys after the pool was created, or take a manual backup at one point with the "Export Legacy Encryption (GELI) Keys" option checked?

 

[nate-wilkins]

That's a funny story but yes I do have the key.

Should I disconnect the pool and re-import it?

 

[HoneyBadger]

That's a funny story but yes I do have the key.

I'm interested in the funny story, but even more glad to hear that it won't be a tragedy if the answer had been "no."

Should I disconnect the pool and re-import it?

Yes, and be absolutely certain that you don't check the "Destroy Data" option. On re-import, simply answer the same question "Yes, the pool is encrypted" and provide your keyfile/passphrase when prompted. It should decrypt the GELI layer and present the decrypted pool for mount.

 

[nate-wilkins]

I'm interested in the funny story, but even more glad to hear that it won't be a tragedy if the answer had been "no."

Oh absolutely.

Well, when I first saw that the drive was encrypted I freaked out a little bit because I didn't know when/where I made a backup or if I had backed up this encryption key. So there was that. But then I remembered I had backed up the encryption key on a external HDD because I wouldn't encrypt my drive without a backup of the key (probably lol) and I was putting backup stuff on this drive. But, of course, I had just moved from CA to NY soo I didn't know where that drive was at all haha so had to go through boxes and calling people that helped me move - it was very concerning.

Long story short - I found the HDD and it did indeed have the encryption key on it.

Should I disconnect the pool and re-import it?

There is another option which I wanted to ask you about as well which is the "Delete configuration of shares that used this pool".

I do remember getting the sharing functionality on windows to work properly was horrible and I don't really want to figure that out again.

If I leave that unchecked will I be able to apply that configuration to the new pool?

 

[HoneyBadger]

There is another option which I wanted to ask you about as well which is the "Delete configuration of shares that used this pool".

I do remember getting the sharing functionality on windows to work properly was horrible and I don't really want to figure that out again.

If I leave that unchecked will I be able to apply that configuration to the new pool?

Yes, uncheck that dialog - the SMB service and any other shares will likely complain loudly about their underlying dataset being gone, but you should be able to do just what you're asking, although I admit I haven't tried this with a GELI pool. You also have a recent backup of the configuration as well, that you ideally could restore again if this fails.

 

[nate-wilkins]

Perfect, everything seems to be working as it was before it!

Thanks for all the help @HoneyBadger this was very informative and I feel a lot more comfortable in a disaster recovery situation.

Just for anyone coming to this thread these are the steps I took to get it working.

1. Unplug pool storage disks/drives
2. Start the server
3. Export the config "System" -> "General" -> "Save Config"
4. Install latest TrueNAS on boot drive
5. Import the config "System" -> "General" -> "Upload Config"
6. Plug in all my storage disks/drives
7. Navigate to "Storage" -> "Pools"
8. Disconnect the pool (mine was encrypted)
   1. Make sure *NOT* to delete the data in the pool
   2. Make sure to uncheck the "Delete configuration of shares that used this pool"
9. Import the pool "Storage" -> "Pools" -> "Add"
   1. Select the disks that the pool had
   2. If the disks are encrypted then upload your backed-up encryption key