FreeNAS Stuck in Bootloop

[ekarni]

Hello,

So I woke up today to my server not working when I tried to reach it via its IP. After some troubleshooting steps, I got it hooked up to a monitor and found that the server is just continuously restarting.

I tried reading over the log, but couldn't understand anything. I recorded it in slow motion in hopes of making it easier for anyone willing to help to read it.

I'm worried I lost all of my data, so any help would be greatly appreciated

Thanks in advance

Bootup Video - https://youtu.be/fUoOJb6PGZ0

 

[HoneyBadger]

For the convenience of others, a screenshot of the error before the reboot:

This does look like metadata corruption (range_tree_space) - so we should try to get the pool mounted read-only to ensure it's healthy. Can you interrupt the boot sequence and boot into single-user mode?

 

[ekarni]

For the convenience of others, a screenshot of the error before the reboot:
View attachment 43616

Thanks for that! I wouldn't know what part of that is important, so I appreciate the help :D

This does look like metadata corruption (range_tree_space) - so we should try to get the pool mounted read-only to ensure it's healthy. Can you interrupt the boot sequence and boot into single-user mode?

I'm not sure what you mean by single-user mode, sorry! I'm really new to FreeNAS, but have some programming and hardware experience if that helps

 

[HoneyBadger]

On the initial FreeNAS boot screen where it offers the text menu, choose "3 - configure boot options" and in there should be an option to boot into single-user mode. The goal here is to avoid it auto-mounting the pool (and thus rebooting again) - if that doesn't work, you can also make a fresh FreeNAS install on a separate USB stick and boot from that temporarily to check pool health.

Can you post your full system specs as well? Seeing the "MSI Gaming Motherboard" splash screen means it's definitely desktop hardware. You aren't overclocking or trying to run anything other than extremely safe settings for RAM timing/voltage/etc by any chance?

 

[ekarni]

I can't do it at the moment, but as soon as I'm able to, I'll give it a shot!

Thanks!!!

 

[ekarni]

On the initial FreeNAS boot screen where it offers the text menu, choose "3 - configure boot options" and in there should be an option to boot into single-user mode. The goal here is to avoid it auto-mounting the pool (and thus rebooting again) - if that doesn't work, you can also make a fresh FreeNAS install on a separate USB stick and boot from that temporarily to check pool health.

Can you post your full system specs as well? Seeing the "MSI Gaming Motherboard" splash screen means it's definitely desktop hardware. You aren't overclocking or trying to run anything other than extremely safe settings for RAM timing/voltage/etc by any chance?

I tried the Single-User boot option, but it asks me the following:

Trying to mount foot from zfs:freenas-boot/ROOT/default []...
Enter full pathname of shell or RETURN for /bin/sh:

What should I do?

 

[HoneyBadger]

Just accept the defaults for the single-user prompt. If it errors out there, you might need to make a new USB stick or other boot media.

If you can get to a shell, try zpool import -o readonly=on FamilyDataPool

 

[ekarni]

One sec, I'll give that a shot!

 

[ekarni]

It just kept putting # on new lines and I couldn't get it to continue. Should I try using a flash drive with an installer?

 

[HoneyBadger]

If you're sitting at a prompt that's just a "#" then you have a shell.

Try simple commands like "pwd" (print working directory) or "ls" (list) to see where you're at.

 

[ekarni]

I use WSL for web development so I'm familiar with Linux; is this the same?

 

[HoneyBadger]

Yes, you've got a shell now, but in single-user mode.

Basically we're hoping to be able to mount the pool. Start with readonly - if you can get it there, then we go about copying all the data (if it's feasible) to another set of disks and then recreate the pool.

If it's not possible, then we have to look at rewinding the transactions via zpool import -nF PoolName switch to rewind and discard the last updated data. (The "n" will tell it not to do it right away, but rather simulate it first.)

 

[ekarni]

Tried `ls` and it does list out the directories. Should I navigate somewhere specific?

 

[ekarni]

Are you available on Discord or some other RTC? I don't want to mess this up and delete my data or whatever else I can screw up

 

[HoneyBadger]

Quick update for those following along:

We've tried importing with the -F to auto-rollback, manually specifying a rollback TXG with -T from one pulled from the vdev, and it's just hanging. Single-drive vdev, so no opportunity to resilver from the other half of a mirror. @ekarni thankfully does have backups via BackBlaze but downloads could take some time.

Anyone else have any suggestions?

 

[HolyK]

Uhm ... no HW specs, version, no pool/storage details ... Don't get me wrong but these should be always included if you're asking for help otherwise people will just don't bother much.

I could only guess what caused it but as we have zero info about the HW it is hard to say...

 

[ekarni]

Uhm ... no HW specs, version, no pool/storage details ... Don't get me wrong but these should be always included if you're asking for help otherwise people will just don't bother much.

I could only guess what caused it but as we have zero info about the HW it is hard to say...

My apologies!!!!!!!!

This is the data I know about my system:

Show : FreeNAS Info

OS: FreeNAS-11.3-U4.1 on Samsung Evo 860 120GB
CPU: Intel Core i5-4670K
RAM: GSkill 16GB RAM
GPU: Nvidia GTX 660
Storage (Pool?): 10TB WD Red Plus (CMR)

 

[HoneyBadger]

Uhm ... no HW specs, version, no pool/storage details ... Don't get me wrong but these should be always included if you're asking for help otherwise people will just don't bother much.

I could only guess what caused it but as we have zero info about the HW it is hard to say...

I bothered for quite a bit yesterday via the chat functionality.

Basic specs have been posted but yes, it's a repurposed gaming machine so there's no ECC in play. It's also a single-drive pool with no redundancy, so any kind of metadata corruption would be permanent. And unfortunately that's what I think has happened here.

Overnight, the attempted import with the -T txg rollback failed out after several hours of attempting to mount. I suspect there's some manner of hardware failure in play here; @ekarni could try running memtest86 to see if it throws an error on CPU/RAM, and running a SMART test on the HDD, but if it's some other motherboard component then there isn't an easy test for that.

OP does have a donated Dell T610 server; probably a better candidate if it can be fired up, and the gaming machine can be put back into service for gaming or sold off to recoup some of the costs of building the T610 into a stronger/more resilient machine.

 

[ekarni]

A bit of an update.

I installed a fresh copy of FreeNAS on a different HDD. I then managed to boot it and went to the pool import section. With the HDD containing the old pool, I went to try and import it, but I don't see any available options.

Any advice?

 

[ekarni]

UPDATE! I restarted the machine, and now it is able to detect the old pool.

I tried importing it, but the server crashed and rebooted. I removed that drive and restarted the machine, and now it is up and running again. So that does mean the issue is with the drive huh?