My Freenas System stuck in bootloop after upload config file

[jhc]

Hi, this is the problem I's trying to resolve.

Recently, after a weekend, I received the following alerts



Uh-oh...

This is a basic system, which I set up in 2013, with an MSI motherboard, i3 CPU and 2x8GB of non-ECC RAM.

(It hasn't had much use on a daily basis and most of the stuff just lives there. All has gone well up to now, maybe I've just been lucky to date)

Don't think disks are encrypted and Dedup is not enabled.




On reboot, bios reported RAM had changed, memtest revealed a defective stick of RAM.

Replaced RAM.

• I tried rebooting, boot device is a usb3 flashdrive, got nothing but a bootloop.
• I downloaded and installed Freenas 11.3 onto a usb2 flashdrive I had spare. Booted okay, but when I uploaded config file, I got another bootloop.
• Tried again without uploading config file, shell'd, and <zpool import> yielded the following

and



HomeStor1 is a mirrored array of 2 disks
HomeStor2 is a striped mirrored array of 2x2 disks
HomeStor 3 is a single disk (might even be NTFS import) and is only used for temporary storage of large avi files when being edited (complements second drive on main PC)

I was able to import HomeStor3 (and it's dataset) successfully. As it's contents are not important I though this was the best one to try first.



HomeStor1 is docs and photos which are much more important and is backed up to another off-the-shelf NAS, most recent backup was about 2 weeks before the crash.
HomeStor2 is Rips of 400 DVDs and BluRays, many of which I could do again but it would take ages.

I set this box up back in 2013 as a NAS for my home to share files between (windows) PCs and provide some protection against data loss and I have periodically replaced disks and boot drives without problems.

I've had a look around the forums but I am unsure how to proceed.

I cannot remember what I named the various shares, users accounts, etc etc. Maybe this is not important, and maybe it is.

I have both a tar and db file of the config (although tar is definitely causing crashes)

Also have original Flashdrive with Freenas on it.



Just remembered, this original flashdrive reported it was full, about 6 weeks ago, and I removed a number of older versions of freenas using the GUI.



Apologies for the length but wanted to post as much information as possible.

Any suggestions greatly appreciated.

J

 

[Samuel Tai]

It appears your pools are OK. Reboot again without importing the config, and see if all 3 pools show up in the GUI. If they're all there, try importing the config at that point.

 

[jhc]

Hi,

Thanks for that.

I've just tried importing via GUI.

I managed to get HomeStor2 to import, however when I then tried to import HomeStor1 the machine crashed and automatically tried to reboot and crashed again.

Do I need to clean the usb and create a freenas boot?

J

 

[Samuel Tai]

You're at the point where USB sticks are way past the point of diminishing returns. Try something like a small M.2 SSD 16-32 GB in size, connected to a USB adapter. This will give you a super-fast USB stick on which to install.

 

[jhc]

I've got a new SanDisk SSD Plus 120 GB which I was going to use a scratch disk in main PC. Will this be good enough?

I've connected that to a spare SATA port, other 7 disks (zpools) are attached to an IBM ServeRaid M1015 flashed to LSI9211.

Have installed Freenas-11.3-U3.2. Enabled swap(?) option as well.

All three zpools visible to add from GUI and all 7 disks available in shell.

What should I try next?

Thanks

j

 

[jhc]

Is HomeStor1 toast and does that mean the config file is causing crashes?

J

 

[Samuel Tai]

If all zpools are visible for import, then go ahead and import them. We'll have to assume your saved config may be corrupt, so you'll have to reconfigure by hand.

 

[jhc]

Hi,

Thanks again.

HomeStor3 and HomeStor2 are importing okay, HomeStor1 is causing a problem on import, crashes freenas (at least it did using a flashdrive for freenas)

 

[jhc]

Okay,

I tried to import zpool HomeStor1 from shell, as settings changed through CLI ...

Box crashed, on reboot GUI still lists HomeStor1 as available for import.

If the zpool itself is a bust, as it is a mirror, is there anything that can be done to extract data.

(I've discovered that most stuff is backed up, but of course the Western Digital My Book mirror announced that there were to errors, run disk scan etc and a small amount of files have scrambled)

Also, can I interrogate the config files to see what configuration I've used, I've got some older *.tar files available.

Thanks

John

 

[pschatz100]

If this is an old system, running since 2013, you might try replacing the bios backup battery and verify that all your bios settings are OK. I know this isn't a FreeNAS fix per se, but I have old machines as well and have had weird problems show up when powering down or rebooting after the system has been running for many years. It may not solve your problem, but is good maintenance and easy to do.

 

[Samuel Tai]

This is kind of drastic, but try importing HomeStor1 with just a single disk.

 

[jhc]

Okay, I'll shut it down and change the battery, I'm sure I have one about here some where.

How would I import it as a single disk? just disconnect one of the mirrored disks? And will the zpool be permanently broken if it doesn't work?

The fact the the zpool is online and "available" presumably does not mean that it is healthy, ie it could be already broken?



There's no sort of force import or ignore errors option?

Thanks again

 

[jhc]

Also does *.tar have any human readable information that could jog my memory about the shares I set up?

J

 

[Samuel Tai]

Okay, I'll shut it down and change the battery, I'm sure I have one about here some where.

How would I import it as a single disk? just disconnect one of the mirrored disks? And will the zpool be permanently broken if it doesn't work?

The fact the the zpool is online and "available" presumably does not mean that it is healthy, ie it could be already broken?

View attachment 40238

There's no sort of force import or ignore errors option?

Thanks again

I suspect one of the disks in the mirror may have a corrupt superblock, which is preventing the import from succeeding. Taking the presumably corrupt disk offline won't break the pool, but it may import in a degraded state. Unfortunately, you'll have to try each in turn.

You could also try zpool import -f HomeStor1 to force the import, or zpool import -N HomeStor1 to try just importing without mounting. Worst case, you could try zpool import -F HomeStor1 to ignore the contents of the in-pool ZIL.

All of these options, including the single-disk import, have a chance of data loss, especially the last one.

 

[jhc]

Okay, I'll try them all in turn until I get some sort of return.

And if I lose some data, as we say in Ireland, "half a loaf is better than no bread"

Thanks

J

 

[Samuel Tai]

Also does *.tar have any human readable information that could jog my memory about the shares I set up?

J

The freenas-v1.db is a binary file, but you can examine its contents in Notepad. I've seen fields about 2/3'rds down with user and group info. Your shares should definitely be reflected in there somewhere.

 

[jhc]

I'll have a look, all this might take me some as the kids are around this week.

J

 

[jhc]

Hi

Due to being away with work and then family, I only managed to get back to this a two weeks ago.

The steps taken so far…

Bought a SanDisk SSD PLUS 120 GB as the boot drive and tried the following

Connecting only the relevant disks,

HomeStor2 and HomeStor3:
I was able to get both HomeStor2 (a striped mirror of 4 disks)
And
HomeStor3 (a single disk, used as a scratch disk), to import without problems and all files are available on the network.

HomeStor1
Connected a single disk (say DiskA) of HomeStor1 (single mirrored disk) and tried
zpool import -f HomeStor1 and then zpool import -N HomeStor1

I noticed, after a boot scrub, the boot drive had 100+ errors, checked forums and tried turning TRIM off etc as recommended but this didn’t fix it.

(Using this as a boot drive appears problematic for Freenas, so will use a scratch disk in main PC)

Ordered Kingston SSD A400 120GB, installed Freenas and scrubbed without errors.

Using DiskA

Disk visible, HomeStor1 visible as a pool to import but import still causing crashes/ reboots. On reboot, HomeStor 1 listed as offline and I can export/ remove it but will not subsequently import without crashing.

Using DiskB
Disk Visible, HomeStor1 is not visible to import.

Have tried swapping power and SATA ports but this has no effect.
Also ran Memtest for a number of hours as a double check on the RAM.

Using an Ubuntu USB (20.04 LTS) I ran SMART tests on both drives, which completed without errors, so the disks appear to be good.

Is there any log or output that I could post that might show what is causing the crashes?

At this point I don’t know what else I can do, would it be possible to image either DiskA or DiskB before trying zpool import -F HomeStor1 ?

Obviously something has gone wrong with the structure of filesystem, but being a single disk (mirrored) the files should still there somewhere.

Is there anything I could use to scan and extract the files, eg. with an NTFS system I could use testdisk or some other file recovery option.

Thanks

J