panic: Solaris(panic): blkptr at 0x.. has invalid CHECKSUM

[JEREMY J NIEUWOUDT]

Issue:
I've been running FreeNAS for a few years now, since v9.3. I've been applying updates via the GUI all this time, but the last time I did this the system panic'd (see attached) while trying to mount the ZFS pool. At the time, I was on v11.0-u4. Previous boot environment available are v11.0-u2 and initial install. (Performed a rebuild early last year after ditching Corral).

I tried going back to a previous boot environment (there were 2 previous available) and had the same failure at ZFS pool import.

Then also tried booting to v11.1-U2, v11.1-U3 with the same result - panic during ZFS import.

I ran a 24hr memory test and all passed.

I then also booted the system in single user mode and tried mounting the volume - still the same result.

Now I suspect perhaps a hard drive is at fault. I don't have a deep technical understanding of the FreeNAS architecture, but do know my way around the unix operating system a little.

Can someone please point me in the right direction to resolving this issue? Specifically any troubleshooting/command tricks I can try to verify various aspects of the build.

My last resort is a rebuild, but my latest backup is more than a month old, and I'd like to recover as much as possible before rebuilding.

Any help is greatly appreciated.

Thanks in advance.
------------------------

System Config:

Code:

CASE:   Fractal Design Define Mini
M/B:	Asus H97M-PLUS  BIOS Ver.2202
CPU:	Intel Pentium G3258 (Anniversary Edition)
RAM:	32GB DDR3/1333MHz
HDDs:	4 x 3TB Seagate NAS HDD (ST3000VN000)
PSU:	Corsair 450W 80-plus
BOOT:	2 x Strontium 16GB USBs (mirrored)
RAID:	Z2-RAID
NAS:	FreeNAS v11.0-U4

------------------------

Initial error during boot.

Output of 'zpool import' command (single user mode)

Followed by 'zpool import -f VOL1' (single user mode).

 

[Ericloewe]

Your images are completely broken. This is exactly why we ask you to upload images to the forum instead of using external hosts.

 

[JEREMY J NIEUWOUDT]

My apologies. Here they are:

1. Initial error during boot, post initiating update from the GUI.

2. Output of 'zpool import' cmd, after booting into single user mode in a prior boot environment:

3. Output of ensuing 'zpool import -f VOL1' cmd (single user mode)

Thank you again.

 

[Ericloewe]

Looks like some pretty nasty metadata corruption. If you're lucky and this was recent, you might be able to rollback a few TXGs to get to the bulk of your data.

You can also try to import it read-only. I doubt it'll work, but can't hurt to try.

This could very well have been caused by an error in RAM, but that's impossible to determine now.

A ZFS expert might have some other ideas that might allow for some data to be read, but don't count on it.

 

[nightshade00013]

https://illumos.org/msg/ZFS-8000-EY

The pool has been written to from another host, and was not cleanly exported from the other system. Actively importing a pool on multiple systems will corrupt the pool and leave it in an unrecoverable state.

So sounds like you will have to force the import with -f which is also listed on the error message page at the very bottom. Basically something messed up during the shut down and rolling back makes the system think it's from a different system altogether is my guess. As long as you didn't change something on the pool, like update feature flags, that makes it incompatible with a different version it should be fine with a force import.

Actually going to the error message page is a pretty good first step, never seen theirs before but it explains everything.

 

[JEREMY J NIEUWOUDT]

@Ericloewe I did attempt a READ ONLY import, but that failed similarly.
@nightshade00013 The 3rd screenshot shows the output from an attempt at importing with the -f parameter. Same result.
Thanks for the quick responses !!

 

[nightshade00013]

Ahhh, it's probably too late for me to be trying to read then.

Anyway https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222734 Probably explains what has happened and drives home the reason for ECC ram as in that posting a "bit flip" happened causing an issue.

Might try -Fn and see if it will recover with minimal issues and then try -F if the first one looks good.

https://www.freebsd.org/cgi/man.cgi?zpool(8)

-F Recovery mode for a non-importable pool. Attempt to return
the pool to an importable state by discarding the last few
transactions. Not all damaged pools can be recovered by using
this option. If successful, the data from the discarded
transactions is irretrievably lost. This option is ignored if
the pool is importable or already imported.

-n Used with the -F recovery option. Determines whether a non-
importable pool can be made importable again, but does not
actually perform the pool recovery. For more details about
pool recovery mode, see the -F option, above.

 

[Ericloewe]

You probably need lowercase f, too, so -f -F -n.

 

[JEREMY J NIEUWOUDT]

Is it possible to rollback txg's when you're not able to successfully mount a pool? If so, what's the procedure?

 

[mrpizat]

were you ever able to resolve this issue? having a similar error. only getting there was a bit different. (lost a drive, replaced the drive, resilvered, there was bad ram, os went bad. trying too import in a new environment, getting that error)