Zpool is refusing to stay online

[SarcasmoRex]

During a few power outages a few days ago, my system did not come back up happy. After some google-fu, I managed to see that there were no SMART issues; I merely needed to do an import of the pool via the CLI. Which I did. UI reflected the happy state of the system. I figured the correct and prudent next step was to do a system restart so that my apps and VMs could come up healthy and happy. Upon reboot, the zpool is back in an 'offline' state. If I (via the UI) export and re-import the pool, I can get back to regular operation, but once there's a reboot, I'm back to 'offline.' Is there a recommended solution to get this back to a happy state?

Thank you in advance for any help.

v/R,
SarcasmoRex

 

[Arwen]

Please list your hardware, and pool configuration. And yes, this can make a difference.

 

[SarcasmoRex]

Please list your hardware, and pool configuration. And yes, this can make a difference.

The hardware is a TrueNAS Mini XL+ with 8 14TB drives. One large raidz2 pool (which was the recommendation from the UI). Forgive me if this is inadequate information; I have not used Truenas before and just followed the advice from the UI.

 

[Arwen]

Sorry, the one condition that I was thinking of, does not apply in your case, (with your hardware).

Perhaps someone else can chime in.

 

[jgreco]

Sorry, the one condition that I was thinking of, does not apply in your case, (with your hardware).

And that thought would be ... Improper use of a RAID card?

Really there's very little to go on here, I'm sorry. You're going to have to do a little digging to see if you can determine what's going on. The export/import fix thing is kinda weird. This could be something unusual like a failing drive mucking up the at-boot import but later on stabilizing and becoming usable, but that's pretty unusual.

Could you please post the output of

# zpool status

when the system is freshly booted and the pool is unavailable? This might provide some clues. Additionally, if you can post the output of

# smartctl -a /dev/daX

for each drive, which may be like da0, da1, da2, or possibly ada0, ada1, ada2, etc., I'm curious to see if the drives are all self-reporting as healthy.

 

[Arwen]

And that thought would be ... Improper use of a RAID card?

...

Yes.

 

[Davvo]

Try peeking into system logs, you could find a lead.

 

[Ericloewe]

One detail I'd like to see is:

Could you please post the output of

# zpool status

when the system is freshly booted and the pool is unavailable? This might provide some clues.

Pools don't go OFFLINE for no reason and zpool status is definitely the place to start: Is it one disk acting up, multiple disks? Are they missing or just reporting junk and need to be dropped? Etc.

 

[jgreco]

OFFLINE for no reason

Well, if the pool failed to import to begin with, doesn't it end up in offline status? Still a bit puzzling.

 

[Ericloewe]

Well, if the pool failed to import to begin with, doesn't it end up in offline status? Still a bit puzzling.

Wouldn't it just never read the imported state? I guess it depends on where the import fails and how cleanly it fails...

 

[SarcasmoRex]

And that thought would be ... Improper use of a RAID card?

Really there's very little to go on here, I'm sorry. You're going to have to do a little digging to see if you can determine what's going on. The export/import fix thing is kinda weird. This could be something unusual like a failing drive mucking up the at-boot import but later on stabilizing and becoming usable, but that's pretty unusual.

Could you please post the output of

# zpool status

when the system is freshly booted and the pool is unavailable? This might provide some clues. Additionally, if you can post the output of

# smartctl -a /dev/daX

for each drive, which may be like da0, da1, da2, or possibly ada0, ada1, ada2, etc., I'm curious to see if the drives are all self-reporting as healthy.

Code:

zpool status
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:25 with 0 errors on Sun Aug 21 03:45:27 2022
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p3  ONLINE       0     0     0

errors: No known data errors

  pool: vault
 state: ONLINE
  scan: scrub repaired 0B in 00:32:41 with 0 errors on Sun Jul 31 00:32:44 2022
config:

        NAME                                      STATE     READ WRITE CKSUM
        vault                                     ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            f8f88755-bd85-4039-a174-993239842284  ONLINE       0     0     0
            366b0867-3d72-42c3-aecb-f367a16a8824  ONLINE       0     0     0
            c3a53432-13a7-47c8-b500-5e6d5d9b8207  ONLINE       0     0     0
            c36f9a87-3af1-4a75-8b79-a8876c1cd9be  ONLINE       0     0     0
            3c3b5e7f-c3f9-400c-becd-7ec9b9c263f7  ONLINE       0     0     0
            b33965a3-87a1-41ed-b401-3cd1f06d837b  ONLINE       0     0     0
            b226e641-1217-47ae-92cc-f54e20385280  ONLINE       0     0     0
            1b26d2c1-305a-4603-aa4b-8cc7d3319414  ONLINE       0     0     0

errors: No known data errors

I ran a smart test and they all came back clean. The issue is that the UI is no longer bringing the pool online automatically on reboot. If I export and re-import it manually, then I can get back to work.

 

[SarcasmoRex]

Well, if the pool failed to import to begin with, doesn't it end up in offline status? Still a bit puzzling.

Yeah, I can manually re import the pool from the command line or I can export and re-import the pool from the UI. It's just not doing it automatically on reboot.

 

[Ericloewe]

The issue is that the UI is no longer bringing the pool online automatically on reboot.

That's the state from which we want to see zpool status.

 

[SarcasmoRex]

That's the state from which we want to see zpool status.

Code:

zpool status
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:00:25 with 0 errors on Sun Aug 21 03:45:27 2022
config:

        NAME         STATE     READ WRITE CKSUM
        boot-pool    ONLINE       0     0     0
          nvme0n1p3  ONLINE       0     0     0

errors: No known data errors

  pool: vault
 state: ONLINE
  scan: scrub repaired 0B in 00:32:41 with 0 errors on Sun Jul 31 00:32:44 2022
config:

        NAME                                      STATE     READ WRITE CKSUM
        vault                                     ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            f8f88755-bd85-4039-a174-993239842284  ONLINE       0     0     0
            366b0867-3d72-42c3-aecb-f367a16a8824  ONLINE       0     0     0
            c3a53432-13a7-47c8-b500-5e6d5d9b8207  ONLINE       0     0     0
            c36f9a87-3af1-4a75-8b79-a8876c1cd9be  ONLINE       0     0     0
            3c3b5e7f-c3f9-400c-becd-7ec9b9c263f7  ONLINE       0     0     0
            b33965a3-87a1-41ed-b401-3cd1f06d837b  ONLINE       0     0     0
            b226e641-1217-47ae-92cc-f54e20385280  ONLINE       0     0     0
            1b26d2c1-305a-4603-aa4b-8cc7d3319414  ONLINE       0     0     0

errors: No known data errors

 

[Ericloewe]

But the pool is online there, and with zero errors, just like your previous post.

 

[Arwen]

@Ericloewe - If the pool is not imported, we would want this command run;

zpool import

Using the status command will show nothing of the data pool. if the pool is not imported.

 

[Ericloewe]

But I think the offending pool here is vault, right @SarcasmoRex? Here it's imported and reporting no errors, whereas the interesting information would be zpool status from when the pool is not okay, and maybe zpool import if, at that time, it's not imported at all.

 

[SarcasmoRex]

But I think the offending pool here is vault, right @SarcasmoRex? Here it's imported and reporting no errors, whereas the interesting information would be zpool status from when the pool is not okay, and maybe zpool import if, at that time, it's not imported at all.

So, the issue is that I can get the pool (yes, vault is correct) back online by export and re-import, but if I reboot, I have to go through the export and import process again. I am trying to figure out how to make the UI bring it back after a reboot like it is supposed to.

 

[SarcasmoRex]

But the pool is online there, and with zero errors, just like your previous post.

You are correct. But, if I reboot the system, it will be in an offline state and I will have to go through the manual process of export and re-import in the UI to get back to a functional system.

 

[SarcasmoRex]

But I think the offending pool here is vault, right @SarcasmoRex? Here it's imported and reporting no errors, whereas the interesting information would be zpool status from when the pool is not okay, and maybe zpool import if, at that time, it's not imported at all.

Sorry, just understood what you were actually asking. I'll get you the data. My apologies.