Pool is in Offline State

[KArmstead]

I am running TrueNAS Core version 12.0-U1.1
I am not sure of the CPU, not sure where to find it. It has 64Gb of RAM. The boot drive is da0 and is a 92.2Gb drive. There are 12 2Tb SATA drives.
I had created a pool from those drives. I am running out of space on that pool, so I thought that I would try to replace a drive to see if I can then replace all of them eventually with larger ones. I bought a 4Tb Seagate Ironwolf drive. I choose da12 and put it in an offline state. I put the new drive in its place and then everything kind of stopped working. I pulled the new drive out and put the old one back in it's place. Now the pool shows as offline.

Code:

   pool: pool01
     id: 3296608596151641203
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        pool01                                          UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            gptid/45188cd7-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/4564e458-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45879be1-2ae6-11ec-9d4b-001e678ed53e  UNAVAIL  cannot open
            gptid/459fccd5-2ae6-11ec-9d4b-001e678ed53e  OFFLINE
            gptid/45b8f45f-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45c9cf14-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45e29906-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45fa9076-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/4600a96f-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/46205b62-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/461a9dd3-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/462ff1eb-2ae6-11ec-9d4b-001e678ed53e  ONLINE

Now another drive is showing as unavailable. I need to bring the offline drive online, and see if I can bring the pool online. I am not sure which drive is the unavailable drive.

 

[homer27081990]

I am running TrueNAS Core version 12.0-U1.1
I am not sure of the CPU, not sure where to find it. It has 64Gb of RAM. The boot drive is da0 and is a 92.2Gb drive. There are 12 2Tb SATA drives.
I had created a pool from those drives. I am running out of space on that pool, so I thought that I would try to replace a drive to see if I can then replace all of them eventually with larger ones. I bought a 4Tb Seagate Ironwolf drive. I choose da12 and put it in an offline state. I put the new drive in its place and then everything kind of stopped working. I pulled the new drive out and put the old one back in it's place. Now the pool shows as offline.

Code:

   pool: pool01
     id: 3296608596151641203
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-3C
 config:

        pool01                                          UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            gptid/45188cd7-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/4564e458-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45879be1-2ae6-11ec-9d4b-001e678ed53e  UNAVAIL  cannot open
            gptid/459fccd5-2ae6-11ec-9d4b-001e678ed53e  OFFLINE
            gptid/45b8f45f-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45c9cf14-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45e29906-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/45fa9076-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/4600a96f-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/46205b62-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/461a9dd3-2ae6-11ec-9d4b-001e678ed53e  ONLINE
            gptid/462ff1eb-2ae6-11ec-9d4b-001e678ed53e  ONLINE

Now another drive is showing as unavailable. I need to bring the offline drive online, and see if I can bring the pool online. I am not sure which drive is the unavailable drive.

You have a RAIDZ-1 array. See here and here. Can't remove whatever drives you want from a RAIDZ-1 pool.

 

[homer27081990]

gptid/45879be1-2ae6-11ec-9d4b-001e678ed53e UNAVAIL
cannot open gptid/459fccd5-2ae6-11ec-9d4b-001e678ed53e OFFLINE.

Also, it seems you maybe deactivated the wrong drive and pulled out another? Not sure.

 

[KArmstead]

So does that mean that I can not bring it back online? Can this be fixed or do I have total loss of data and have to start over?

 

[homer27081990]

So does that mean that I can not bring it back online? Can this be fixed or do I have total loss of data and have to start over?

If after re-checking it, it still cannot mount at least one of the two drives, then the array is lost in RAIDZ-1. It has, by definition, a tolerance of failure of exactly 1 and only 1 drive. That is the reason that it is highly discouraged to use it. One slip of the hand (or the keyboard) and poof! But this is just a battle at the moment, not the war. You need to mount at least 1 of the drives and get the array back up (don't tinker until you do). If you can't, the whole array has to be taken offline and reimported. I don't think you will lose everything (that's extreme) but if you don't remount a drive, you will suffer data corruption. It goes without saying, until everything is up and running again, no writing to the array, no VMs or jails running from it, all shares deactivated. Every write adds more damage.

 

[KArmstead]

The problem is that I do not know how to take the whole array offline and then reimport it. If you have a link to the documentation I can look it up and try it myself

 

[KArmstead]

The question is how do I bring a drive online that is showing offline? Can it be done from the cli?

 

[Samuel Tai]

The question is how do I bring a drive online that is showing offline? Can it be done from the cli?

Try zpool online pool01 gptid/459fccd5-2ae6-11ec-9d4b-001e678ed53e.

Alternatively, in the UI, go to Storage->Pools. When you see your pool, click the gear icon and select Status. This will bring up your pool status, similar to the zpool status output. Click the 3 dots to the right of the offline drive, and select ONLINE.

 

[homer27081990]

The question is how do I bring a drive online that is showing offline? Can it be done from the cli?

gptid/45879be1-2ae6-11ec-9d4b-001e678ed53e UNAVAIL cannot open
gptid/459fccd5-2ae6-11ec-9d4b-001e678ed53e OFFLINE

You had two drives offline in the log. Is the second one back up and working?

 

[KArmstead]

Try zpool online pool01 gptid/459fccd5-2ae6-11ec-9d4b-001e678ed53e.

I had tried this before and it does not see pool01

Alternatively, in the UI, go to Storage->Pools. When you see your pool, click the gear icon and select Status. This will bring up your pool status, similar to the zpool status output. Click the 3 dots to the right of the offline drive, and select ONLINE.

That is not available, I would have clicked that if it was.

I feel that it is lost at this point.

You had two drives offline in the log. Is the second one back up and working?

It still shows that same that it has shown when I use the cli and zpool import.

 

[Samuel Tai]

OK, pool01 is UNAVAILABLE, because 2 members are out of this RAIDZ1 pool, and it's not mounted anymore to perform any commands. This also makes the pool non-importable.

I feel that it is lost at this point.

Unfortunately, I have to agree with you. If you have backups, you can wipe the drives and recreate your pool from backups. If you don't have backups, and need the data on your pool, your only recourse is recovery software like Klennet ZFS Recovery.

 

[KArmstead]

Ok, then I will rebuild it. Thank you for the help.

 

[homer27081990]

Ok, then I will rebuild it. Thank you for the help.

Best of luck!