I replaced the wrong drive, but I still hold some hope of recovery -- can you help?

[TIMOOOO]

I have a single-parity raid array with 5 6TB drives running under FreeNAS-11.3-U5. One drive was reporting a SMART error, so this morning I decided to proactively replace it. However, I foolishly matched the model number of the drive instead of the serial number, and ended up removing and erasing the wrong drive.

Now, I have added the erased drive back into my NAS. I've also left the original, SMART-erroring drive in. But my pool's status is "unknown" and I can't recover.

My hope is that there is some way for me to tell FreeNAS that the drive that was failing still has useful parity data, then to use it to rebuild the non-failing drive I erased, then to (properly) replace the failing drive.

My problem is that I do not know how to do this. zpool status -v results in no information about the pool that is in the unknown state (nothing in this code block is likely relevant -- it lists two other pools on my NAS that are working properly, and does NOT list the pool that is in the "unknown" state):

Code:

root@freenas[~]# zpool status -v

  pool: Mirror
 state: ONLINE
  scan: resilvered 13.2G in 0 days 00:01:56 with 0 errors on Thu Sep  9 09:21:38 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        Mirror                                          ONLINE       0     0 0
          mirror-0                                      ONLINE       0     0 0
            gptid/ca6343ff-0767-11eb-b65b-086266a243e6  ONLINE       0     0 0
            gptid/ac89273a-1189-11ec-b465-086266a243e6  ONLINE       0     0 0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:06 with 0 errors on Thu Sep 16 03:45:06 2021
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          ada2p2    ONLINE       0     0     0

When I run zpool import, it gives me info about the pool I want to fix:

Code:

root@freenas[~]# zpool import
   pool: TIMOSTATION
     id: 16831627943878747789
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        TIMOSTATION                                     UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            gptid/8c69fc78-fdef-11ea-8fdf-086266a243e6  ONLINE
            gptid/8cce67dd-fdef-11ea-8fdf-086266a243e6  ONLINE
            14458387268443437899                        UNAVAIL  cannot open
            gptid/ded2ee6c-0b7d-11ec-9798-086266a243e6  ONLINE
            4321565689443348002                         OFFLINE

4321565689443348002 is the drive that I offlined to replace, the drive that had a SMART error but that, I'm hoping, still has good data.
14458387268443437899 is the drive that I incorrectly removed and erased.
But when I try to run zpool online TIMOSTATION 4321565689443348002, it just tells me that there is 'no such pool'.

Is there any way to do the thing I am describing?

Thanks for any advice you can provide.

 

[Ericloewe]

4321565689443348002 is the drive that I offlined to replace, the drive that had a SMART error but that, I'm hoping, still has good data.

So put it back.

14458387268443437899 is the drive that I incorrectly removed and erased.

Why'd you erase it? At least wait until you're sure the system is okay... Next, more relevant question, how did you erase it? Did you just nuke the partitions? That ought to be recoverable fairly easily.

 

[TIMOOOO]

So put it back.

I did! :) But it says it's "offline", and when I try to change that, it complains that the pool does not exist.

...how did you erase it? Did you just nuke the partitions? That ought to be recoverable fairly easily.

I nuked the partitions, formatted it, and began overwriting it with random data. Obviously, this was a huge mistake and I will proceed more carefully in the future, but here we are.

 

[Ericloewe]

Well, that's going to render it likely gone, unless it was a few MB at most.

I did! :) But it says it's "offline", and when I try to change that, it complains that the pool does not exist.

Ok, so you offlined the correct disk and physically removed the wrong one... In that case, try zpool online TIMOSTATION 4321565689443348002 and pray that the disk is in good enough shape to get your data off of it safely.

 

[TIMOOOO]

Ok, so you offlined the correct disk and physically removed the wrong one... In that case, try zpool online TIMOSTATION 4321565689443348002 and pray that the disk is in good enough shape to get your data off of it safely.

I tried this, but it responds:
cannot open 'TIMOSTATION': no such pool

I'm guessing that it's not even trying to look at the disk's data and it's stopping early because the pool is missing 2 disks. Is there any way around that?

 

[Ericloewe]

Hmm... I think a reboot might help clear the "offlined" status.

 

[TIMOOOO]

Hmm... I think a reboot might help clear the "offlined" status.

I gave this a try, but no luck (I was checking in the Pool tab in the GUI -- please let me know if I should be using the CLI). The Pool tab still showed an UNKNOWN status for my pool, with no drives listed in it.
I also tried removing the drive that I incorrectly erased and rebooting.

 

[NugentS]

I think you have killed your pool
Do you have a backup?

 

[TIMOOOO]

I think you have killed your pool
Do you have a backup?

Unfortunately, I have no backup, but fortunately nothing on these disks was irreplaceable.

I am considering executing zpool import -F TIMOSTATION, but when I run the command with -n I get no output, which I think is ominous. But I feel like I am running out of options.

 

[Ericloewe]

Did you list the available pools with zpool import first? It's not clear if the GUI has cached that information about the pool or if it indeed is not available for importing.

 

[TIMOOOO]

Did you list the available pools with zpool import first? It's not clear if the GUI has cached that information about the pool or if it indeed is not available for importing.

Yeah, the output of the command "zpool import" is in the second code block in my original post (let me know if you mean something else).

 

[Ericloewe]

Yeah, the output of the command "zpool import" is in the second code block in my original post (let me know if you mean something else).

But that's from before the reboot, right?

 

[TIMOOOO]

But that's from before the reboot, right?

Oh, I understand now. No, this is from after several reboots.

The process I went through was:
1. Offline the drive with the SMART errors
2. Shut down the system
3. Remove the wrong drive and start erasing it
4. Replace the wrong drive with a new drive
5. Start up, notice issue
6. Shut down and replace new drive with old, partially-formatted drive (this is a drive I never meant to remove, since it was exhibiting no errors)

So all of the info I've been giving you has been after these actions were performed. I have also restarted the system several times since then. I have also used the FreeNAS wipe option on the drive I already started erasing, thinking that maybe my partial erase was making it impossible for the system to do anything with. Unfortunately, this didn't change anything.

 

[Ericloewe]

Alright then, doesn't sound like there's much that can be easily done here. We're deep in HDD data recovery territory.