Removed a device, now my pool stopped working due to "corruption"

[Hexanilix]

Hi guys,
I'm having this small issue with my pool. It's not starting up due to a drive being corrupt.
Now I'll quickly tell you what I did for more info. I baisacaly added 4 usb stick and 2 sd cards to my existing pool consisting of one 1tb hard drive kinda as a joke. What I did was disconnect the one of the sd cards from the system in an attempt to remove it from the pool but also to see what will happen. Well, now I know. There is only 1 drive that has any data on it from all 7 in the system but it won't start just cuz that sd card corrupted it's self.

Here is what it throws at me with zpool import:

That one UNAVALL drive is the one I unplugged, and all others are 4 usb stick and a nother sd card (plz don't ask why they showed up as drives so i used them) while one is also my 1tb hard drive with all of the acctual data. Please help me here since this pool has a stupid amount of important data (I'm very dumb for messing around with this ik)

 

[Ericloewe]

Well, you're rather screwed and should've paid attention to the bold red warnings you got when you

added 4 usb stick and 2 sd cards to my existing pool consisting of one 1tb hard drive kinda as a joke

So, either get the missing SD card back intact or say bye to your data. If you get extremely lucky, you might be able to import the pool read-only and extract most of your data.

 

[Hexanilix]

Ok, let me give you extra info bout this. The thing is I didn't do anything with the data on there. I didn't write, change, or even really read anything. The time between me adding the drives to the pool and removing the card was in fact I'd say 8h but I only opened a folder in there once for a second and closed it. I'd like to ask if I could get away with just removing the card from the vdev

 

[Hexanilix]

Ok, let me give you extra info bout this. The thing is I didn't do anything with the data on there. I didn't write, change, or even really read anything. The time between me adding the drives to the pool and removing the card was in fact I'd say 8h but I only opened a folder in there once for a second and closed it. I'd like to ask if I could get away with just removing the card from the vdev

Oh and also I disconnected the card while the system was off cuz I'm not that dumb as I may seem. Although I did plug it back in while it was on

 

[jgreco]

one 1tb hard drive kinda as a joke.

Unfortunately, ZFS is well known for its lack of a sense of humor. It didn't think that was funny, and now it cannot puzzle how to put Humpty Dumpty back together again.

There are companies out there that specialize in recovery from ZFS pools, and I would expect one of them will be happy to charge you a good amount to recover your data, which I'm guessing is totally recoverable if things are as described. However, recovery of this sort is generally beyond the scope of this forum. We don't have the specialized software tools, and only a few of us have any expertise of the type necessary.

I do wonder if it could be possible to just rollback some txg's during import to see if that might get you there. It could be worth it to make a full copy of the contents of all devices and then try that. This is probably the only thing that would be accessible to a non-ZFS guru as remediation here.

 

[Hexanilix]

Tanks to both of you for your help I appreciate it. I just have a few questions. How does a stripe array work? Does it split the data between all the drives or fill them one by one? Does TrueNas Core write existing data to a added drive in a stripe, like when I added the flash storage did the data split between them or did the data just stay as it it just with those extra drives?

 

[danb35]

I'm very dumb for messing around with this ik

I don't want to pile on, but yes, really, this is the case--if you're storing important data, you really need to understand how the filesystem works. You've created a striped pool, which can't tolerate the failure of any device--as you're seeing. If your data is important, you should be setting up the pool with redundancy, so your pool will tolerate the failure of at least one device, and perhaps more.

The good news is that this is reversible, iff you can put that SD card back online. If you can, you can then remove the "joke" devices from your pool, and return it to the single disk you meant it to be. And you can then add a second disk as a mirror of the first, so that you'll have two copies of your data.

 

[Ericloewe]

Ok, let me give you extra info bout this. The thing is I didn't do anything with the data on there. I didn't write, change, or even really read anything. The time between me adding the drives to the pool and removing the card was in fact I'd say 8h

Which is about 7-and-half hours too long for a decent chance of rolling this back...

How does a stripe array work? Does it split the data between all the drives or fill them one by one? Does TrueNas Core write existing data to a added drive in a stripe, like when I added the flash storage did the data split between them or did the data just stay as it it just with those extra drives?

Old data is not moved, new data gets split "evenly", according to ZFS' cost function.

 

[Hexanilix]

Which is about 7-and-half hours too long for a decent chance of rolling this back...


Old data is not moved, new data gets split "evenly", according to ZFS' cost function.

1. So even tho it was unplugged for like 10 mins it's that bad?
2. So in theory, I could get away with just removing the drive from the pool? Prob not though

 

[danb35]

So in theory, I could get away with just removing the drive from the pool?

Yes, but it has to be online first. If that SD card is still available and you haven't damaged it or its contents, put it back in and your pool should be fine. Once you've done that, you can remove it and all the other "joke" devices, and you'll be back to a single drive holding all your valuable data with no redundancy.

 

[Ericloewe]

1. So even tho it was unplugged for like 10 mins it's that bad?

10 minutes, 10 days, irrelevant. You need to get it back.

 

[Hexanilix]

10 minutes, 10 days, irrelevant. You need to get it back.

Wait is there a way to wipe that card while still having it connected to that pool?

 

[jgreco]

Wait is there a way to wipe that card while still having it connected to that pool?

Sure. Of course. You wipe the drive, with DBAN or dd or whatever. But it would destroy the pool.

 

[Ericloewe]

For avoidance of doubt do not wipe the card.

 

[Hexanilix]

Crap ok. How to I uncorrupt it then or what do I do here?

 

[danb35]

what do I do here?

Have you not been seeing my messages? Put the SD card back into the system, and then import the pool--if you haven't done anything to it, this ought to be possible. Once the pool comes online, through the GUI, remove every device except for the hard drive.

 

[Hexanilix]

Have you not been seeing my messages? Put the SD card back into the system, and then import the pool--if you haven't done anything to it, this ought to be possible. Once the pool comes online, through the GUI, remove every device except for the hard drive.

Yea it's in and it says it's corrupted. That's why I'm asking what I do

 

[danb35]

Yea it's in and it says it's corrupted.

OK, somehow you never got around to mentioning that you'd put it back in. Did you do anything else with it while it was removed from the server?

In any event, if ZFS sees that device as corrupted, your options are pretty limited. To see if you can do a forced import--perhaps with losing some data--try zpool import -Fn drive and see what it tells you.

 

[Hexanilix]

OK, somehow you never got around to mentioning that you'd put it back in. Did you do anything else with it while it was removed from the server?

In any event, if ZFS sees that device as corrupted, your options are pretty limited. To see if you can do a forced import--perhaps with losing some data--try zpool import -Fn drive and see what it tells you.

I only put it into my computer with windows but didn't format it or anything.

 

[Hexanilix]

Ok now hold on a minute. Is a stripe pool supposed to not work when a drive is not present?