Uh-Oh Zpool Offline - :(

[artlessknave]

I was reading the first post and I saw the poolname "Sadness" and was just like "oh, no, this foreshadowing is scary!"
far too much like planning for failure for me to brave enough to try that....

 

[artlessknave]

then I saw you had backups and was like "ok, the world is stormy but at least it's not ending"

 

[NickF]

I was reading the first post and I saw the poolname "Sadness" and was just like "oh, no, this foreshadowing is scary!"
far too much like planning for failure for me to brave enough to try that....

There’s a story behind this.

I had a pair of DNS servers called serenity and tranquility that lived a very short life and failed in epic proportions. I replaced them with servers called misery and sadness, and they ran for years.

My hope was that by naming the pool something which embraced bad vibes I would be preventing a jinx or something. Obviously that didn’t work :)

If nothing else I hope this thread in general serves as a cautionary tale

 

[artlessknave]

hah. well that makes more sense then.

 

[NickF]

So I've gotten quote a few of my files moving off the pool. Based on my calculations, I've eaten through my cold spares and some other pools I have in my lab. I will have JUST ENOUGH space to get all of my data off if I can remove the two drives marked as spare in the broken pool.

Is that safe?

root@prod[/sadness/movies/need rencode]# zpool status


pool: sadness
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
scan: resilvered 9.09G in 00:01:55 with 23920 errors on Mon Apr 17 02:09:10 2023
remove: Removal of vdev 14 copied 3.06G in 0h0m, completed on Fri Dec 30 16:59:46 2022
214K memory used for removed device mappings
config:

NAME STATE READ WRITE CKSUM
sadness DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
255f91c5-6fd8-4d11-bfe1-bb0b0995bde1 ONLINE 0 0 0
2db30682-bb8d-44b4-8279-960e7071ed66 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
e3fbe854-0307-473e-9f39-37a84d4747d1 ONLINE 0 0 0
49e58faf-2b18-43b6-bd50-29ef9c9bc30f ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
480d7ade-f786-4511-bb76-6e7c0b64ab48 ONLINE 0 0 0
a6b0d83f-4413-45af-91bb-f26a27c56165 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
10ebed85-ab73-472c-b556-c25c14afd966 ONLINE 0 0 0
a299b22e-e339-4e48-8c5b-a980a4057237 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
762e8aa7-1be0-4e75-b297-f53161ecb047 ONLINE 0 0 0
a1f3d1eb-55e0-4e4a-8015-20c372c3001a ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
d2aef666-ff6a-4d4a-9442-cd70f409f43c ONLINE 0 0 0
8d399a3a-7ecc-496f-bfbd-6ae48a2f89ee ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
4b60c0ba-f4cf-477a-b230-cb8c4e310112 ONLINE 0 0 0
749b9f6f-c208-4900-b210-e623146c830f ONLINE 0 0 0
mirror-7 ONLINE 0 0 0
e44f5a4c-6463-40a2-8042-d0b9dea3a4c5 ONLINE 0 0 0
ce167dd2-9f11-4bf8-9ccb-e86042d4aa11 ONLINE 0 0 0
mirror-8 ONLINE 0 0 0
ca2fed1e-edd8-4f91-9126-a9a2f667dc34 ONLINE 0 0 0
34cfa66f-66c5-4bf1-a084-7d018f18efdd ONLINE 0 0 0
mirror-15 ONLINE 0 0 0
6d9d3acf-94b3-4819-a78c-1c23b53212a2 ONLINE 0 0 0
b976ef3b-a7eb-4347-9d36-245f738098be ONLINE 0 0 0
mirror-16 ONLINE 0 0 0
430faa5c-a3f4-44fd-8e99-5db535f146d6 ONLINE 0 0 0
a81914dd-31c0-4e83-a369-cd4568484c42 ONLINE 0 0 0
mirror-17 ONLINE 0 0 0
c2640dc1-ecde-4638-8937-169b740b88aa ONLINE 0 0 0
379daf79-69ac-4968-9a27-5a6b503bbcc4 ONLINE 0 0 0
mirror-18 ONLINE 0 0 0
822db76c-4def-4ead-9a8c-5b1175a49be8 ONLINE 0 0 0
1dd58ac2-e8b8-4afd-a358-01d5e69bd07e ONLINE 0 0 0
mirror-19 UNAVAIL 0 0 0 insufficient replicas
10910638645217480881 UNAVAIL 0 0 0 was /dev/disk/by-partuuid/9318f0f4-72fd-4ad1-8292-21e8a8
17665183671171580697 UNAVAIL 0 0 0 was /dev/disk/by-partuuid/e2f3d3c3-0033-4300-8c38-a7a565
spare-20 DEGRADED 0 0 0
e1da746c-2b0a-4297-bb4b-a30088cec248 DEGRADED 0 0 0 too many errors
7793a8b5-da95-4d28-893e-fdf468afdc1c ONLINE 0 0 0
special
mirror-11 ONLINE 0 0 0
63de864e-c4c4-41c0-b495-1bd1fd723c64 ONLINE 0 0 0
be807c16-c6d9-417e-a5b9-19b6af5ec837 ONLINE 0 0 0
mirror-12 ONLINE 0 0 0
cad9688f-85b8-41f2-8f89-5a66c67789a7 ONLINE 0 0 0
323a1964-db36-40c2-be4f-93bc2cb24843 ONLINE 0 0 0
cache
81f7d0b3-f493-4674-8aeb-6e1da5059321 ONLINE 0 0 0
d41f4ac8-4850-46a9-ae77-6fb0a4317b58 ONLINE 0 0 0
spares
7793a8b5-da95-4d28-893e-fdf468afdc1c INUSE currently in use


```'

If it is, how do I remove drives by the UUID? I've only ever done it by /dev/adax or whatever.

 

[artlessknave]

that is REALLY hard to read. was that code tags or quote?

spares can be removed anytime from a healthy pool. im not sure about an unhealthy pool, but it should apply the same.
also, one of your spares appears to be unhealthly

additionally, mirrors can be evacuated from a pool, although I'm not sure how or if that would work on a degraded pool
spares should just be something like
zpool detach UUID
could be remove.

 

[Davvo]

As you get closer to 100% you seed dradtically drops.

 

[NickF]

If anyone sees this and also cares, I've put in a feature request for multipathd

[NAS-121576] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[NickF]

For those following along at home,
You CANNOT remove a device - even a spare - from a read-only dataset. With some cajiggering, I was able to remove the UUID from the spare disks, and ignore the fact that the "spares" were "damaged" in the pool. I was then able to use them as "new" disks for a differant pool.

With the one VDEV damaged there was some data loss, but that particular VDEV was very new and so it was minimal. Most of the lost data was successfully backed up to my off-site backup at a friend's house, and I can pull it down (while he has gigabit download, he has only 30 Mbps upload, which is why I tried so hard to not rely on it).

As for the rest of the data, I copied it to some individual disks (in other words, I have 6 individual "pools" of 1 disk). To move the data off of the read only pool, I am just using

Code:

cp -r 

That process is completed. I've validated that the data is in-fact good and the files are not corrupted. ZFS will TELL YOU when there's something wrong. If this were ANY OTHER FILESYSTEM, I'd be receiving files with holes in them and I wouldn't even know. THANK YOU ZFS. In other words, if you give it a file, it will in fact give you the same file back as it was when it first got it.

Now I have re-created my pool, again as mirrors, only 7 wide for now. I am copying the data off of those "pools" and onto my "new" pool using the same process. If all goes well and nothing dies over the next day or so we should be back to square one.

Hopefully this thread acts as both a cautionary tale, as well as one that celebrates why we are all here in the first place. TN really is great.

Also, I hate shell games. I feel like most of what I have done as a professional in IT is some form of shell game. lol.

 

[artlessknave]

cp -r

I would have suggested rsync --archive --progress if I had thought of it. rsync is generally better than cp, and it's vastly superior for anything that might get interrupted, since it can handle resuming gracefully.

If this were ANY OTHER FILESYSTEM, I'd be receiving files with holes in them and I wouldn't even know.

yup.

only 7 wide for now.

with normal technospeak, this would mean you have 7 drives in a single mirror vdev. hopefully this is not the case.

Also, I hate shell games.

"shell game"?

 

[Davvo]

(in other words, I have 6 individual "pools" of 1 disk).

Oh no.

 

[NickF]

I would have suggested rsync --archive --progress if I had thought of it. rsync is generally better than cp, and it's vastly superior for anything that might get interrupted, since it can handle resuming gracefully.

yup.

with normal technospeak, this would mean you have 7 drives in a single mirror vdev. hopefully this is not the case.

"shell game"?

It's 7 2-way mirrors, so 14 drives total right now.

Oh no.

Sometimes you have to use the tools that are afforded to you, even if they aren't the right tools for the job. I'm not made of money :)

 

[artlessknave]

It's 7 2-way mirrors, so 14 drives total right now.

aha, ok. that's what i was hoping. 7 wide mirrors would technically work but be such insane overkill....