SOLVED ZFS Pool gone / corrupted after reboot: corrupted device label in non-replicated configuration

[Humongous6045]

Hey,

I am very new to all of this, and I think there is no hope for this pool, and I will have to destroy it, which would mean data loss, but I am posting here for some help if I can save it. I have searched the forum, and I can't find my issue exactly. If I missed it, sorry.

This was supposed to be my failsafe running a raidz2 :(, I should have had the 3-2-1 in place before trusting this source. Oh well, tears shed, lessons learned the hard way, as usual. Any help much appreciated.

I did use the GUI to shutdown the system and then powered it off to install a video card. The card didn't end up working for me, so I powered back up without the card just as things were before, and then this happened. These drives are brand new, not that that really means anything, other than wow, three failures seems excessive.

---

Hardware:
OS Version:TrueNAS-SCALE-22.12.4.2
Product:PowerEdge T110 II
Model:Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
Memory:16 GiB

HDDs
3 x Seagate IronWolf 8TB NAS Internal Hard Drive HDD – 3.5 Inch SATA 6Gb/s 7200 RPM 256MB Cache for RAID Network Attached Storage
1 x Western Digital 8TB WD Red Plus NAS Internal Hard Drive HDD - 5640 RPM, SATA 6 Gb/s, CMR, 128 MB Cache, 3.5" - WD80EFZZ

Hard disk controller
SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family 6 port Desktop SATA AHCI Controller (rev 04)

Network card
Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5722 Gigabit Ethernet PCI Express

---

Code:

zpool import

   pool: xxx
     id: 2250464788851145926
  state: UNAVAIL
status: One or more devices contains corrupted data.
 action: The pool cannot be imported due to damaged devices or data.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E
 config:

        xxx                                       UNAVAIL  insufficient replicas
          raidz2-0                                UNAVAIL  insufficient replicas
            8b0dfeba-db7f-4036-90f4-a2319cbd30bd  ONLINE
            4a48b028-2470-4a49-a186-83db9f1766ea  UNAVAIL
            bb45f488-07ca-4130-8c3e-4bd98a479062  UNAVAIL
            2ea3f03f-5d5d-4403-845f-e93835ff3394  UNAVAIL

 

[Arwen]

Unless you can get your missing / UNAVAIL drives working, the pool is gone.

Some causes of this are:

• Power to the drives
• Loose data cables to the drives
• Data cables to the drives
• Power supply failed
• Power supply can't supply enough power to allow the drives to spin up

Others may have additional suggestions, but I would start with these.

 

[winnielinnie]

Can you access the drives SMART data? Does the BIOS event detect them?

 

[Humongous6045]

Unless you can get your missing / UNAVAIL drives working, the pool is gone.

Some causes of this are:

• Power to the drives
• Loose data cables to the drives
• Data cables to the drives
• Power supply failed
• Power supply can't supply enough power to allow the drives to spin up

Others may have additional suggestions, but I would start with these4.

Thanks for these ideas. I reached out to a friend who is in this space.

We are going to test the drives in a known good-working server, as it should help me determine if the problem is with the drives themselves or if there's an issue with the T110 server

 

[Humongous6045]

Can you access the drives SMART data? Does the BIOS event detect them?

Also, thank you. I don't have the tools installed for the SMART data, and yes, the BIOS does detect them, and as I mentioned, I am bringing the server to a friend who will see if anything can be done.

I will update and close the thread once I determine the next steps.

 

[Arwen]

Just make sure your friend does not partition or format the drives. That has on occasion been a problem where during troubleshooting someone formatted a ZFS drive with MS-Windows. That would be bad.

 

[Humongous6045]

Hey,

Sorry for the delay in posting. The issue was resolved. Thanks everyone!

For some reason, the drives, although visible on boot-up, were in fact turned off in the BIOS. re-set the BIOS, and everything works. Being that the server is 12 years old, my friend replaced the CMOS battery in case there was some issue there, and as Arwen mentioned, he also recommended and replaced the cables for the drives and said to find a backup power supply.