My pool has gone unavailable due to 1 drive dying and another drives part ion UUID does not match.
I am running the newest version of TrueNAS Scale. RAIDZ1 with 4 12TB drives and 2 SSDs. The back-plane of my dell T330 was starting to fail. I ejected the drive and put it in another bay but I it looks like the damage was already done.
Fast forward and I built a new system and this time setting it up in Proxmox, passing through a 12 port SATA card, uploading the config, etc. The 12TB drive is still unavailable. So I ordered in a few 10TB drives, made a new pool and started moving everything to it. Then after a reboot the cache drive started showing up as unavailable. Now with 2 drives down I can no longer get the pool online.
I matched up the drives to the pool and noticed that the cache drive does not match the UUID of any of the drives. I have tried new TrueNAS VMs, new installs of Proxmox, running TrueNAS on bare metal and even putting the drives back in the old server. I even tried repairing the bad 12TB drive.
Any help would be greatly appreciated.
pool: tank id: 4695044492445768575 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config:
tank UNAVAIL insufficient replicas
raidz1-0 UNAVAIL insufficient replicas
180a178f-227a-4f59-bf63-13c204ea5e3d ONLINE sdb ZTN0E7JS
bd500474-e688-4ad9-8218-3baedaae18e3 ONLINE sde ZL006VBP
61f8b3c1-acdf-4536-8fa9-7ebc7081a3d0 UNAVAIL
ce8e3970-ffa6-4648-8fda-da5518ff2248 ONLINE sdg ZHZ75WHF
2bf0f77f-986f-45a9-96bc-61c6e8a6b43e UNAVAIL
b36b9199-be21-4bb0-9390-b94902ff0987 ONLINE sdd 222303A00691
smartctl -a /dev/sda smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.107+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION === Model Family: Seagate Exos X14 Device Model: ST12000NM0008-2H3101 Serial Number: ZHZ3DPZ7 LU WWN Device Id: 5 000c50 0c29be84f Firmware Version: SN03 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Oct 13 01:11:56 2023 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
Read SMART Data failed: scsi error badly formed scsi parameters
=== START OF READ SMART DATA SECTION === SMART Status command failed: scsi error badly formed scsi parameters SMART overall-health self-assessment test result: UNKNOWN! SMART Status, Attributes and Thresholds cannot be read.
Read SMART Log Directory failed: scsi error badly formed scsi parameters
Read SMART Error Log failed: scsi error badly formed scsi parameters
Read SMART Self-test Log failed: scsi error badly formed scsi parameters
Selective Self-tests/Logging not supported
fdisk -x /dev/sdf
Disk /dev/sdf: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors Disk model: ST12000NM0008-2H Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: EB48CE70-4290-4FC9-9B75-B84B757886AD First LBA: 34 Last LBA: 23437770718 Alternative LBA: 23437770751 Partition entries LBA: 2 Allocated partition entries: 128
Device Start End Sectors Type-UUID UUID Name Attrs /dev/sdf1 2048 4194304 4192257 0657FD6D-A4AB-43C4-84E5-0933C84B4F4F 2E1C447A-E993-4F85-A36D-12862075E4F5
/dev/sdf2 4196352 23437770718 23433574367 6A898CC3-1DD2-11B2-99A6-080020736631 6CD68B47-7B86-4E7B-B33E-AD8BE3F81D72
fdisk -x /dev/sdc Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: SanDisk SSD PLUS Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: FFB2375D-4962-48F3-8B68-89CE87D03723 First LBA: 34 Last LBA: 1953525134 Alternative LBA: 1953525167 Partition entries LBA: 2 Allocated partition entries: 128
Device Start End Sectors Type-UUID UUID Name Attrs /dev/sdc1 40 1953525134 1953525095 6A898CC3-1DD2-11B2-99A6-080020736631 4D52B9D4-014B-4AE0-B3B4-2276D1C95264
I am running the newest version of TrueNAS Scale. RAIDZ1 with 4 12TB drives and 2 SSDs. The back-plane of my dell T330 was starting to fail. I ejected the drive and put it in another bay but I it looks like the damage was already done.
Fast forward and I built a new system and this time setting it up in Proxmox, passing through a 12 port SATA card, uploading the config, etc. The 12TB drive is still unavailable. So I ordered in a few 10TB drives, made a new pool and started moving everything to it. Then after a reboot the cache drive started showing up as unavailable. Now with 2 drives down I can no longer get the pool online.
I matched up the drives to the pool and noticed that the cache drive does not match the UUID of any of the drives. I have tried new TrueNAS VMs, new installs of Proxmox, running TrueNAS on bare metal and even putting the drives back in the old server. I even tried repairing the bad 12TB drive.
Any help would be greatly appreciated.
pool: tank id: 4695044492445768575 state: UNAVAIL status: One or more devices contains corrupted data. action: The pool cannot be imported due to damaged devices or data. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-5E config:
tank UNAVAIL insufficient replicas
raidz1-0 UNAVAIL insufficient replicas
180a178f-227a-4f59-bf63-13c204ea5e3d ONLINE sdb ZTN0E7JS
bd500474-e688-4ad9-8218-3baedaae18e3 ONLINE sde ZL006VBP
61f8b3c1-acdf-4536-8fa9-7ebc7081a3d0 UNAVAIL
ce8e3970-ffa6-4648-8fda-da5518ff2248 ONLINE sdg ZHZ75WHF
2bf0f77f-986f-45a9-96bc-61c6e8a6b43e UNAVAIL
b36b9199-be21-4bb0-9390-b94902ff0987 ONLINE sdd 222303A00691
smartctl -a /dev/sda smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.107+truenas] (local build) Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION === Model Family: Seagate Exos X14 Device Model: ST12000NM0008-2H3101 Serial Number: ZHZ3DPZ7 LU WWN Device Id: 5 000c50 0c29be84f Firmware Version: SN03 User Capacity: 12,000,138,625,024 bytes [12.0 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-4 (minor revision not indicated) SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Fri Oct 13 01:11:56 2023 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled
Read SMART Data failed: scsi error badly formed scsi parameters
=== START OF READ SMART DATA SECTION === SMART Status command failed: scsi error badly formed scsi parameters SMART overall-health self-assessment test result: UNKNOWN! SMART Status, Attributes and Thresholds cannot be read.
Read SMART Log Directory failed: scsi error badly formed scsi parameters
Read SMART Error Log failed: scsi error badly formed scsi parameters
Read SMART Self-test Log failed: scsi error badly formed scsi parameters
Selective Self-tests/Logging not supported
fdisk -x /dev/sdf
Disk /dev/sdf: 10.91 TiB, 12000138625024 bytes, 23437770752 sectors Disk model: ST12000NM0008-2H Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: EB48CE70-4290-4FC9-9B75-B84B757886AD First LBA: 34 Last LBA: 23437770718 Alternative LBA: 23437770751 Partition entries LBA: 2 Allocated partition entries: 128
Device Start End Sectors Type-UUID UUID Name Attrs /dev/sdf1 2048 4194304 4192257 0657FD6D-A4AB-43C4-84E5-0933C84B4F4F 2E1C447A-E993-4F85-A36D-12862075E4F5
/dev/sdf2 4196352 23437770718 23433574367 6A898CC3-1DD2-11B2-99A6-080020736631 6CD68B47-7B86-4E7B-B33E-AD8BE3F81D72
fdisk -x /dev/sdc Disk /dev/sdc: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors Disk model: SanDisk SSD PLUS Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: FFB2375D-4962-48F3-8B68-89CE87D03723 First LBA: 34 Last LBA: 1953525134 Alternative LBA: 1953525167 Partition entries LBA: 2 Allocated partition entries: 128
Device Start End Sectors Type-UUID UUID Name Attrs /dev/sdc1 40 1953525134 1953525095 6A898CC3-1DD2-11B2-99A6-080020736631 4D52B9D4-014B-4AE0-B3B4-2276D1C95264