Pool offline after server failure - how to rectify?

S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
Hi,

TrueNAS core : 13.0 U2

I have recently suffered a server failure, however, the disks used by TrueNAS pool should be OK (I hope). I had TrueNAS set up as a VM in ESXi to use passthrough PCI for SATA disks. These show up in TrueNAS, but the pool is showing as offline. There was an additional slog disk as part of the pool; however, that died.

How can I recover the pool? I did try to import a pool, but nothing shows up. I see two critical notifications but cannot find the docs to get to more system logs.

CRITICAL​

Pool Pool1 state is OFFLINE: None​

2022-12-29 20:36:53 (Europe/London)

CRITICAL​

Device: /dev/ada3, 1 Currently unreadable (pending) sectors.​

2022-12-29 21:03:03 (Europe/London)

Any help would be appreciated.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
FWIW, the dashboard is also dead (shows nothing), and during boot, it looks like there might be a traceback, but I cannot read what is going on.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
OK, checking out `/var/log/message`, I can't see why the pool doesn't mount (or I don't think I can). Here are the even from when the boot poot is mounted, the SATA disks are discovered:

Code:
Dec 29 22:57:26 truenas ZFS filesystem version: 5
Dec 29 22:57:26 truenas ZFS storage pool version: features support (5000)
Dec 29 22:57:26 truenas ugen1.1: <(0x15ad) EHCI root HUB> at usbus1
Dec 29 22:57:26 truenas ugen0.1: <(0x15ad) UHCI root HUB> at usbus0
Dec 29 22:57:26 truenas uhub0 on usbus1
Dec 29 22:57:26 truenas uhub0: <(0x15ad) EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
Dec 29 22:57:26 truenas ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
Dec 29 22:57:26 truenas Trying to mount root from zfs:boot-pool/ROOT/default []...
Dec 29 22:57:26 truenas uhub1 on usbus0
Dec 29 22:57:26 truenas uhub1: <(0x15ad) UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Dec 29 22:57:26 truenas uhub1: 2 ports with 2 removable, self powered
Dec 29 22:57:26 truenas Root mount waiting for: CAM usbus0 usbus1
Dec 29 22:57:26 truenas ugen0.2: <VMware VMware Virtual USB Mouse> at usbus0
Dec 29 22:57:26 truenas uhid0 on uhub1
Dec 29 22:57:26 truenas uhid0: <VMware> on usbus0
Dec 29 22:57:26 truenas ums0 on uhub1
Dec 29 22:57:26 truenas ums0: <VMware> on usbus0
Dec 29 22:57:26 truenas ums0: 16 buttons and [XYZT] coordinates ID=0
Dec 29 22:57:26 truenas Root mount waiting for: CAM usbus0 usbus1
Dec 29 22:57:26 truenas ugen0.3: <VMware, Inc. VMware Virtual USB Hub> at usbus0
Dec 29 22:57:26 truenas uhub2 on uhub1
Dec 29 22:57:26 truenas uhub2: <VMware, Inc.> on usbus0
Dec 29 22:57:26 truenas uhub0: 6 ports with 6 removable, self powered
Dec 29 22:57:26 truenas uhub2: 7 ports with 7 removable, self powered
Dec 29 22:57:26 truenas da0 at mpt0 bus 0 scbus32 target 0 lun 0
Dec 29 22:57:26 truenas da0: <VMware Virtual disk 2.0> Fixed Direct Access SPC-4 SCSI device
Dec 29 22:57:26 truenas da0: 300.000MB/s transfers
Dec 29 22:57:26 truenas da0: Command Queueing enabled
Dec 29 22:57:26 truenas da0: 16384MB (33554432 512 byte sectors)
Dec 29 22:57:26 truenas da0: quirks=0x140<RETRY_BUSY,STRICT_UNMAP>
Dec 29 22:57:26 truenas cd0 at ahcich0 bus 0 scbus2 target 0 lun 0
Dec 29 22:57:26 truenas cd0: <NECVMWar VMware SATA CD00 1.00> Removable CD-ROM SCSI device
Dec 29 22:57:26 truenas cd0: Serial Number 00000000000000000001
Dec 29 22:57:26 truenas cd0: 600.000MB/s transfers (SATA 3.x, UDMA2, ATAPI 12bytes, PIO 8192bytes)
Dec 29 22:57:26 truenas cd0: Attempt to query device size failed: NOT READY, Medium not present
Dec 29 22:57:26 truenas ses0 at ahciem0 bus 0 scbus37 target 0 lun 0
Dec 29 22:57:26 truenas ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
Dec 29 22:57:26 truenas ses0: SEMB SES Device
Dec 29 22:57:26 truenas ada0 at ahcich30 bus 0 scbus33 target 0 lun 0
Dec 29 22:57:26 truenas ada0: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
Dec 29 22:57:26 truenas ada0: Serial Number WD-WCC4N1XKCXV8
Dec 29 22:57:26 truenas ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 29 22:57:26 truenas ada0: Command Queueing enabled
Dec 29 22:57:26 truenas ada0: 2861588MB (5860533168 512 byte sectors)
Dec 29 22:57:26 truenas ada0: quirks=0x1<4K>
Dec 29 22:57:26 truenas ses0: pass2,ada0 in 'Slot 00', SATA Slot: scbus33 target 0
Dec 29 22:57:26 truenas ada1 at ahcich31 bus 0 scbus34 target 0 lun 0
Dec 29 22:57:26 truenas ada1: <WDC WD30EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
Dec 29 22:57:26 truenas ada1: Serial Number WD-WCC7K7FE14K1
Dec 29 22:57:26 truenas ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 29 22:57:26 truenas ada1: Command Queueing enabled
Dec 29 22:57:26 truenas ada1: 2861588MB (5860533168 512 byte sectors)
Dec 29 22:57:26 truenas ada1: quirks=0x1<4K>
Dec 29 22:57:26 truenas ses0: pass3,ada1 in 'Slot 01', SATA Slot: scbus34 target 0
Dec 29 22:57:26 truenas ada2 at ahcich32 bus 0 scbus35 target 0 lun 0
Dec 29 22:57:26 truenas ada2: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
Dec 29 22:57:26 truenas ada2: Serial Number WD-WCC4N1FZ5X6T
Dec 29 22:57:26 truenas ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 29 22:57:26 truenas ada2: Command Queueing enabled
Dec 29 22:57:26 truenas ada2: 2861588MB (5860533168 512 byte sectors)
Dec 29 22:57:26 truenas ada2: quirks=0x1<4K>
Dec 29 22:57:26 truenas ses0: pass4,ada2 in 'Slot 02', SATA Slot: scbus35 target 0
Dec 29 22:57:26 truenas ada3 at ahcich33 bus 0 scbus36 target 0 lun 0
Dec 29 22:57:26 truenas ada3: <WDC WD30EFRX-68EUZN0 82.00A82> ACS-2 ATA SATA 3.x device
Dec 29 22:57:26 truenas ada3: Serial Number WD-WCC4N1FZ59AD
Dec 29 22:57:26 truenas ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
Dec 29 22:57:26 truenas ada3: Command Queueing enabled
Dec 29 22:57:26 truenas ada3: 2861588MB (5860533168 512 byte sectors)
Dec 29 22:57:26 truenas ada3: quirks=0x1<4K>
Dec 29 22:57:26 truenas ses0: pass5,ada3 in 'Slot 03', SATA Slot: scbus36 target 0
Dec 29 22:57:26 truenas intsmb0: <Intel PIIX4 SMBUS Interface> port 0x1040-0x104f at device 7.3 on pci0
Dec 29 22:57:26 truenas intsmb0: intr SMI disabled revision 0
Dec 29 22:57:26 truenas smbus0: <System Management Bus> on intsmb0
Dec 29 22:57:26 truenas kernel: lo0: link state changed to UP
Dec 29 22:57:26 truenas vmx_modinit: processor does not support VMX operation
Dec 29 22:57:26 truenas module_register_init: MOD_LOAD (vmm, 0xffffffff83604200, 0) error 6
Dec 29 22:57:27 truenas CPU: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz (2397.22-MHz K8-class CPU)
Dec 29 22:57:27 truenas Origin="GenuineIntel"  Id=0x406f1  Family=0x6  Model=0x4f  Stepping=1
Dec 29 22:57:27 truenas Features=0xf8bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,SS>
Dec 29 22:57:27 truenas Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
Dec 29 22:57:27 truenas AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Dec 29 22:57:27 truenas AMD Features2=0x121<LAHF,ABM,Prefetch>
Dec 29 22:57:27 truenas Structured Extended Features=0x1c27ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,RDSEED,ADX,SMAP>
Dec 29 22:57:27 truenas Structured Extended Features3=0xbc000400<MD_CLEAR,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
Dec 29 22:57:27 truenas XSAVE Features=0x1<XSAVEOPT>
Dec 29 22:57:27 truenas IA32_ARCH_CAPS=0xc<RSBA,SKIP_L1DFL_VME>
Dec 29 22:57:27 truenas TSC: P-state invariant
Dec 29 22:57:27 truenas Hypervisor: Origin = "VMwareVMware"
Dec 29 22:57:27 truenas VMware memory control driver initialized
Dec 29 22:57:27 truenas kernel: vmx0: link state changed to UP
Dec 29 22:57:27 truenas hwpc_core: unknown PMC architecture: 1
Dec 29 22:57:27 truenas hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>
Dec 29 22:57:27 truenas 1 2022-12-29T22:57:27.076306+00:00 truenas.local mountd 1104 - - bad exports list line '/mnt/Pool1/ISO': symbolic link in export path or statfs failed
Dec 29 22:57:27 truenas 1 2022-12-29T22:57:27.076673+00:00 truenas.local mountd 1104 - - bad exports list line '/mnt/Pool1/VM_Backup': symbolic link in export path or statfs failed
Dec 29 22:57:27 truenas 1 2022-12-29T22:57:27.076691+00:00 truenas.local mountd 1104 - - bad exports list line '/mnt/Pool1/PC_Backup': symbolic link in export path or statfs failed
Dec 29 22:57:27 truenas nfsd: can't register svc name


After ntp, we see the issue with the third HDD logged regarding bad sectors, but I don't think this would stop the pool from mounting, then, 20+ min after boot, there is a sig 6 core dump, although this again doesn't indicate what happens to the pool during boot.

Code:
Dec 29 22:57:30 truenas 1 2022-12-29T22:57:30.761849+00:00 truenas.local daemon 1239 - - 2022-12-29 22:57:30,755:wsdd WARNING(pid 1240): no interface given, using all interfaces
Dec 29 22:57:31 truenas 1 2022-12-29T22:57:31.268806+00:00 truenas.local smartd 1255 - - Device: /dev/ada3, 1 Currently unreadable (pending) sectors
Dec 29 22:57:43 truenas 1 2022-12-29T22:57:43.246051+00:00 truenas.local daemon 1498 - - 2022-12-29 22:57:43,245:wsdd WARNING(pid 1499): no interface given, using all interfaces
Dec 29 23:21:15 truenas 1 2022-12-29T23:21:15.708868+00:00 truenas.local middlewared 222 - - ECDSA Fingerprint of the SSH KEY: SHA256:xxxxxxxxxxxx
Dec 29 23:21:16 truenas syslog-ng[1899]: syslog-ng starting up; version='3.35.1'
Dec 29 23:21:16 truenas kernel: pid 1056 (syslog-ng), jid 0, uid 0: exited on signal 6 (core dumped)
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
Ah, in `/var/log/console`, I see:

Code:
Dec 29 22:57:27 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing


Please don't say that this is screwed :|
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
And the full logs from console:

Code:
Dec 29 22:57:26 truenas Importing Pool1
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939040
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939040
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): spa_load_retry: rewind, max txg: 939039
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939039
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939039
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): spa_load_retry: rewind, max txg: 939038
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939038
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939038
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): spa_load_retry: rewind, max txg: 939037
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939036
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939036
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:26 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:26 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:26 truenas spa.c:6242:spa_tryimport(): spa_tryimport: importing Pool1
Dec 29 22:57:26 truenas spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): LOADING
Dec 29 22:57:26 truenas spa.c:8358:spa_async_request(): spa=$import async request task=1
Dec 29 22:57:26 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa $import. txg 939040
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=939040
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas spa_misc.c:404:spa_load_failed(): spa_load($import, config trusted): FAILED: some log devices are missing
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load($import, config trusted): UNLOADING
Dec 29 22:57:27 truenas spa.c:6098:spa_import(): spa_import: importing Pool1
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939040
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939040
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): spa_load_retry: rewind, max txg: 939039
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939039
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939039
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): spa_load_retry: rewind, max txg: 939038
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939038
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939038
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas spa_misc.c:404:spa_load_failed(): spa_load(Pool1, config trusted): FAILED: some log devices are missing
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():   vdev 0: root, guid: 10244173158489965879, path: N/A, degraded
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 0: raidz, guid: 12177903282200206971, path: N/A, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 0: disk, guid: 13976681131252786576, path: /dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 1: disk, guid: 17866708632328693978, path: /dev/gptid/54b63620-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 2: disk, guid: 5406912277672557823, path: /dev/gptid/546086c5-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():       vdev 3: disk, guid: 12928131187433198755, path: /dev/gptid/549656d2-5264-11ed-8151-000c29f3c3dd, healthy
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): UNLOADING
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): spa_load_retry: rewind, max txg: 939037
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config trusted): LOADING
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd': best uberblock found for spa Pool1. txg 939036
Dec 29 22:57:27 truenas spa_misc.c:419:spa_load_note(): spa_load(Pool1, config untrusted): using uberblock with txg=939036
Dec 29 22:57:27 truenas vdev.c:153:vdev_dbgmsg(): disk vdev '/dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd': vdev_geom_open: failed to open [error=2]
Dec 29 22:57:27 truenas Pools import completed
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Please provide your full hardware specs as per forum rules.
Please paste the output of zpool status.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
Code:
root@truenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:11 with 0 errors on Sat Dec 10 03:45:12 2022
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          da0p2     ONLINE       0     0     0

errors: No known data errors


However, as seen from the logs above, the `Pool1` doesn't mount, so it doesn't appear in the list.

The host is a Dell T430 running ESXi. TrueNAS is a guest VM. As mentioned in the first post, the SATA drives are passed through to the guest using PCI passthrough from the standard Intel SATA controller. There are four WDC WD30EFRX-68EUZN0 drives in the ZFS array for `Pool1`

I hope this helps.

This drive is the cause of the issue (I believe), as it got killed accidentally when trying to recover something from the dead server.

Code:
Dec 29 22:57:27 truenas vdev.c:208:vdev_dbgmsg_print_tree():     vdev 1: disk (log), guid: 1254988834239897657, path: /dev/gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd, can't open
 
Last edited:
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
swinster said:
This drive is the cause of the issue (I believe), as it got killed accidentally when trying to recover something from the dead server.
Click to expand...
Please elaborate. How did it get "killed"?
Try zpool import and post the output.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
I royally screwed up
Davvo said:
Please elaborate. How did it get "killed"?
Click to expand...
For your amusement only - I royally screwed up and managed to overwrite it when reconfiguring ESXi on another host, unwittingly writing ESXi OS data to the drive :eek:. Safe to say, any data on it has gone.

Code:
root@truenas[~]# zpool import
   pool: Pool1
     id: 10244173158489965879
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-6X
 config:

        Pool1                                           UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/54b63620-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/546086c5-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/549656d2-5264-11ed-8151-000c29f3c3dd  ONLINE
        logs
          gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd    UNAVAIL  cannot open

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
If that was a metadata vdev you are royally screwed, there is no way to import the pool without that drive.
A SLOG should be detachable iirc, and that would being your pool back.
 
Last edited:
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
For reference, as the pool is not mounted so not viewable/editable in the GUI, I guess the following CLI command should do the trick:

Code:
zpool remove Pool1 gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd


?
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Try zpool remove Pool-1 gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
Hmm,

Code:
root@truenas[~]# zpool remove Pool1 gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd
cannot open 'Pool1': no such pool
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Try zpool import -m.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
:frown:
Code:
root@truenas[~]# zpool import -f
   pool: Pool1
     id: 10244173158489965879
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-6X
 config:

        Pool1                                           UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/54b63620-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/546086c5-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/549656d2-5264-11ed-8151-000c29f3c3dd  ONLINE
        logs
          gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd    UNAVAIL  cannot open

        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
 
Davvo

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Yeah my bad, typed f instead of m.
You should be able to mount with the -m parameter.
 
S

swinster

Dabbler
Joined
Oct 10, 2022
Messages
26
still no joy :(

Code:
root@truenas[~]# zpool import -m
   pool: Pool1
     id: 10244173158489965879
  state: UNAVAIL
status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-6X
 config:


        Pool1                                           UNAVAIL  missing device
          raidz2-0                                      ONLINE
            gptid/54ad8b2b-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/54b63620-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/546086c5-5264-11ed-8151-000c29f3c3dd  ONLINE
            gptid/549656d2-5264-11ed-8151-000c29f3c3dd  ONLINE
        logs
          gptid/52eab7bc-5264-11ed-8151-000c29f3c3dd    UNAVAIL  cannot open


        Additional devices are known to be part of this pool, though their
        exact configuration cannot be determined.
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Pool offline after server failure - how to rectify?"

Similar threads

M
Error on a disk
2
Replies
21
Views
5K
Max90
M
J
TrueNas Scale Alert - Don't Understand
Replies
3
Views
1K
Ebedorian
Ebedorian
ingestre
Sanity check needed for my approach to "8 offline uncorrectable sectors"
Replies
9
Views
3K
ingestre
ingestre
C
How to modify application configuration when using TrueCharts NextCloud
Replies
4
Views
3K
victort
victort
Z
Pool goes offline after ZFS version upgrade message appears
Replies
1
Views
2K
jgreco
jgreco
Top