Missing top level device

[ghostrunner]

Originally my system was suffering from a degraded drive I removed one of the drives in the mirror and ran WD diagnostic test no problems detected turns out it was the good drive in the mirror. I took out the second drive in the mirror which was a Seagate drive ran a check and found that the drive did indeed have issues but I failed to detach the drive properly and received my v0 volume could not determine it's size. I ran both test using a USB SATAIII cable on a Windows system. My original motivation for doing these checks was to see if the bad drive could be fixed turns out this was a bad idea because now v0 can not detect any drives and worst yet I ended up detaching the whole v0 volume which I can not auto import now. Now I'm getting a message "ZFS-80006X" when typing "zpool import".

Is there anyway I can bring the system back to the degraded drive state? (i.e. Attache v0 back to a degraded state so I can properly resilver the drive)

I should mentioned that I had periodic system config backups but they resided on v0 which I can no longer import. I was thinking about importing the drive with the configs but I"m a little afraid to continue. I also think that running the diagnostics scans on the drives did something to the zfs table but I took a look on my Arch Linux system and it shows them as ZFS file systems. I also must mention that I tried running a repair on the faulty Seagate drive but this didn't work using Segates diagnostic software.

Here is some additional background information...
======================================================================
Current system specifications:

Mobo: Gigabyte Micro ATX Motherboard GA-A75M-S2V
RAM: 16GB
CPU: AMD A4-3300 APU
SATAIII Card: IO Crest SATAII 4 Port PCI-e 2.0 x1
​

FreeNAS specifications:

Version: FreeNAS-9.2.1.8-RELEASE-x64 (e625626)
ZFS Volumes:

Volume name: v0

Size: ~8TB
Num of drives: 6
Setup: Mirrored
Config:
v0
mirror-0*

Seagate 3TB
Seagate 3TB​

mirror-1*

Seagate 2TB
Seagate 2TB​

mirror-2*

Seagate 3TB (Original degraded drive)
Western Digital 3TB​

Volume name: v1

Size: ~ 1.5TB
Setup: Striped
Num of drives: 1
Setup: Striped
Config:
v1

Seagate 1.5TB
​

*Please note that I'm not exactly sure if the mirror naming convention was this before I had the problem.
Thank you,

======================================================================
dmesg

Code:

~# dmesg
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.2-RELEASE-p12 #0 r262572+b043649: Sun Sep 28 23:03:31 PDT 2014
    root@build3.ixsystems.com:/tank/home/jkh/build/921/freenas/os-base/amd64/fusion/jkh/921/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: AMD A4-3300 APU with Radeon(tm) HD Graphics (2500.00-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x300f10  Family = 0x12  Model = 0x1  Stepping = 0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16002158592 (15260 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <GBT    GBTUACPI>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
padlock0: No ACE support.
acpi0: <GBT GBTUACPI> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, cfca0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf800-0xf8ff mem 0xd0000000-0xdfffffff,0xfdfc0000-0xfdffffff irq 18 at device 1.0 on pci0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xde00-0xdeff mem 0xfddff000-0xfddfffff,0xfddf8000-0xfddfbfff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: turning off MSI enable bit.
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 50:e5:49:cd:85:52
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci0: <Marvell 88SE9215 AHCI SATA controller> port 0xef00-0xef07,0xee00-0xee03,0xed00-0xed07,0xec00-0xec03,0xeb00-0xeb1f mem 0xfdcff000-0xfdcff7ff irq 17 at device 0.0 on pci2
ahci0: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
pci0: <serial bus, USB> at device 16.0 (no driver attached)
pci0: <serial bus, USB> at device 16.1 (no driver attached)
ahci1: <AMD Hudson-2 AHCI SATA controller> port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f mem 0xfe02f000-0xfe02f7ff irq 19 at device 17.0 on pci0
ahci1: AHCI v1.30 with 6 6Gbps ports, Port Multiplier supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
ahcich6: <AHCI channel> at channel 2 on ahci1
ahcich7: <AHCI channel> at channel 3 on ahci1
ahcich8: <AHCI channel> at channel 4 on ahci1
ahcich9: <AHCI channel> at channel 5 on ahci1
ohci0: <OHCI (generic) USB controller> mem 0xfe02e000-0xfe02efff irq 18 at device 18.0 on pci0
usbus0 on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe02d000-0xfe02d0ff irq 17 at device 18.2 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfe02c000-0xfe02cfff irq 18 at device 19.0 on pci0
usbus2 on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe02b000-0xfe02b0ff irq 17 at device 19.2 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci3: <ACPI PCI bus> on pcib3
ohci2: <OHCI (generic) USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 20.5 on pci0
usbus4 on ohci2
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe0003e24100 with IPv6 initialized (100409)
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
ugen0.1: <AMD> at usbus0
uhub0: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <AMD> at usbus1
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <AMD> at usbus2
uhub2: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <AMD> at usbus3
uhub3: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <AMD> at usbus4
uhub4: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
(aprobe5:ahcich9:0:0:0): SET_MULTI. ACB: c6 00 00 00 00 40 00 00 00 00 10 00
(aprobe5:ahcich9:0:0:0): CAM status: ATA Status Error
(aprobe5:ahcich9:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
(aprobe5:ahcich9:0:0:0): RES: 51 04 00 00 00 40 00 00 00 10 00
(aprobe5:ahcich9:0:0:0): Retrying command
(aprobe5:ahcich9:0:0:0): SET_MULTI. ACB: c6 00 00 00 00 40 00 00 00 00 10 00
(aprobe5:ahcich9:0:0:0): CAM status: ATA Status Error
(aprobe5:ahcich9:0:0:0): ATA status: 51 (DRDY SERV ERR), error: 04 (ABRT )
(aprobe5:ahcich9:0:0:0): RES: 51 04 00 00 00 40 00 00 00 10 00
(aprobe5:ahcich9:0:0:0): Error 5, Retries exhausted
ada0 at ahcich3 bus 0 scbus3 target 0 lun 0
ada0: <ST31500341AS CC1H> ATA-8 SATA 2.x device
ada0: Serial Number 9VS1S7B1
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1430798MB (2930275055 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad10
ada1 at ahcich4 bus 0 scbus4 target 0 lun 0
ada1: <ST2000DM001-9YN164 CC4H> ATA-8 SATA 3.x device
ada1: Serial Number Z1E3YEL4
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad12
ada2 at ahcich5 bus 0 scbus5 target 0 lun 0
ada2: <ST2000DM001-9YN164 CC4H> ATA-8 SATA 3.x device
ada2: Serial Number S1E0B385
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad14
ada3 at ahcich6 bus 0 scbus6 target 0 lun 0
ada3: <ST3000DM001-1CH166 CC49> ATA-9 SATA 3.x device
ada3: Serial Number Z1F4D193
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad16
ada4 at ahcich8 bus 0 scbus8 target 0 lun 0
ada4: <WDC WD30EZRX-00D8PB0 80.00A80> ATA-9 SATA 3.x device
ada4: Serial Number WD-WMC4N0909180
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad20
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1250000679 Hz quality 800
uhub4: 2 ports with 2 removable, self powered
uhub0: 5 ports with 5 removable, self powered
uhub2: 5 ports with 5 removable, self powered
GEOM: ada4: the secondary GPT header is not in the last LBA.
GEOM_PART: integrity check failed (ada4, GPT)
Root mount waiting for: usbus3 usbus1
uhub1: 5 ports with 5 removable, self powered
uhub3: 5 ports with 5 removable, self powered
Root mount waiting for: usbus3
ugen3.2: <hp> at usbus3
umass0: <hp v125w, class 0/0, rev 2.00/10.00, addr 2> on usbus3
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:11:0:-1: Attached to scbus11
Trying to mount root from ufs:/dev/ufs/FreeNASs2a [ro]...
mountroot: waiting for device /dev/ufs/FreeNASs2a ...
da0 at umass-sim0 bus 0 scbus11 target 0 lun 0
da0: <hp v125w 1.00> Removable Direct Access SCSI-4 device
da0: Serial Number 002354D4527DAC3172CF0065
da0: 40.000MB/s transfers
da0: 15300MB (31334400 512 byte sectors: 255H 63S/T 1950C)
da0: quirks=0x2<NO_6_BYTE>
GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
vboxdrv: fAsync=0 offMin=0x5cd offMax=0x1b64

======================================================================
zpool status -v

Code:

~# zpool status -v
  pool: v1
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
    still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 2h7m with 0 errors on Sun Jun  7 02:07:47 2015
config:

    NAME                                          STATE     READ WRITE CKSUM
    v1                                            ONLINE       0     0     0
      gptid/e0d7536e-53d3-11e3-9789-50e549cd8552  ONLINE       0     0     0

errors: No known data errors

I've connected all the drives but I can't seem to get the system to recognize them.

======================================================================
This is what my "View Disks" tab looks like.

======================================================================
zpool import

Code:

~# zpool import
   pool: v0
     id: 6859802526440201018
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
    devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
config:

    v0                                              UNAVAIL  missing device
      mirror-1                                      ONLINE
        gptid/47a9fa47-71bc-11e3-9a43-50e549cd8552  ONLINE
        gptid/475baea4-53cd-11e3-bc1a-50e549cd8552  ONLINE
      mirror-2                                      DEGRADED
        6440307554711772873                         UNAVAIL  cannot open
        gptid/8524bf6a-62cc-11e4-bb8a-50e549cd8552  ONLINE

    Additional devices are known to be part of this pool, though their
    exact configuration cannot be determined.

 

[mjws00]

It looks like you have a few things going on, but for sure you need to get the board and OS seeing all the drives. If they don't show up in 'dmesg' and 'camcontrol devlist' there is definitely no way that the zpool can use them. At this point, the critical bit will be getting a device back from the mirror-0 vdev.

Once you have the drives back online the pool should at least be visible even if not mounted or able to auto-import.

 

[ghostrunner]

Code:

~# camcontrol devlist
<ST31500341AS CC1H>                at scbus3 target 0 lun 0 (pass0,ada0)
<ST2000DM001-9YN164 CC4H>          at scbus4 target 0 lun 0 (pass1,ada1)
<ST2000DM001-9YN164 CC4H>          at scbus5 target 0 lun 0 (pass2,ada2)
<ST3000DM001-1CH166 CC49>          at scbus6 target 0 lun 0 (pass3,ada3)
<WDC WD30EZRX-00D8PB0 80.00A80>    at scbus8 target 0 lun 0 (pass4,ada4)
<hp v125w 1.00>                    at scbus11 target 0 lun 0 (pass5,da0)

Well it looks like I'm missing one drive... based on "camcontrol devlist" what do you suggest I do? Is it possible that by attaching the drives using a SATAIII to USB to a adapter that it modified some sectors in the drives?

Code:

ada0 at ahcich3 bus 0 scbus3 target 0 lun 0
ada0: <ST31500341AS CC1H> ATA-8 SATA 2.x device
ada0: Serial Number 9VS1S7B1
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1430798MB (2930275055 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad10
ada1 at ahcich4 bus 0 scbus4 target 0 lun 0
ada1: <ST2000DM001-9YN164 CC4H> ATA-8 SATA 3.x device
ada1: Serial Number Z1E3YEL4
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad12
ada2 at ahcich5 bus 0 scbus5 target 0 lun 0
ada2: <ST2000DM001-9YN164 CC4H> ATA-8 SATA 3.x device
ada2: Serial Number S1E0B385
ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad14
ada3 at ahcich6 bus 0 scbus6 target 0 lun 0
ada3: <ST3000DM001-1CH166 CC49> ATA-9 SATA 3.x device
ada3: Serial Number Z1F4D193
ada3: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad16
ada4 at ahcich8 bus 0 scbus8 target 0 lun 0
ada4: <WDC WD30EZRX-00D8PB0 80.00A80> ATA-9 SATA 3.x device
ada4: Serial Number WD-WMC4N0909180
ada4: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4: quirks=0x1<4K>
ada4: Previously was known as ad20

Based on this "dmesg" it looks like I have 5 drives so I'm missing one.

 

[mjws00]

Even with modified sectors it still needs to show up to the OS. At this point you have 4 drives from your mirrored pool. Plus the single drive pool. The bad part is both missing drives are from the same vdev, dead vdev means loss of the pool. So you need to work on getting 5 of the 6 drives back, or 4 of 6 if the missing drives are from different mirrored sets.

 

[ghostrunner]

Any suggestions on how I can get at least one of the drives recognized?

Thxs,

 

[mjws00]

There isn't a whole lot you can do. I'd likely disconnect all the drives except the first of the two I am trying to test. Should be easy to see if they are recognized. I'd also try plugging them in one of the working ports as well as that working cable. Not many pieces you can play with. The bios needs to see them, then FreeNAS, then we can worry about the pool.

If you can't get the drives to read on that system. Put the entire pool in a different box, run a clean install of the same version and auto-import the pool.

Sorry, no real magic to be done. Just isolate components and swap things to determine what is going on. Bottom line is you NEED 1 drive from each vdev or the pool is gone. 3 drives is enough.

 

[ghostrunner]

Well looks like the two drives which I thought were bad are now showing up... I discounted all the other drives that were attached to that pool and connected them to different ports.

I'm going to check the SATA port keeping the same cables on the same drives and then check the cables.

 

[ghostrunner]

I found that the culprit drive which I did not remove initially from the system. Turns out I removed the working hard drive and not the faulty one. However, the bad drive makes a clicking sounds and does not make itself recognizable to the system which is preventing me from mounting the volume.

I'm going to try and connect all the drives now excluding the faulty one and see if I can mount the volume. I don't expect this to work but it's worth a try.

Is there anyway I can force it to mount?

If I purchase a new 3TB HDD and connect it will I be able to recreate the volume with the new drive?

 

[ghostrunner]

So I know which physical drive is not being recognized by the system should I try running a Seagate diagnostic test to see if it can repair the drive?

 

[mjws00]

If the pool mounts with that drive out. I'd just grab a new one and resilver. If you have a non-critical use... relegate the seagate there. No way I'd trust a noisy and or failed drive again. But I am fussy and like my data intact.

 

[ghostrunner]

Well the volume will not mount I'm still getting the same error... this makes sense since the faulty drive is virtually dead.

Code:

[root@fs0b] ~# zpool import
   pool: v0
     id: 6859802526440201018
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
    devices and try again.
   see: http://illumos.org/msg/ZFS-8000-6X
config:

    v0                                              UNAVAIL  missing device
      mirror-1                                      ONLINE
        gptid/47a9fa47-71bc-11e3-9a43-50e549cd8552  ONLINE
        gptid/475baea4-53cd-11e3-bc1a-50e549cd8552  ONLINE
      mirror-2                                      DEGRADED
        6440307554711772873                         UNAVAIL  cannot open
        gptid/8524bf6a-62cc-11e4-bb8a-50e549cd8552  ONLINE

Is the only other options to recreate the pool and recover the data one drive at a time?

 

[mjws00]

There is no recovering the data without a data recovery service. The data is striped across the mirrors so ~1/3 of the blocks are missing. I'm still not sure what is up, you found the missing drive so you should have 5 of 6, no? Did the other drive in the first mirror not read either?

 

[ghostrunner]

All the drives are showing up on the View Disks tab minus the faulty one. I don't know why the other two disks are not showing up in pool.

 

[mjws00]

So at this point you should have that single 1.5tb drive out, just to simplify. You have narrowed down the exact drive that was failing and removed it. Might as well post the picture of the drive list and dmesg. Unfortunately it looks like you may have destroyed the zfs metadata, if so we are kinda dead in the water and things get expensive.

 

[ghostrunner]

Are you suggesting I detach the 1.5tb volume?

I can't afford paying $500+ for data recovery services... I do have a back up of the whole system in San Diego but the data is a year old but at least I can restore from that point.

 

[mjws00]

I'd only remove it because it seems a little misleading. Your pool that is degraded is only seeing 3 drives and none from the first vdev. I'd be pulling drives and labeling each and every vdev / gptid / serial and figuring out what is really going on. You have 3 drives missing from the pool currently.... why is that? If only one has failed? Is there other hardware at fault, have you hurt the meta-data. Odd ball tools run outside the system may well have left you in an unrecoverable state. Hard to say without going through all the motions.

Recovery from ZFS is not cheap, it can hit 5 figures easily. Might as well get the backup, or keep working on what is up with the drives. Sorry there's not better news.

 

[ghostrunner]

I pulled each and label all of them based on size and serial number... I'm not sure what you mean by vdev / gptid / serial.

 

[mjws00]

Your v0 pool should have 3 vdevs mirror-0, mirror-1, mirror-2. Each drive is listed by it's gptid i.e gptid/47a9fa47-71bc-11e3-9a43-50e549cd8552 (first disk in mirror-1). 'glabel status' will show you which disk has which id. 'smartctl -a /dev/ada0' for each device (0-5) will help you link serials to drives. The problem seem to be there are either more disks failed than you think. Or they have been damaged such that zfs can't tell they are part of a pool. Or there is a further hardware or cabling issue.