Cannot Import a ZPOOL one disk Offline cannot be put Online

[Ricardo Cardoso]

I put a disk offline to be replaced and something gone wrong and now I can't put the offlined disk online again.
Sequence os events:

• Freenas 9.1.0 RC1 (updated one week before these events) Server ZFS raid-1 "RAIDZ" online and Healthy.
• Using the GUI set one disk offline for upgrade disk size
• shutdown
• changed the physical disk
• boot
• unable to import pool
• zpool status: no pool available
• zpool import:

Code:

  pool: RAIDZ
    id: 3634811780714734649
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
    devices and try again.
  see: http://www.sun.com/msg/ZFS-8000-3C
config:
 
    RAIDZ                                          UNAVAIL  insufficient replicas
      raidz1-0                                      UNAVAIL  insufficient replicas
        gptid/618319ef-915b-11e1-8fc4-386077d5d4c9  ONLINE
        13020385326369302278                        OFFLINE
        gptid/6287ea8f-915b-11e1-8fc4-386077d5d4c9  ONLINE
        9895012226997409189                        UNAVAIL  cannot open

• shutdown
• put back the off lined disk
• still unable to import, same error, same replies to zpool status and zpool import
• zpool online RAIDZ ada0: cannot open 'RAIDZ': no such pool

So, put a disk off line, replaced, other disk get UNAVAIL, put the offlined disk back but how to get it online if the pool cannot be imported?

Additional commands and info in the attached file.

Thanks for you all for any help!

 

[cyberjock]

Post the output of the following commands either as code or attached text file...

gpart status
gpart list
camcontrol devlist
dmesg

And please upgrade to 9.1-RELEASE if you haven't already.

 

[Ricardo Cardoso]

Code:

[root@freenas] /data/zfs# gpart status
  Name  Status  Components
da0s1      OK  da0
da0s2      OK  da0
da0s3      OK  da0
da0s4      OK  da0
da0s1a      OK  da0s1
ada0p1      OK  ada0
ada1p1      OK  ada1
ada1p2      OK  ada1
ada2p1      OK  ada2
ada2p2      OK  ada2
ada3p1      OK  ada3
ada3p2      OK  ada3

Code:

[root@freenas] /data/zfs# gpart list
Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7835647
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
  Mediasize: 1838301696 (1.7G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 32256
  Mode: r1w0e1
  attrib: active
  rawtype: 165
  length: 1838301696
  offset: 32256
  type: freebsd
  index: 1
  end: 3590495
  start: 63
2. Name: da0s2
  Mediasize: 1838301696 (1.7G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 1838366208
  Mode: r0w0e0
  rawtype: 165
  length: 1838301696
  offset: 1838366208
  type: freebsd
  index: 2
  end: 7180991
  start: 3590559
3. Name: da0s3
  Mediasize: 1548288 (1.5M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 3676667904
  Mode: r0w0e0
  rawtype: 165
  length: 1548288
  offset: 3676667904
  type: freebsd
  index: 3
  end: 7184015
  start: 7180992
4. Name: da0s4
  Mediasize: 21159936 (20M)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 3678216192
  Mode: r1w1e2
  rawtype: 165
  length: 21159936
  offset: 3678216192
  type: freebsd
  index: 4
  end: 7225343
  start: 7184016
Consumers:
1. Name: da0
  Mediasize: 4011851776 (3.8G)
  Sectorsize: 512
  Mode: r2w1e4
 
Geom name: da0s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 3590432
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s1a
  Mediasize: 1838293504 (1.7G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 40448
  Mode: r1w0e1
  rawtype: 0
  length: 1838293504
  offset: 8192
  type: !0
  index: 1
  end: 3590432
  start: 16
Consumers:
1. Name: da0s1
  Mediasize: 1838301696 (1.7G)
  Sectorsize: 512
  Stripesize: 0
  Stripeoffset: 32256
  Mode: r1w0e1
 
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
  Mediasize: 3000592896000 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 909c7a13-f62a-11e2-8b68-386077d5d4c9
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 3000592896000
  offset: 65536
  type: freebsd-zfs
  index: 1
  end: 5860533127
  start: 128
Consumers:
1. Name: ada0
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
 
Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 61ebcaed-915b-11e1-8fc4-386077d5d4c9
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada1p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 62090488-915b-11e1-8fc4-386077d5d4c9
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 1998251367936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 3907029134
  start: 4194432
Consumers:
1. Name: ada1
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 6269a8f9-915b-11e1-8fc4-386077d5d4c9
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada2p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 6287ea8f-915b-11e1-8fc4-386077d5d4c9
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 1998251367936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 3907029134
  start: 4194432
Consumers:
1. Name: ada2
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2
 
Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 3907029134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
  Mediasize: 2147483648 (2.0G)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e1
  rawuuid: 61681ddd-915b-11e1-8fc4-386077d5d4c9
  rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 2147483648
  offset: 65536
  type: freebsd-swap
  index: 1
  end: 4194431
  start: 128
2. Name: ada3p2
  Mediasize: 1998251367936 (1.8T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r0w0e0
  rawuuid: 618319ef-915b-11e1-8fc4-386077d5d4c9
  rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
  label: (null)
  length: 1998251367936
  offset: 2147549184
  type: freebsd-zfs
  index: 2
  end: 3907029134
  start: 4194432
Consumers:
1. Name: ada3
  Mediasize: 2000398934016 (1.8T)
  Sectorsize: 512
  Stripesize: 4096
  Stripeoffset: 0
  Mode: r1w1e2

Code:

[root@freenas] /data/zfs# dmesg
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.1-STABLE #0 r+7f710c8: Fri Jul 12 15:24:36 PDT 2013
    root@build.ixsystems.com:/tank/home/alfred/fn/9.1/os-base/amd64/tank/home/alfred/fn/9.1/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Atom(TM) CPU D525  @ 1.80GHz (1800.11-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x106ca  Family = 0x6  Model = 0x1c  Stepping = 10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 4081893376 (3892 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <INTEL  D525MW  >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP/HT): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP/HT): APIC ID:  3
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 8
ioapic0 <Version 2.0> irqs 0-23 on motherboard
lapic0: Forcing LINT1 to edge trigger
kbd1 at kbdmux0
aesni0: No AESNI support.
cryptosoft0: <software crypto> on motherboard
acpi0: <INTEL D525MW> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed03fff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_button0: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0x30c0-0x30c7 mem 0xf0300000-0xf037ffff,0xe0000000-0xefffffff,0xf0200000-0xf02fffff irq 16 at device 2.0 on pci0
agp0: <Intel Pineview SVGA controller> on vgapci0
agp0: aperture size is 256M, detected 8188k stolen memory
pcib1: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0x2000-0x20ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 38:60:77:d5:d4:c9
pcib2: <ACPI PCI-PCI bridge> at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
atapci0: <JMicron JMB362 SATA300 controller> port 0x1018-0x101f,0x1024-0x1027,0x1010-0x1017,0x1020-0x1023,0x1000-0x100f mem 0xf0100000-0xf01001ff irq 17 at device 0.0 on pci2
ahci0: <JMicron JMB362 AHCI SATA controller> at channel -1 on atapci0
ahci0: AHCI v1.10 with 2 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib3: <ACPI PCI-PCI bridge> at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0x3080-0x309f irq 23 at device 29.0 on pci0
usbus0 on uhci0
uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0x3060-0x307f irq 19 at device 29.1 on pci0
usbus1 on uhci1
uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0x3040-0x305f irq 18 at device 29.2 on pci0
usbus2 on uhci2
uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0x3020-0x303f irq 16 at device 29.3 on pci0
usbus3 on uhci3
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xf0380400-0xf03807ff irq 23 at device 29.7 on pci0
usbus4: EHCI version 1.0
usbus4 on ehci0
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci1: <Intel ICH7 AHCI SATA controller> port 0x30b8-0x30bf,0x30cc-0x30cf,0x30b0-0x30b7,0x30c8-0x30cb,0x30a0-0x30af mem 0xf0380000-0xf03803ff irq 18 at device 31.2 on pci0
ahci1: AHCI v1.10 with 4 3Gbps ports, Port Multiplier not supported
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe00022dd480 with IPv6 initialized (100409)
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ugen1.2: <vendor 0x04f3> at usbus1
ukbd0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.05, addr 2> on usbus1
kbd2 at ukbd0
uhid0: <vendor 0x04f3 product 0x0103, class 0/0, rev 1.10/1.05, addr 2> on usbus1
ugen4.2: <Kingston> at usbus4
umass0: <Kingston DT 101 G2, class 0/0, rev 2.00/1.00, addr 2> on usbus4
da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
da0: <Kingston DT 101 G2 PMAP> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 3826MB (7835648 512 byte sectors: 255H 63S/T 487C)
da0: quirks=0x2<NO_6_BYTE>
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <WDC WD30EZRX-00MMMB0 80.00A80> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad4
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada1: quirks=0x1<4K>
ada1: Previously was known as ad6
ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
ada2: <ST2000DL003-9VT166 CC3C> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada2: quirks=0x1<4K>
ada2: Previously was known as ad8
ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
ada3: <ST2000DL003-9VT166 CC3D> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
ada3: quirks=0x1<4K>
ada3: Previously was known as ad10
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic2: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
lapic3: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
Timecounter "TSC" frequency 1800105102 Hz quality 1000
Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
GEOM_RAID5: Module loaded, version 1.1.20110927.40 (rev 00ce00e5abb4)
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: software
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)

Code:

[root@freenas] /data/zfs# camcontrol devlist
<WDC WD30EZRX-00MMMB0 80.00A80>    at scbus0 target 0 lun 0 (ada0,pass0)
<ST2000DL003-9VT166 CC3D>          at scbus1 target 0 lun 0 (ada1,pass1)
<ST2000DL003-9VT166 CC3C>          at scbus2 target 0 lun 0 (ada2,pass2)
<ST2000DL003-9VT166 CC3D>          at scbus3 target 0 lun 0 (ada3,pass3)
<Kingston DT 101 G2 PMAP>          at scbus4 target 0 lun 0 (pass4,da0)

 

[cyberjock]

Hmm..

ada0 has no ZFS partition. It does have 2.7TB of disk spae that isn't allocated to any partition though.
ada1 has a ZFS partition, but isn't part of the pool.
ada2 and ada3 have ZFS partitions and they are valid.

Just in case you don't know what you are looking at, you have 2 disks that aren't usable with your RAIDZ1 vdev. Because of this you can't mount the zpool.

Can you post the output of:

smartctl -a -q noserial /dev/ada0
smartctl -a -q noserial /dev/ada1
smartctl -a -q noserial /dev/ada2
smartctl -a -q noserial /dev/ada3

 

[Ricardo Cardoso]

ada0 is the disk that went UNAVAIL after I put ada1 OFFLINE. I'm hopping to get ada1 online again backup and rebuild the pool.

Code:

[root@freenas] /data/zfs# smartctl -a -q noserial /dev/ada0
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Western Digital Caviar Green (AF, SATA 6Gb/s)
Device Model:    WDC WD30EZRX-00MMMB0
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Aug  2 17:45:43 2013 BRT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (49500) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (  2) minutes.
Extended self-test routine
recommended polling time:      ( 475) minutes.
Conveyance self-test routine
recommended polling time:      (  5) minutes.
SCT capabilities:            (0x3035)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  200  200  051    Pre-fail  Always      -      0
  3 Spin_Up_Time            0x0027  147  147  021    Pre-fail  Always      -      9608
  4 Start_Stop_Count        0x0032  100  100  000    Old_age  Always      -      125
  5 Reallocated_Sector_Ct  0x0033  200  200  140    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  100  253  000    Old_age  Always      -      0
  9 Power_On_Hours          0x0032  092  092  000    Old_age  Always      -      6045
10 Spin_Retry_Count        0x0032  100  100  000    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  100  100  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      121
192 Power-Off_Retract_Count 0x0032  200  200  000    Old_age  Always      -      86
193 Load_Cycle_Count        0x0032  166  166  000    Old_age  Always      -      104466
194 Temperature_Celsius    0x0022  120  111  000    Old_age  Always      -      32
196 Reallocated_Event_Count 0x0032  200  200  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  200  200  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0030  200  200  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0032  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x0008  200  200  000    Old_age  Offline      -      0
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline      Completed without error      00%      1190        -
# 2  Short offline      Completed without error      00%      1189        -
# 3  Short offline      Completed without error      00%      1188        -
# 4  Short offline      Completed without error      00%      1187        -
# 5  Short offline      Completed without error      00%      1186        -
# 6  Short offline      Completed without error      00%      1185        -
# 7  Short offline      Completed without error      00%      1184        -
# 8  Short offline      Completed without error      00%      1183        -
# 9  Short offline      Completed without error      00%      1182        -
#10  Short offline      Completed without error      00%      1181        -
#11  Short offline      Completed without error      00%      1180        -
#12  Short offline      Completed without error      00%      1179        -
#13  Short offline      Completed without error      00%      1178        -
#14  Short offline      Completed without error      00%      1177        -
#15  Short offline      Completed without error      00%      1176        -
#16  Short offline      Completed without error      00%      1175        -
#17  Short offline      Completed without error      00%      1174        -
#18  Short offline      Completed without error      00%      1173        -
#19  Short offline      Completed without error      00%      1172        -
#20  Short offline      Completed without error      00%      1171        -
#21  Short offline      Completed without error      00%      1170        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@freenas] ~# smartctl -a -q noserial /dev/ada1
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Seagate Barracuda Green (AF)
Device Model:    ST2000DL003-9VT166
Firmware Version: CC3D
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Aug  2 17:59:35 2013 BRT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (  633) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (  1) minutes.
Extended self-test routine
recommended polling time:      ( 338) minutes.
Conveyance self-test routine
recommended polling time:      (  2) minutes.
SCT capabilities:            (0x30b7)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x000f  117  099  006    Pre-fail  Always      -      128887208
  3 Spin_Up_Time            0x0003  093  093  000    Pre-fail  Always      -      0
  4 Start_Stop_Count        0x0032  100  100  020    Old_age  Always      -      152
  5 Reallocated_Sector_Ct  0x0033  100  100  036    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x000f  080  060  030    Pre-fail  Always      -      8811245534
  9 Power_On_Hours          0x0032  088  088  000    Old_age  Always      -      10593
10 Spin_Retry_Count        0x0013  100  100  097    Pre-fail  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  020    Old_age  Always      -      150
183 Runtime_Bad_Block      0x0032  100  100  000    Old_age  Always      -      0
184 End-to-End_Error        0x0032  100  100  099    Old_age  Always      -      0
187 Reported_Uncorrect      0x0032  100  100  000    Old_age  Always      -      0
188 Command_Timeout        0x0032  100  097  000    Old_age  Always      -      65546
189 High_Fly_Writes        0x003a  100  100  000    Old_age  Always      -      0
190 Airflow_Temperature_Cel 0x0022  068  053  045    Old_age  Always      -      32 (Min/Max 24/32)
191 G-Sense_Error_Rate      0x0032  100  100  000    Old_age  Always      -      0
192 Power-Off_Retract_Count 0x0032  100  100  000    Old_age  Always      -      104
193 Load_Cycle_Count        0x0032  100  100  000    Old_age  Always      -      151
194 Temperature_Celsius    0x0022  032  047  000    Old_age  Always      -      32 (0 22 0 0 0)
195 Hardware_ECC_Recovered  0x001a  021  003  000    Old_age  Always      -      128887208
197 Current_Pending_Sector  0x0012  100  100  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0010  100  100  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x003e  200  200  000    Old_age  Always      -      164
240 Head_Flying_Hours      0x0000  100  253  000    Old_age  Offline      -      66907000547683
241 Total_LBAs_Written      0x0000  100  253  000    Old_age  Offline      -      156582801788493
242 Total_LBAs_Read        0x0000  100  253  000    Old_age  Offline      -      167325624837594
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline      Completed without error      00%    10586        -
# 2  Short offline      Completed without error      00%      5747        -
# 3  Short offline      Completed without error      00%      5746        -
# 4  Short offline      Completed without error      00%      5745        -
# 5  Short offline      Completed without error      00%      5744        -
# 6  Short offline      Completed without error      00%      5743        -
# 7  Short offline      Completed without error      00%      5742        -
# 8  Short offline      Completed without error      00%      5741        -
# 9  Short offline      Completed without error      00%      5740        -
#10  Short offline      Completed without error      00%      5739        -
#11  Short offline      Completed without error      00%      5738        -
#12  Short offline      Completed without error      00%      5737        -
#13  Short offline      Completed without error      00%      5736        -
#14  Short offline      Completed without error      00%      5735        -
#15  Short offline      Completed without error      00%      5734        -
#16  Short offline      Completed without error      00%      5733        -
#17  Short offline      Completed without error      00%      5732        -
#18  Short offline      Completed without error      00%      5731        -
#19  Short offline      Completed without error      00%      5730        -
#20  Short offline      Completed without error      00%      5729        -
#21  Short offline      Completed without error      00%      5728        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@freenas] ~# smartctl -a -q noserial /dev/ada2
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Seagate Barracuda Green (AF)
Device Model:    ST2000DL003-9VT166
Firmware Version: CC3C
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Aug  2 18:00:25 2013 BRT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (  612) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (  1) minutes.
Extended self-test routine
recommended polling time:      ( 336) minutes.
Conveyance self-test routine
recommended polling time:      (  2) minutes.
SCT capabilities:            (0x30b7)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x000f  103  099  006    Pre-fail  Always      -      6201992
  3 Spin_Up_Time            0x0003  092  092  000    Pre-fail  Always      -      0
  4 Start_Stop_Count        0x0032  100  100  020    Old_age  Always      -      154
  5 Reallocated_Sector_Ct  0x0033  100  100  036    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x000f  080  060  030    Pre-fail  Always      -      8819121940
  9 Power_On_Hours          0x0032  088  088  000    Old_age  Always      -      10755
10 Spin_Retry_Count        0x0013  100  100  097    Pre-fail  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  020    Old_age  Always      -      154
183 Runtime_Bad_Block      0x0032  100  100  000    Old_age  Always      -      0
184 End-to-End_Error        0x0032  100  100  099    Old_age  Always      -      0
187 Reported_Uncorrect      0x0032  100  100  000    Old_age  Always      -      0
188 Command_Timeout        0x0032  100  097  000    Old_age  Always      -      7
189 High_Fly_Writes        0x003a  100  100  000    Old_age  Always      -      0
190 Airflow_Temperature_Cel 0x0022  068  052  045    Old_age  Always      -      32 (Min/Max 24/32)
191 G-Sense_Error_Rate      0x0032  100  100  000    Old_age  Always      -      0
192 Power-Off_Retract_Count 0x0032  100  100  000    Old_age  Always      -      105
193 Load_Cycle_Count        0x0032  100  100  000    Old_age  Always      -      154
194 Temperature_Celsius    0x0022  032  048  000    Old_age  Always      -      32 (0 22 0 0 0)
195 Hardware_ECC_Recovered  0x001a  007  006  000    Old_age  Always      -      6201992
197 Current_Pending_Sector  0x0012  100  100  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0010  100  100  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x003e  200  200  000    Old_age  Always      -      0
240 Head_Flying_Hours      0x0000  100  253  000    Old_age  Offline      -      58823872096773
241 Total_LBAs_Written      0x0000  100  253  000    Old_age  Offline      -      3045105125
242 Total_LBAs_Read        0x0000  100  253  000    Old_age  Offline      -      2373377357
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline      Completed without error      00%    10748        -
# 2  Short offline      Completed without error      00%      4985        -
# 3  Short offline      Completed without error      00%      4984        -
# 4  Short offline      Completed without error      00%      4983        -
# 5  Short offline      Completed without error      00%      4982        -
# 6  Short offline      Completed without error      00%      4981        -
# 7  Short offline      Completed without error      00%      4980        -
# 8  Short offline      Completed without error      00%      4979        -
# 9  Short offline      Completed without error      00%      4978        -
#10  Short offline      Completed without error      00%      4977        -
#11  Short offline      Completed without error      00%      4976        -
#12  Short offline      Completed without error      00%      4975        -
#13  Short offline      Completed without error      00%      4974        -
#14  Short offline      Completed without error      00%      4973        -
#15  Short offline      Completed without error      00%      4972        -
#16  Short offline      Completed without error      00%      4971        -
#17  Short offline      Completed without error      00%      4970        -
#18  Short offline      Completed without error      00%      4970        -
#19  Short offline      Completed without error      00%      4966        -
#20  Short offline      Completed without error      00%      4965        -
#21  Short offline      Completed without error      00%      4964        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

[root@freenas] ~# smartctl -a -q noserial /dev/ada3
smartctl 6.1 2013-03-16 r3800 [FreeBSD 9.1-STABLE amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    Seagate Barracuda Green (AF)
Device Model:    ST2000DL003-9VT166
Firmware Version: CC3D
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:    512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Fri Aug  2 18:00:52 2013 BRT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      (  0)    The previous self-test routine completed
                    without error or no self-test has ever
                    been run.
Total time to complete Offline
data collection:        (  623) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (  1) minutes.
Extended self-test routine
recommended polling time:      ( 346) minutes.
Conveyance self-test routine
recommended polling time:      (  2) minutes.
SCT capabilities:            (0x30b7)    SCT Status supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x000f  117  099  006    Pre-fail  Always      -      126369528
  3 Spin_Up_Time            0x0003  093  092  000    Pre-fail  Always      -      0
  4 Start_Stop_Count        0x0032  100  100  020    Old_age  Always      -      164
  5 Reallocated_Sector_Ct  0x0033  100  100  036    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x000f  076  060  030    Pre-fail  Always      -      17369808080
  9 Power_On_Hours          0x0032  088  088  000    Old_age  Always      -      10806
10 Spin_Retry_Count        0x0013  100  100  097    Pre-fail  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  020    Old_age  Always      -      162
183 Runtime_Bad_Block      0x0032  100  100  000    Old_age  Always      -      0
184 End-to-End_Error        0x0032  100  100  099    Old_age  Always      -      0
187 Reported_Uncorrect      0x0032  100  100  000    Old_age  Always      -      0
188 Command_Timeout        0x0032  100  098  000    Old_age  Always      -      4295098378
189 High_Fly_Writes        0x003a  100  100  000    Old_age  Always      -      0
190 Airflow_Temperature_Cel 0x0022  068  053  045    Old_age  Always      -      32 (Min/Max 24/32)
191 G-Sense_Error_Rate      0x0032  100  100  000    Old_age  Always      -      0
192 Power-Off_Retract_Count 0x0032  100  100  000    Old_age  Always      -      113
193 Load_Cycle_Count        0x0032  100  100  000    Old_age  Always      -      163
194 Temperature_Celsius    0x0022  032  047  000    Old_age  Always      -      32 (0 22 0 0 0)
195 Hardware_ECC_Recovered  0x001a  021  003  000    Old_age  Always      -      126369528
197 Current_Pending_Sector  0x0012  100  100  000    Old_age  Always      -      0
198 Offline_Uncorrectable  0x0010  100  100  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x003e  200  200  000    Old_age  Always      -      172
240 Head_Flying_Hours      0x0000  100  253  000    Old_age  Offline      -      144852067035704
241 Total_LBAs_Written      0x0000  100  253  000    Old_age  Offline      -      147511021467354
242 Total_LBAs_Read        0x0000  100  253  000    Old_age  Offline      -      123147207839266
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline      Completed without error      00%    10799        -
# 2  Short offline      Completed without error      00%      6372        -
# 3  Short offline      Completed without error      00%      6367        -
# 4  Short offline      Completed without error      00%      4369        -
# 5  Short offline      Completed without error      00%      4368        -
# 6  Short offline      Completed without error      00%      4367        -
# 7  Short offline      Completed without error      00%      4366        -
# 8  Short offline      Completed without error      00%      4365        -
# 9  Short offline      Completed without error      00%      4364        -
#10  Short offline      Completed without error      00%      4363        -
#11  Short offline      Completed without error      00%      4362        -
#12  Short offline      Completed without error      00%      4361        -
#13  Short offline      Completed without error      00%      4360        -
#14  Short offline      Completed without error      00%      4359        -
#15  Short offline      Completed without error      00%      4358        -
#16  Short offline      Completed without error      00%      4357        -
#17  Short offline      Completed without error      00%      4356        -
#18  Short offline      Completed without error      00%      4355        -
#19  Short offline      Completed without error      00%      4354        -
#20  Short offline      Completed without error      00%      4353        -
#21  Short offline      Completed without error      00%      4352        -
 
SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[cyberjock]

All disks appear to be in excellent health.

Did you upgrade your zpool to the new v5000 supported by FreeNAS 9.1?

post the output of zdb -l /dev/ada1p2

 

[Ricardo Cardoso]

No that I remember.

I can't issue a direct command in the freeNAS right now, but in the attached file you will find the result of zdb -l for all four devices.

 

[cyberjock]

Can you go back to 8.3.1 and do a zpool import?

Everything is looking like the UNAVAIL disk should be online...

 

[Ricardo Cardoso]

I already tried import with 8.1.3 and a FreeBSD 9.1 liveCD. Same results. But I'll try again as soon as I get access to it.
Seems to me that something is lost from ada0, zdb -l shows nothing in the labels and its gptid is different from reported from zdb -C. It appears to be fresh formatted, but I don't know how to inspect the partition to see if there is some data on.
What really puzzles me is the other disks are with the gptid correct, zdb -l shows the same information that zdb -C and the offline disk can't be put online again.
Where this information is stored? Can it be edited?

 

[cyberjock]

I'm sure it "can" be edited. But whether that's humanly possible easily is another question entirely. I sent a PM to someone to see what they think...

My hunch is that this is fixable. I'm just not sure how...

 

[Ricardo Cardoso]

I already tried import with 8.1.3 and a FreeBSD 9.1 liveCD. Same results. But I'll try again as soon as I get access to it.

FreeNAS 8.3.1 again. Same pen drive before the upgrade to 9.1: Just as before, same results.

 

[cyberjock]

Damn. So weird! I actually have no clue why it doesn't work.. LOL

 

[Ricardo Cardoso]

Just to get a little weirder: if I boot the system with only the offline disk connected and issue a zpool import it says the disk is online and the others are UNAVAILABLE or missing, obviously.
If I add any of the other working disks it goes offline.

If you don't have a clue...

 

[cyberjock]

Waiting for a PM response. He'll probably post in this thread.. so just sit tight until he posts... that is really weird!

 

[delphij]

Hi,

It looks like you have lose the underlying drive for 9895012226997409189 (I can't seem to find it in your zdb -l outputs). This suggests you might have offlined one disk and removed another, and therefore the pool lose two drives at the same time.

Could you please put the drive back and make 13020385326369302278 online, then do a scrub? After that you should be able to repeat the operation you wanted to do again.

 

[Ricardo Cardoso]

The pool lost two drives, one offline and the other appears to be removed, but it was not. It never leave its bay.

The pool was online and healthy, I put one drive offline to change for a bigger one.
Shutdown.
Swapped the offline drive.
Boot.
Boom! Two missing drives!
Shutdown.
Put back the offlined disk. Returning everything back as it was.
Boot.
The pool cannot be put online, one disk missing one disk offline. <- I am stuck here.

The drive that is missing never was pulled from the machine. But now it appears with a new gptid and zdb -l says all the labels are empty.
gpart says it have a FreeBSD-zfs partition.
I don't have a clue what happened with this drive. It appears to be formatted.
Maybe it got formatted when I issued zpool import after swapping the offline disk?
I don't know what happened and have no idea how to repair or recover the missing disk.
But I still have 3 disks, but one of them is offline. I was hopping to change its state to online and recover the missing disk, but zpool command cannot put the disk online because the pool cannot be put online just because this disk is offline.
It's like have locked the drawer with the key inside!
We can try to recover or repair the missing disk, but I don't know where to start.

 

[delphij]

The pool lost two drives, one offline and the other appears to be removed, but it was not. It never leave its bay.

The pool was online and healthy, I put one drive offline to change for a bigger one.
Shutdown.
Swapped the offline drive.
Boot.
Boom! Two missing drives!
Shutdown.
Put back the offlined disk. Returning everything back as it was.
Boot.
The pool cannot be put online, one disk missing one disk offline. <- I am stuck here.

But both gpart and gpt output suggests you still have the new drive in the pool.

The drive that is missing never was pulled from the machine. But now it appears with a new gptid and zdb -l says all the labels are empty.
gpart says it have a FreeBSD-zfs partition.
I don't have a clue what happened with this drive. It appears to be formatted.
Maybe it got formatted when I issued zpool import after swapping the offline disk?
I don't know what happened and have no idea how to repair or recover the missing disk.
But I still have 3 disks, but one of them is offline. I was hopping to change its state to online and recover the missing disk, but zpool command cannot put the disk online because the pool cannot be put online just because this disk is offline.
It's like have locked the drawer with the key inside!
We can try to recover or repair the missing disk, but I don't know where to start.

Please put all old drives back and start from there. Failing of doing that may cause further damage of your data.

 

[Ricardo Cardoso]

But both gpart and gpt output suggests you still have the new drive in the pool.

Please put all old drives back and start from there. Failing of doing that may cause further damage of your data.

Maybe I couldn't be clear enough, I ask your forgiveness, english is not my first language.

But it's not the new drive. It appears to be a new drive but it is the old one. All the disks on the system are the original ones.

I track them by serial numbers.

I have two disks with problems:

Serial: gptid (actual) guid Obs
WD-WCAWZ2048219 909c7a13-f62a-11e2-8b68-386077d5d4c9 This disk was gptid d526c082-2cd7-11e2-ac7a-386077d5d4c9, guid 9895012226997409189. Never leaved its bay until the issue appear.
6YD18NRA 62090488-915b-11e1-8fc4-386077d5d4c9 13020385326369302278 Offline - this disk was chosen for replace for a bigger one. If put alone in the system it says "online".

Edited the table formatting.

 

[cyberjock]

Can you just put all of your disks, old and new in the machine at the same time and do a "zpool import" and "gpart status" and "gpart list"?

 

[Ricardo Cardoso]

Now I think it is too late, I dd the missing disk (WCAWZ2048219) to the new drive before try something in its partition map. Actually its not finished yet.

Anyway, I did try that before, the new disk is WMC...:

Code:

[root@freenas] ~# smartctl -a /dev/ada0 | grep Serial
Serial Number:    WD-WCAWZ2048219
[root@freenas] ~# smartctl -a /dev/ada1 | grep Serial
Serial Number:    5YD6F4L1
[root@freenas] ~# smartctl -a /dev/ada2 | grep Serial
Serial Number:    WD-WMC1T3730955
[root@freenas] ~# smartctl -a /dev/ada3 | grep Serial
Serial Number:    6YD18NRA
[root@freenas] ~# smartctl -a /dev/ada4 | grep Serial
Serial Number:    6YD1923L
[root@freenas] ~# zpool online
missing pool name
usage:
    online [-e] <pool> <device> ...
[root@freenas] ~# zpool import
  pool: RAIDZ
    id: 3634811780714734649
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
    devices and try again.
  see: http://illumos.org/msg/ZFS-8000-3C
config:
 
    RAIDZ                                          UNAVAIL  insufficient replicas
      raidz1-0                                      UNAVAIL  insufficient replicas
        gptid/618319ef-915b-11e1-8fc4-386077d5d4c9  ONLINE
        13020385326369302278                        OFFLINE
        gptid/6287ea8f-915b-11e1-8fc4-386077d5d4c9  ONLINE
        9895012226997409189                        UNAVAIL  cannot open
[root@freenas] ~# zpool status -v
no pools available
[root@freenas] ~# gpart show
=>    63  7835585  da0  MBR  (3.8G)
      63  3590433    1  freebsd  [active]  (1.7G)
  3590496      63      - free -  (31k)
  3590559  3590433    2  freebsd  (1.7G)
  7180992    3024    3  freebsd  (1.5M)
  7184016    41328    4  freebsd  (20M)
  7225344  610304      - free -  (298M)
 
=>      0  3590433  da0s1  BSD  (1.7G)
        0      16        - free -  (8.0k)
      16  3590417      1  !0  (1.7G)
 
=>        34  5860533101  ada0  GPT  (2.7T)
          34          94        - free -  (47k)
        128  5860533000    1  freebsd-zfs  (2.7T)
  5860533128          7        - free -  (3.5k)
 
=>        34  3907029101  ada1  GPT  (1.8T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  3902834703    2  freebsd-zfs  (1.8T)
 
=>        34  3907029101  ada3  GPT  (1.8T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  3902834703    2  freebsd-zfs  (1.8T)
 
=>        34  3907029101  ada4  GPT  (1.8T)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  3902834703    2  freebsd-zfs  (1.8T)

The disk was not even formatted.

Maybe I put the wrong drive? Well, nobody is perfect, but all bays are labeled with each disk serial number.