FreeNAS 8.3.0: Unable to import volume, insufficient replicas

Status
Not open for further replies.

bodzki

Cadet
Joined
Jan 30, 2015
Messages
3
Hi,

Good day.

I have a FreeNAS box since 2012 built from desktop parts. It has 3 3.0 TB hard disks. A month ago two hard disks failed (not at the same time). I was able to replace both of them. Yesterday, the 3rd or the original hard disks failed and it said data corruption. Today when I checked, two hard drives have now failed. I am no longer able to import the volume because of insufficient replicas.

Will I still be able to replace the hard drives and have my data intact? How do I check if the hard drives are dead? Thanks.

Code:
[root@coloma-nas] ~# zpool status
no pools available
[root@coloma-nas] ~# zpool import
   pool: coloma1
     id: 2731042974407350178
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://www.sun.com/msg/ZFS-8000-3C
config:

        coloma1                                         UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            12341737926192160343                        UNAVAIL  cannot open
            17158621580375364828                        UNAVAIL  cannot open
            gptid/72d8e5fd-8abb-11e4-9595-8c89a583af35  ONLINE
[root@coloma-nas] ~#


Code:
[root@coloma-nas] ~# gpart status
  Name  Status  Components
ada0p1      OK  ada0
ada0p2      OK  ada0
da0s1      OK  da0
da0s2      OK  da0
da0s3      OK  da0
da0s4      OK  da0
da0s1a      OK  da0s1


Code:
[root@coloma-nas] ~# gpart list
Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada0p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e0
   rawuuid: 72c80e2a-8abb-11e4-9595-8c89a583af35
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada0p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: 72d8e5fd-8abb-11e4-9595-8c89a583af35
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada0
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e1

Geom name: da0
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 7821311
first: 63
entries: 4
scheme: MBR
Providers:
1. Name: da0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w0e1
   attrib: active
   rawtype: 165
   length: 988291584
   offset: 32256
   type: freebsd
   index: 1
   end: 1930319
   start: 63
2. Name: da0s2
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 988356096
   Mode: r0w0e0
   rawtype: 165
   length: 988291584
   offset: 988356096
   type: freebsd
   index: 2
   end: 3860639
   start: 1930383
3. Name: da0s3
   Mediasize: 1548288 (1.5M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1976647680
   Mode: r0w0e0
   rawtype: 165
   length: 1548288
   offset: 1976647680
   type: freebsd
   index: 3
   end: 3863663
   start: 3860640
4. Name: da0s4
   Mediasize: 21159936 (20M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 1978195968
   Mode: r1w1e2
   rawtype: 165
   length: 21159936
   offset: 1978195968
   type: freebsd
   index: 4
   end: 3904991
   start: 3863664
Consumers:
1. Name: da0
   Mediasize: 4004511744 (3.7G)
   Sectorsize: 512
   Mode: r2w1e4

Geom name: da0s1
modified: false
state: OK
fwheads: 255
fwsectors: 63
last: 1930256
first: 0
entries: 8
scheme: BSD
Providers:
1. Name: da0s1a
   Mediasize: 988283392 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 40448
   Mode: r1w0e1
   rawtype: 0
   length: 988283392
   offset: 8192
   type: !0
   index: 1
   end: 1930256
   start: 16
Consumers:
1. Name: da0s1
   Mediasize: 988291584 (942M)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 32256
   Mode: r1w0e1


Code:
[root@coloma-nas] ~# camcontrol devlist
<ST3000VN000-1H4167 SC43>          at scbus2 target 0 lun 0 (pass0,ada0)
<ATAPI DVD A  DH20A3S 9P56>        at scbus5 target 0 lun 0 (pass1,cd0)
<SanDisk Cruzer Switch 1.20>       at scbus6 target 0 lun 0 (pass2,da0)


Code:
[root@coloma-nas] ~# dmesg
Copyright (c) 1992-2012 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.3-RELEASE-p4 #0 r241984M: Wed Oct 24 00:57:10 PDT 2012
    root@build.ixsystems.com:/usr/home/jpaetzel/8.3.0-RELEASE/os-base/amd64/usr/home/jpaetzel/8.3.0-RELEASE/FreeBSD/src/sys/FREENAS.amd64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD A4-3300 APU with Radeon(tm) HD Graphics (2495.48-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x300f10  Family = 12  Model = 1  Stepping = 0
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  TSC: P-state invariant
real memory  = 17179869184 (16384 MB)
avail memory = 15994908672 (15253 MB)
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI Warning: Optional field Pm2ControlBlock has zero address or length: 0x0000000000000000/0x1 (20101013/tbfadt-655)
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
hpt27xx: RocketRAID 27xx controller driver v1.0 (Oct 24 2012 00:56:59)
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
acpi0: <ALASKA A M I> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
unknown: I/O range not supported
acpi0: reservation of 67, 9 (4) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf0ff mem 0xc0000000-0xcfffffff,0xfef00000-0xfef3ffff irq 18 at device 1.0 on pci0
pcib1: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
pci2: <ACPI PCI bus> on pcib2
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xd0004000-0xd0004fff,0xd0000000-0xd0003fff irq 16 at device 0.0 on pci2
re0: Using 1 MSI-X message
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 8c:89:a5:83:af:35
re0: [ITHREAD]
pci0: <serial bus, USB> at device 16.0 (no driver attached)
pci0: <serial bus, USB> at device 16.1 (no driver attached)
ahci0: <AHCI SATA controller> port 0xf140-0xf147,0xf130-0xf133,0xf120-0xf127,0xf110-0xf113,0xf100-0xf10f mem 0xfef4e000-0xfef4e7ff irq 19 at device 17.0 on pci0
ahci0: [ITHREAD]
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich1: [ITHREAD]
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich2: [ITHREAD]
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich3: [ITHREAD]
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich4: [ITHREAD]
ahcich5: <AHCI channel> at channel 5 on ahci0
ahcich5: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfef4d000-0xfef4dfff irq 18 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfef4c000-0xfef4c0ff irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <EHCI (generic) USB 2.0 controller> on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfef4b000-0xfef4bfff irq 18 at device 19.0 on pci0
ohci1: [ITHREAD]
usbus2: <OHCI (generic) USB controller> on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfef4a000-0xfef4a0ff irq 17 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
pci0: <multimedia, HDA> at device 20.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci3: <ACPI PCI bus> on pcib3
ohci2: <OHCI (generic) USB controller> mem 0xfef49000-0xfef49fff irq 18 at device 20.5 on pci0
ohci2: [ITHREAD]
usbus4: <OHCI (generic) USB controller> on ohci2
pci0: <base peripheral, SD host controller> at device 20.7 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
ppc1: <Parallel port> port 0x378-0x37f irq 5 drq 3 on acpi0
ppc1: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc1: [ITHREAD]
ppbus0: <Parallel port bus> on ppc1
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
acpi_throttle1: <ACPI CPU Throttling> on cpu1
acpi_throttle1: failed to attach P_CNT
device_attach: acpi_throttle1 attach returned 6
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
Timecounters tick every 1.000 msec
hpt27xx: no controller detected.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
ugen0.1: <AMD> at usbus0
uhub0: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <AMD> at usbus1
uhub1: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <AMD> at usbus2
uhub2: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <AMD> at usbus3
uhub3: <AMD EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <AMD> at usbus4
uhub4: <AMD OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ahcich1: Poll timeout on slot 0 port 15
ahcich1: is 04000000 cs 00000001 ss 00000000 rs 00000001 tfd 150 serr 00400000 cmd 0000e017
ahcich1: Poll timeout on slot 0 port 0
ahcich1: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 150 serr 00000000 cmd 0000e017
uhub4: 2 ports with 2 removable, self powered
uhub0: 5 ports with 5 removable, self powered
uhub2: 5 ports with 5 removable, self powered
uhub1: 5 ports with 5 removable, self powered
uhub3: 5 ports with 5 removable, self powered
ugen3.2: <SanDisk> at usbus3
umass0: <SanDisk Cruzer Switch, class 0/0, rev 2.00/1.03, addr 2> on usbus3
ugen2.2: <American Power Conversion> at usbus2
ahcich0: Timeout on slot 0 port 0
ahcich0: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 1d0 serr 00000000 cmd 0000e017
run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
ahcich0: Timeout on slot 0 port 0
ahcich0: is 00000000 cs 00000001 ss 00000000 rs 00000001 tfd 1d0 serr 00000000 cmd 0000e017
ada0 at ahcich2 bus 0 scbus2 target 0 lun 0
ada0: <ST3000VN000-1H4167 SC43> ATA-9 SATA 3.x device
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
da0: <SanDisk Cruzer Switch 1.20> Removable Direct Access SCSI-5 device
da0: 40.000MB/s transfers
da0: 3819MB (7821312 512 byte sectors: 255H 63S/T 486C)
cd0 at ahcich5 bus 0 scbus5 target 0 lun 0
cd0: <ATAPI DVD A  DH20A3S 9P56> Removable CD-ROM SCSI-0 device
cd0: 150.000MB/s transfers (SATA 1.x, UDMA5, ATAPI 12bytes, PIO 8192bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present - tray closed
GEOM: da0s1: geometry does not match label (16h,63s != 255h,63s).
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
ZFS filesystem version 5
ZFS storage pool version 28
bridge0: Ethernet address: 02:a7:28:5f:50:00
epair0a: Ethernet address: 02:88:27:00:09:0a
epair0b: Ethernet address: 02:88:27:00:0a:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: promiscuous mode enabled
re0: promiscuous
re0: link state changed to DOWN

mode enabled
re0: link state changed to UP
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Those 2 disks aren't even detected by the system. So unless you can get 2 of 3 of the disks that are supposed to be in the pool right now to work, your data is gone for good.

It is a bit odd that 2 disks would fail so quickly and suddenly. I bet you've got a power cable or sata cable not attached. ;)

You can't replace anything as your zpool is unavailable. Gotta get the pool online before you do anything else.
 

bodzki

Cadet
Joined
Jan 30, 2015
Messages
3
Thank you cyberjock for your reply. You are right. I was able to make the other hard drive online again but the old hard drive back in 2012 can no longer be detected despite of replacing the SATA cable. On an important note, I noticed this hard drive spins and stops 3 to 5 times before stopping for good when turn-on my machine.

Anyway with the two hard drives online, I'm now able to zpool import the drives. However the auto-import in the web UI still does not work. There are also errors detected in some files.

Code:
[root@coloma-nas] ~# zpool status -v
  pool: coloma1
state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
  scan: scrub repaired 384K in 10h43m with 0 errors on Sun Jan 25 10:43:52 2015
config:

        NAME                                            STATE     READ WRITE CKSUM
        coloma1                                         DEGRADED     0     0    46
          raidz1-0                                      DEGRADED     0     0   184
            12341737926192160343                        UNAVAIL      0     0     0  was /dev/gptid/6805ae65-2abd-11e2-b592-8c89a583af35
            gptid/c74b348c-8f7c-11e4-8c3b-8c89a583af35  ONLINE       0     0     0
            gptid/72d8e5fd-8abb-11e4-9595-8c89a583af35  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:


Should I proceed in replacing the "dead" hard drive? Or should I delete the files with permanent errors first? Thanks.
 

bodzki

Cadet
Joined
Jan 30, 2015
Messages
3
zpool status after first scrub.
Code:
[root@coloma-nas] ~# zpool status
  pool: coloma1
state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
  scan: scrub in progress since Sat Jan 31 23:08:09 2015
        10.4G scanned out of 4.59T at 81.9M/s, 16h17m to go
        0 repaired, 0.22% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        coloma1                                         DEGRADED     0     0  429K
          raidz1-0                                      DEGRADED     0     0 1.31M
            12341737926192160343                        UNAVAIL      0     0     0  was /dev/gptid/6805ae65-2abd-11e2-b592-8c89a583af35
            gptid/c74b348c-8f7c-11e4-8c3b-8c89a583af35  ONLINE       0     0 28.0K
            gptid/72d8e5fd-8abb-11e4-9595-8c89a583af35  ONLINE       0     0     0

errors: 268486 data errors, use '-v' for a list
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
yeah.. you are kind of screwed. 268k data errors. Do "zpool status -v" to see your list of corrupt files. It'll be pretty long, so don't be surprised. That's what you get for going with RAIDZ1....
 
Status
Not open for further replies.
Top