Clean reboot, zpool not mounting

[thrca]

I wanted to ask for help before doing anything, as I really don't want to rebuild a bunch of data.

I needed to move my physical NAS, so I did a shutdown via the GUI. When the box turned off, I moved it to a new location, and fired it back up. Once it started, I found that my pool says "Error getting available space" and doesn't seem to be mounted.

From the SSH console, running "zpool status" reports no pools. Running "zpool import" shows that 2 of my disks cannot be opened.

Code:

[root@vault] /# zpool import
  pool: WDRED30-1
    id: 5071830854820780409
  state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing
        devices and try again.
  see: http://illumos.org/msg/ZFS-8000-3C
config:
 
        WDRED30-1                                      UNAVAIL  insufficient replicas
          raidz1-0                                      UNAVAIL  insufficient replicas
            15551499780479034600                        UNAVAIL  cannot open
            995324661768935554                          UNAVAIL  cannot open
            gptid/83c51276-a2b7-11e3-97a0-4061866430bd  ONLINE

Did I do something wrong or shut this thing down improperly?

 

[thrca]

I guess I should elaborate that this is a FreeNAS 9.2.1.2-RELEASE-x64 version which I installed fresh about 3 weeks ago. The pool was build fresh at the time of installation and has been working without any issue until this shutdown/restart.

 

[Rand]

Well if it's been moved ensure that all data and power cables are seated properly.
Check whether the disks show up in bios and check for any messages in dmesg.

 

[thrca]

dmesg: (It sees the disks...)

Code:

FreeBSD 9.2-RELEASE-p3 #0 r262572+825d682: Sat Mar  1 10:02:19 PST 2014
    root@build.ixsystems.com:/home/jkh/checkout/freenas/os-base/amd64/tank/home/jkh/checkout/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: AMD Athlon(tm) II X2 250 Processor (3000.07-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f62  Family = 0x10  Model = 0x6  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  TSC: P-state invariant
real memory  = 4831838208 (4608 MB)
avail memory = 3966414848 (3782 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <7612MS A7612200>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 1.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
padlock0: No ACE support.
acpi0: <7612MS A7612200> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <memory, RAM> at device 0.0 (no driver attached)
isab0: <PCI-ISA bridge> port 0x2f00-0x2fff at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
pci0: <memory, RAM> at device 1.2 (no driver attached)
pci0: <processor> at device 1.3 (no driver attached)
pci0: <memory, RAM> at device 1.4 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xfbf7f000-0xfbf7ffff irq 22 at device 2.0 on pci0
usbus0 on ohci0
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfbf7ec00-0xfbf7ecff irq 23 at device 2.1 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
ohci1: <OHCI (generic) USB controller> mem 0xfbf7d000-0xfbf7dfff irq 20 at device 4.0 on pci0
usbus2 on ohci1
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfbf7e800-0xfbf7e8ff irq 21 at device 4.1 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci1
pcib1: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci1: <ACPI PCI bus> on pcib1
nfe0: <NVIDIA nForce MCP77 Networking Adapter> port 0xc080-0xc087 mem 0xfbf7c000-0xfbf7cfff,0xfbf7e400-0xfbf7e4ff,0xfbf7e000-0xfbf7e00f irq 22 at device 10.0 on pci0
miibus0: <MII bus> on nfe0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
nfe0: Ethernet address: 40:61:86:64:30:bd
pcib2: <ACPI PCI-PCI bridge> at device 11.0 on pci0
pci2: <ACPI PCI bus> on pcib2
vgapci0: <VGA-compatible display> port 0xdc00-0xdc7f mem 0xfc000000-0xfcffffff,0xf0000000-0xf7ffffff,0xf8000000-0xf9ffffff irq 23 at device 0.0 on pci2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 16.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci3
pci4: <PCI bus> on pcib4
pcib5: <PCI-PCI bridge> at device 0.0 on pci4
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 2.0 on pci4
pci6: <PCI bus> on pcib6
pcib7: <PCI-PCI bridge> at device 3.0 on pci4
pci7: <PCI bus> on pcib7
arcmsr0: <Areca SAS 6G Host Adapter RAID Controller (RAID6 capable)
arcmsr version 1.20.00.28 2013-09-13
> port 0xe800-0xe8ff mem 0xfdff0000-0xfdffffff,0xfdf80000-0xfdfbffff irq 19 at device 0.0 on pci7
Areca RAID adapter0: ARC-1882 F/W version V1.52 2013-08-29
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 19.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> irq 18 at device 20.0 on pci0
pci9: <ACPI PCI bus> on pcib9
amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb3
acpi_button0: <Power Button> on acpi0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
wbwd0: HEFRAS and EFER do not align: EFER 0x4e DevID 0x07 DevRev 0x23 CR26 0x80
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe000278a000 with IPv6 initialized (100409)
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
load_dn_sched dn_sched QFQ loaded
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ugen0.1: <nVidia> at usbus0
uhub0: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <nVidia> at usbus1
uhub1: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <nVidia> at usbus2
uhub2: <nVidia OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <nVidia> at usbus3
uhub3: <nVidia EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
uhub0: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
da0 at arcmsr0 bus 0 scbus0 target 2 lun 0
da0: <WDC WD30EFRX-68AX9N0 R001> Fixed Direct Access SCSI-5 device
da0: Serial Number WD-WMC1T3166645
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da1 at arcmsr0 bus 0 scbus0 target 2 lun 1
da1: <WDC WD30EFRX-68AX9N0 R001> Fixed Direct Access SCSI-5 device
da1: Serial Number WD-WMC1T3155210
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
pass3 at arcmsr0 bus 0 scbus0 target 16 lun 0
pass3: <Areca RAID controller R001> Fixed Processor SCSI-0 device
da2 at arcmsr0 bus 0 scbus0 target 2 lun 2
da2: <WDC WD30EFRX-68AX9N0 R001> Fixed Direct Access SCSI-5 device
da2: Serial Number WD-WMC1T3958920
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
SMP: AP CPU #1 Launched!
Timecounter "TSC-low" frequency 1500033739 Hz quality 800
GEOM: da0: the secondary GPT table is corrupt or invalid.
GEOM: da0: using the primary only -- recovery suggested.
GEOM: da1: the secondary GPT table is corrupt or invalid.
GEOM: da1: using the primary only -- recovery suggested.
Root mount waiting for: usbus3 usbus1
Root mount waiting for: usbus3 usbus1
uhub1: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
Root mount waiting for: usbus1
ugen1.2: <OCZ Technology> at usbus1
umass0: <OCZ Technology RALLY2, class 0/0, rev 2.00/11.00, addr 2> on usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x0100
umass0:2:0:-1: Attached to scbus2
Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
mountroot: waiting for device /dev/ufs/FreeNASs1a ...
da3 at umass-sim0 bus 0 scbus2 target 0 lun 0
da3: <OCZ RALLY2 1100> Removable Direct Access SCSI-0 device
da3: Serial Number AA04012700336500
da3: 40.000MB/s transfers
da3: 3824MB (7831552 512 byte sectors: 255H 63S/T 487C)
da3: quirks=0x2<NO_6_BYTE>
GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
GEOM_RAID5: RAID5: device created (stripesize=131072).
GEOM_RAID5: RAID5: da1(1): disk attached.
GEOM_RAID5: RAID5: da0(3): disk attached.
GEOM_RAID5: RAID5: activated (forced) (need about 76MiB kmem (max)).
GEOM: raid5/RAID5: the secondary GPT table is corrupt or invalid.
GEOM: raid5/RAID5: using the primary only -- recovery suggested.
GEOM_RAID5: RAID5: request failed: R 5.

 

[thrca]

The disks also show up as normal and not failed in my areca controller.

 

[cyberjock]

Did you virtualize FreeNAS at any point?

Is read and/or write cache enabled in your RAID controller's firmware settings?

Did you ever attempt to use UFS? Cause GEOM_RAID5 seems to think stuff is supposed to be there...

Can you also post your hardware specs?

 

[thrca]

FreeNAS is not virtualized, it is on a OCZ Rally2 4GB USB Thumbdrive, which I wrote do the thumb drive from the downloaded img file.
The controller is an Areca 1882ix-12 in JBOD mode.
Disk write cache mode is "Auto" (I assume this is enabled, by default). The individual disks are setup as PassThroughDisks (always like this when controller is in JBOD mode) and cache mode is Write Back.
Disk power management is disabled.
The system is an AMD Athlon II X2 250 CPU with 4GB RAM.

 

[thrca]

I did not attempt to use UFS at any point, but these disks were formerly in another system. I wiped each one before creating the RAIDz Pool.

 

[thrca]

On another note, I seem to recall that I had to issue "sysctl kern.geom.debugflags=0x10" when I first created the pool, as it wouldnt create properly until I did this. I'm wondering if I forced something that should have actually been fixed.

If I need to rebuild the data to make this not happen again, I will, but I would prefer to try and get this working again without blowing up my data.

 

[cyberjock]

FreeNAS is not virtualized, it is on a OCZ Rally2 4GB USB Thumbdrive, which I wrote do the thumb drive from the downloaded img file.

Good! Cause I'd stop right now and tell you to kiss the data goodbye. :P

The controller is an Areca 1882ix-12 in JBOD mode.
Disk write cache mode is "Auto" (I assume this is enabled, by default). The individual disks are setup as PassThroughDisks (always like this when controller is in JBOD mode) and cache mode is Write Back.

Ok, so you need to figure out what "auto" really means. If it's enabled, that's probably what killed your pool. All write caches should be disabled when you are talking about using ZFS. The "write hole" between ZFS' write and the RAID controller's cache can be a failure for ZFS. ZFS avoids the write hole with it's design, but you can reintroduce it if you don't use appropriate hardware and appropriate settings.

Disk power management is disabled.
The system is an AMD Athlon II X2 250 CPU with 4GB RAM.

And that's probably where you just failed FreeNAS. Read this paragraph from the manual and tell me what you did wrong...

If you don't have at least 8GB of RAM with ZFS or 2GB of RAM with UFS, you should consider getting more powerful hardware before using FreeNAS® to store your data. Otherwise, data loss may result. Plenty of users expect FreeNAS to function with less than these requirements, just at reduced performance. The bottom line is that these minimums are the minimums based on many users' feedback in the forums for FreeNAS® to work, regardless of performance. Do not ask for help on systems that do not meet these requirements in the forums or IRC. They will likely be ignored because of the abundance of information that FreeNAS may not behave properly with <8GB of RAM.

That's straight from the manual.

So... do you have backups?

 

[thrca]

Hrmm, the RTFM pretty much spells it out..

Yes, I have backups. I think I will go back to hardware raid on the controller and just use something more "simple" for the nfs server.

 

[cyberjock]

Hrmm, the RTFM pretty much spells it out..

Yes, I have backups. I think I will go back to hardware raid on the controller and just use something more "simple" for the nfs server.

When you say hardware RAID, you mean ditch FreeNAS and go back to whatever OS you normally use? ZFS + hardware RAID really shouldn't be put together.. it's disaster for many people.

 

[thrca]

Yes, I mean, back to hardware raid+UFS, or hw raid+ubuntu with nfs export, since I really need none of the other stuff.

 

[cyberjock]

I will say that UFS is about to be removed from FreeNAS.. so I wouldn't consider that as a good choice. If you are used to ubuntu that's probably what you should do.

 

[thrca]

Since I plan on restoring this pool anyways, is there anything I might try to make it import in a read-only mode, even temporarily, to save myself a ton of time? suggestions risky to the data are ok, since I assume it's gone. All the disks show up in GEOM, btw..

Code:

[root@vault] ~# geom disk list
Geom name: da0
Providers:
1. Name: da0
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Mode: r1w1e1
  descr: WDC WD30EFRX-68AX9N0
  lunid: 001b4d2013664605
  ident: WD-WMC1T3166645
  fwsectors: 63
  fwheads: 255
 
Geom name: da1
Providers:
1. Name: da1
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Mode: r1w1e1
  descr: WDC WD30EFRX-68AX9N0
  lunid: 001b4d2013551200
  ident: WD-WMC1T3155210
  fwsectors: 63
  fwheads: 255
 
Geom name: da2
Providers:
1. Name: da2
  Mediasize: 3000592982016 (2.7T)
  Sectorsize: 512
  Mode: r1w1e2
  descr: WDC WD30EFRX-68AX9N0
  lunid: 001b4d2093852900
  ident: WD-WMC1T3958920
  fwsectors: 63
  fwheads: 255
 
Geom name: da3
Providers:
1. Name: da3
  Mediasize: 4009754624 (3.8G)
  Sectorsize: 512
  Mode: r2w1e4
  descr: OCZ RALLY2
  ident: AA04012700336500
  fwsectors: 63
  fwheads: 255