lost zpool data

[macstrat]

REcently we had a power outage here and when the power came back on the zpool for my system would not load. I have searched the forums and havent found an answer, many clost, but non seem to be applicable to what im experiencing. I can still see the drives and all the hardware checks out. The issue that i am getting is posted in the image and the data output from the sql data is also posted below. Please help.

Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
The Regents of the University of California. All rights reserved.

FreeBSD 8.2-RELEASE-p1 (FREENAS.amd64) #4: Sat Apr 30 10:39:46 PDT 2011

FreeNAS nanobsd (c) 2009-2010, The FreeNAS Development Team
All rights reserved.
FreeNAS is under the modified BSD license.

For more information, documentation, help or support, go here:
http://freenas.org
Welcome to FreeNAS
freenas# gpart show
=> 63 490234689 ada0 MBR (234G)
63 953505 1 freebsd [active] (466M)
953568 63 - free - (32K)
953631 953505 2 freebsd (466M)
1907136 3024 3 freebsd (1.5M)
1910160 41328 4 freebsd (20M)
1951488 488283264 - free - (233G)

=> 0 953505 ada0s1 BSD (466M)
0 16 - free - (8.0K)
16 953489 1 !0 (466M)

freenas# sysctl kern.disks
kern.disks: ada0 cd0
freenas# dmesg | more
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-RELEASE-p1 #4: Sat Apr 30 10:39:46 PDT 2011
jpaetzel@servant.iXsystems.com:/usr/home/jpaetzel/freenas/obj.amd64/usr/home/jpaetzel/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU 5160 @ 3.00GHz (2992.52-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f6 Family = 6 Model = f Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 4090712064 (3901 MB)
ACPI APIC Table: <DELL B8K >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
netsmb_dev: loaded
cryptosoft0: <software crypto> on motherboard
acpi0: <DELL B8K > on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci2
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci7: <ACPI PCI bus> on pcib7
vgapci0: <VGA-compatible display> port 0xdc80-0xdcff mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff,0xdf000000-0xdfffffff irq 16 at device 0.0 on pci7
pcib8: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pcib9: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci10: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci11: <ACPI PCI bus> on pcib11
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x006002> mem 0xd7ef0000-0xd7efffff irq 16 at device 0.0 on pci11
bge0: CHIP ID 0x00006002; ASIC REV 0x06; CHIP REV 0x60; PCI-E
miibus0: <MII bus> on bge0
brgphy0: <BCM5752 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
bge0: Ethernet address: 00:1a:a0:a6:ba:3d
bge0: [ITHREAD]
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0xff80-0xff9f irq 21 at device 29.0 on pci0
uhci0: [ITHREAD]
usbus0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0xff60-0xff7f irq 22 at device 29.1 on pci0
uhci1: [ITHREAD]
usbus1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0xff40-0xff5f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
usbus2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port 0xff20-0xff3f irq 23 at device 29.3 on pci0
uhci3: [ITHREAD]
usbus3: <Intel 631XESB/632XESB/3100 USB controller USB-4> on uhci3
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xff980800-0xff980bff irq 21 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus4: EHCI version 1.0
usbus4: <Intel 63XXESB USB 2.0 controller> on ehci0
pcib12: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci12: <ACPI PCI bus> on pcib12
pci12: <multimedia, audio> at device 2.0 (no driver attached)
pci12: <serial bus, FireWire> at device 10.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf irq 16 at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
atapci1: <Intel 63XXESB2 SATA300 controller> port 0xfe00-0xfe07,0xfe10-0xfe13,0xfe20-0xfe27,0xfe30-0xfe33,0xfec0-0xfecf mem 0xff970000-0xff9703ff irq 20 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcdfff,0xce000-0xcffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 92a092a0600092a
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 92a092a0600092a
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
RTC BIOS diagnostic error 11<memory_size>
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub4: 8 ports with 8 removable, self powered
ugen3.2: <Silitek> at usbus3
uhub5: <Silitek Silitek USB Hub Keyboard, class 9/0, rev 1.10/1.00, addr 2> on usbus3
uhub5: 2 ports with 1 removable, bus powered
ugen3.3: <Silitek> at usbus3
ukbd0: <Silitek Silitek USB Keyboard, class 0/0, rev 1.10/1.00, addr 3> on usbus3
kbd2 at ukbd0
ada0 at ata2 bus 0 scbus1 target 0 lun 0
ada0: <WDC WD2500YD-01NVB1 10.02E01> ATA-7 SATA 2.x device
ada0: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada0: 239372MB (490234752 512 byte sectors: 16H 63S/T 16383C)
cd0 at ata0 bus 0 scbus0 target 0 lun 0
cd0: <PHILIPS DVD+-RW DVD8801 AD21> Removable CD-ROM SCSI-0 device
cd0: 33.300MB/s transfers (UDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 4
ZFS storage pool version 15
fuse4bsd: version 0.3.9-pre1, FUSE ABI 7.8
freenas# sqlite3 /data/freenas-v1.db "select * from storage_disk;"
1|da0|da0|Member of Media raidz|Auto|60|Disabled|Disabled|1||1
2|da1|da1|Member of Media raidz|Auto|60|Disabled|Disabled|1||1
3|da2|da2|Member of Media raidz|Auto|60|Disabled|Disabled|1||1
4|da3|da3|Member of Media raidz|Auto|60|Disabled|Disabled|1||1
freenas# sqlite3 /data/freenas-v1.db "select * from storage_diskgroup;"
1|Mediaraidz|raidz|1
freenas# sqlite3 /data/freenas-v1.db "select * from storage_volume"
1|Media|ZFS
freenas# sqlite3 /data/freenas-v1.db "select * from storage_mountpoint"
/mnt/Media|0|rw|1|1

 

[cyberjock]

Just a random idea, but the 8.3 nightly builds have ZFSv28. You may want to try installing the nightly build to a spare USB drive and try booting it and the importing to see if it appears. Alot of people have had issues in the past that were corrected by temporarily using 8.3. Just make sure you do NOT run the zpool upgrade command or you will not be able to go back to 8.2.

Other than that, hopefully some admins can provide some good advice. My guess is you may be recovering from backups. :(

You don't have an UPS installed on the server, do you?

Did you look at: http://forums.freenas.org/showthread.php?1764-Error-getting-used-space-(Error) ?

 

[paleoN]

I can still see the drives and all the hardware checks out

Actually, according to sysctl kern.disks and dmesg you do not see all the drives. You only see ada0, your FreeNAS install. You are missing da0, da1, da2, da3. Are they installed to an add-on card?

 

[macstrat]

to noobsauce80:

8.3 is downloading now and i will try that and unfortunately i dont have a ups as it is part of a pair of systems, both dell p690's @ 1000W, a UPS for them is a bit out of my price range at the moment.

to paleoN:

i did not notice that. thank you. I checked the BIOS and the drives aren't showing there either. The drives are using the onboard SATA (4x2tb). Tomorrow after work i will be going to get an addon card and see if that will help show the drives again. The main OS drive is a 250gb drive that came with the machine and is plugged into a different place on the motherboard. Currently i have extracted one of the drives and have some diagnostics running on the drive to see if the drives themselves failed. What would be the chances of all 4 drives failing at the same time as opposed to just the onboard controller?

Will post more after i try the new OS and the controller card

 

[macstrat]

I have just checked all the drives and they all show up in the mac disk utility, they are spinning up and show the partitions on the drive

 

[macstrat]

noobsauce80:

i will try the 8.3 system and see what happens. Also this one of a trio of dell p690 servers running at 1000W each. (they also double as a render farm) a UPS for that much power is a bit out of my price range right now.

paleoN:

after you mentioned that i checked the BIOS and they are not showing in the bios either. They are using the onboard controller. The OS drive seems to be fine and as mentioned above the drives them selves seem fine in a different machine. Tomorrow after work i will be picking up an add on card and see if that helps. I will post more info as it comes in.

 

[paleoN]

i will try the 8.3 system and see what happens.

This won't help if the BIOS can't even see them.

They are using the onboard controller.

Sounds like your motherboard is on the fritz.

 

[StephenFry]

Sounds like your motherboard is on the fritz.

Not unlikely; the power going out is one thing, but who knows what kind of dirty signal might have preceded the outage (or, the switching back on).

 

[cyberjock]

Not unlikely; the power going out is one thing, but who knows what kind of dirty signal might have preceded the outage (or, the switching back on).

That's why I always use an UPS for computers I don't want to have to repair/replace later from an outage. The number of computer parts requiring repair/replacement has been zero since I started using UPSes in all of my machines at home in 2008. I used to lose power every time a storm even came close to my town. There didn't have to be a drop of rain on the ground. A grimacing storm on the horizon was all it took for my power to start flickering and eventually go out.

I'm not sure why someone would spend lots of money building a server and then not "splurge" for a UPS. IMO they're as important as the motherboard. To me they're an insurance policy to prevent me having to spend time and (usually more money) not to mention downtime or having to recover from backups to fix something that broke later because of a storm.

 

[macstrat]

The issue seems to be resolved. got 2 controller cards and they were able to get the drives back online and functional.

I didnt spend that much on them. They are old surplus workstations i picked up for 150$ each with the memory riser card and 16gb per system. an offer to good to pass up. the most expensive piece of hardware in the systems in the hard drives which were 100$ each. so each system is only about 550$. a ups to handle the systems and the monitor would run at least 600$ per system. I do think that it will be the next piece of equipment that i pick up tho.

thanks for the help

 

[Joshua Parker Ruehlig]

maybe you need enterprise level stuff but I use $50 ups' from staples and they seem to do just fine.

 

[macstrat]

I definitely do. The servers double as a render farm. Each system is a 1000w beast. And my collection of computers keeps growing. Right now Ive hit my limit for power comsumption without getting my house rewired for a higher voltage

 

[Joshua Parker Ruehlig]

I definitely do. The servers double as a render farm. Each system is a 1000w beast. And my collection of computers keeps growing. Right now Ive hit my limit for power comsumption without getting my house rewired for a higher voltage

=O
Wish I could do something like that with bitcoin, if only I got more gpu's before bitcoin took off...