SOLVED ZFS GPT Table corrupt?

[silverbullettruck2001]

I am running FreeNAS-9.10-STABLE-201606270534 (dd17351). I haven't made any recent changes, but today FreeNAS started showing failures. It boots up fine since I boot from a separate drive, but I am getting errors when it is trying to load the ZFS disk from my RAID enclosure. I have the contents of the DMESG and GMULTIPATH STATUS results below. Specifically I see there are failures on da1, da2, and da3 with the GPT table. The RAID enclosure is not reporting any failed disks so it is puzzling why these are failing. I have researched this but a lot of the posts are suggesting destructive resolution, I would really like to know if there is anything I can do to resolve this that would allow me to get the data off of the drives. Any help would be appreciated. If there is any other information needed to assist, please let me know and I will gladly provide it.

Show : DMESG

Code:

Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-STABLE #0 455d13d(9.10-STABLE): Sun Jun 26 22:47:03 PDT 2016
    root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/trueos/sys/FreeNAS.amd64 amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
VT(vga): resolution 640x480
module_register: cannot register pci/xhci from kernel; already loaded from xhci.ko
Module pci/xhci failed to register: 17
CPU: Intel(R) Core(TM) i3-3120M CPU @ 2.50GHz (2494.39-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306a9  Family=0x6  Model=0x3a  Stepping=9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x3dbae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX,F16C>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 9116319744 (8694 MB)
avail memory = 7961489408 (7592 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3
random: <Software, Yarrow> initialized
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff80e73aa0, 0) error 19
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
padlock0: No ACE support.
acpi0: <ALASKA A M I> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0
agp0: <IvyBridge mobile GT2 IG> on vgapci0
agp0: aperture size is 256M, detected 262140k stolen memory
vgapci0: Boot video device
xhci0: <Intel Panther Point USB 3.0 controller> mem 0xf7d00000-0xf7d0ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
pci0: <simple comms> at device 22.0 (no driver attached)
ehci0: <Intel Panther Point USB 2.0 controller> mem 0xf7d18000-0xf7d183ff irq 16 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 16 at device 0.0 on pci1
re0: Using 1 MSI-X message
re0: turning off MSI enable bit.
re0: Chip rev. 0x2c800000
re0: MAC rev. 0x00100000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Using defaults for TSO: 65518/35/2048
re0: Ethernet address: 00:01:2e:4d:69:44
pcib2: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci2: <ACPI PCI bus> on pcib2
pci2: <network> at device 0.0 (no driver attached)
ehci1: <Intel Panther Point USB 2.0 controller> mem 0xf7d17000-0xf7d173ff irq 23 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Panther Point AHCI SATA controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xf7d16000-0xf7d167ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich4: <AHCI channel> at channel 4 on ahci0
acpi_button0: <Power Button> on acpi0
ichwd0: <Intel Panther Point watchdog timer> on isa0
wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at port 0x2e-0x2f on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0
uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
random: unblocking device.
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <0x8086> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
uhub0: 8 ports with 8 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen2.2: <vendor 0x8087> at usbus2
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2
ugen1.2: <vendor 0x8087> at usbus1
uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
ugen0.2: <Generic> at usbus0
umass0: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 1> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0xc100
umass0:2:0:-1: Attached to scbus2
uhub4: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
ugen1.3: <vendor 0x8087> at usbus1
ugen1.4: <vendor 0x0409> at usbus1
uhub5: <vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 4> on usbus1
uhub5: 4 ports with 4 removable, self powered
ugen1.5: <HP> at usbus1
ugen1.6: <Microsoft> at usbus1
ukbd0: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 6> on usbus1
kbd2 at ukbd0
ugen1.7: <Generic> at usbus1
ugen0.3: <JMicron> at usbus0
umass1: <JMicron USB to ATAATAPI Bridge, class 0/0, rev 3.00/28.08, addr 2> on usbus0
umass1:  SCSI over Bulk-Only; quirks = 0x4000
umass1:3:1:-1: Attached to scbus3
(probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
(probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error
(probe0:umass-sim1:1:0:0): SCSI status: Check Condition
(probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(probe0:umass-sim1:1:0:0): Error 22, Unretryable error
da0 at umass-sim0 bus 0 scbus2 target 0 lun 0
da0: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device
da0: Serial Number 000000000272
da0: 40.000MB/s transfers
da0: 30040MB (61521920 512 byte sectors)
da0: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE>
da1 at umass-sim1 bus 1 scbus3 target 0 lun 0
da1: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number 000000000000
da1: 400.000MB/s transfers
da1: 2861588MB (5860533168 512 byte sectors)
da1: quirks=0xa<NO_6_BYTE,4K>
da2 at umass-sim1 bus 1 scbus3 target 0 lun 1
da2: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number 000000000000
da2: 400.000MB/s transfers
da2: 2861588MB (5860533168 512 byte sectors)
da2: quirks=0xa<NO_6_BYTE,4K>
da3 at umass-sim1 bus 1 scbus3 target 0 lun 2
da3: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number 000000000000
da3: 400.000MB/s transfers
da3: 2861588MB (5860533168 512 byte sectors)
da3: quirks=0xa<NO_6_BYTE,4K>
da4 at umass-sim1 bus 1 scbus3 target 0 lun 3
da4: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number 000000000000
da4: 400.000MB/s transfers
da4: 2861588MB (5860533168 512 byte sectors)
da4: quirks=0xa<NO_6_BYTE,4K>
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1247196182 Hz quality 1000
GEOM: da1: the secondary GPT table is corrupt or invalid.
GEOM: da1: using the primary only -- recovery suggested.
GEOM: da2: the secondary GPT table is corrupt or invalid.
GEOM: da2: using the primary only -- recovery suggested.
GEOM: da3: the secondary GPT table is corrupt or invalid.
GEOM: da3: using the primary only -- recovery suggested.
Trying to mount root from zfs:freenas-boot/ROOT/9.10-STABLE-201606270534 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
GEOM_MULTIPATH: disk3 created
GEOM_MULTIPATH: da3 added to disk3
GEOM_MULTIPATH: da3 is now active path in disk3
GEOM_MULTIPATH: disk2 created
GEOM_MULTIPATH: da2 added to disk2
GEOM_MULTIPATH: da2 is now active path in disk2
GEOM_MULTIPATH: disk1 created
GEOM_MULTIPATH: da1 added to disk1
GEOM_MULTIPATH: da1 is now active path in disk1
GEOM: multipath/disk3: corrupt or invalid GPT detected.
GEOM: multipath/disk3: GPT rejected -- may not be recoverable.
GEOM: multipath/disk2: corrupt or invalid GPT detected.
GEOM: multipath/disk2: GPT rejected -- may not be recoverable.
GEOM: multipath/disk1: corrupt or invalid GPT detected.
GEOM: multipath/disk1: GPT rejected -- may not be recoverable.
re0: link state changed to UP
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
re0: link state changed to DOWN
re0: link state changed to UP
ums0: <HP HP USB Laser Mouse, class 0/0, rev 2.00/31.00, addr 5> on usbus1
ums0: 3 buttons and [XYZT] coordinates ID=0
uhid0: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 6> on usbus1
ums1: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 6> on usbus1
ums1: 5 buttons and [XYZ] coordinates ID=0
vboxdrv: fAsync=0 offMin=0x322 offMax=0xff4
pid 1406 (syslog-ng), uid 0: exited on signal 6 (core dumped)
(probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
(probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error
(probe0:umass-sim1:1:0:0): SCSI status: Check Condition
(probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(probe0:umass-sim1:1:0:0): Error 22, Unretryable error
ugen1.5: <HP> at usbus1 (disconnected)
ums0: at uhub5, port 2, addr 5 (disconnected)
(probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
(probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error
(probe0:umass-sim1:1:0:0): SCSI status: Check Condition
(probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
(probe0:umass-sim1:1:0:0): Error 22, Unretryable error

Show : gmultipath

Code:

[root@cruze] ~# gmultipath status
           Name    Status  Components
multipath/disk3  DEGRADED  da3 (ACTIVE)
multipath/disk2  DEGRADED  da2 (ACTIVE)
multipath/disk1  DEGRADED  da1 (ACTIVE)

 

[Mirfster]

trying to load the ZFS disk from my RAID enclosure.

Can you please elaborate on this a bit more? Are you talking about Hardware Raid?

Also, please post System Configuration Information.

 

[silverbullettruck2001]

Yes, not an elaborate/expensive hardware RAID, but it's still a hardware RAID enclosure with 4x 3TB WD Red drives. I don't use it for RAID though, it is configured in JBOD mode so it presents the OS w/ 4 drives and I use ZFS to perform the RAID.

Here is the System Config Info:

1. * FreeNAS version: FreeNAS-9.10-STABLE-201606270534 (dd17351)
2. * Hardware: Zotac ZBox ID83
3. * Motherboard (Model): Intel HM76 Express Chipset
4. * CPU (Model): Intel Core i3 3120M (dual core 2.5Ghz)
5. * RAM Size (in GB and model): 8GB
6. Hard Drives (Model), Quantity, and RAIDZ configuration: 4x 3TB Western Digital Red NAS drives. As for RAIDZ config, I don't remember the name of the level, but I have it configured in a RAID 0+1 (mirrored stripes). SANS Digital TR4UTBPN

I also attached the debug info from FreeNAS. Let me know if you need anything else.

 

[Mirfster]

Sorry, but I am still a bit confused here...
So it seems like you are running FreeNAS on is this?
If so, how is it attaching to the JBOD? Hopefully you are not going to say USB...
Can you provide details about the JBOD (like what Controller/HBA)? This is because if it has Hardware Raid in it, that does not necessarily mean that FreeNAS truly has direct access to the drives.

 

[silverbullettruck2001]

I am sorry to say...it is USB. I have been using it for over 3 years and never had any issues like this.

 

[Mirfster]

I am sorry to say...it is USB. I have been using it for over 3 years and never had any issues like this.

Oh dear... Not a good situation at all...

Sorry to say, but your are going to be hard pressed to get any real support. Not to be mean, but more than likely you may get some ridicule...

For references on why this is not encouraged, please see the section titled "Using USB external hard drives" in this thread: "How To Fail ... a guide to things not-to-do."

Please tell me that you maintained regular backups?

The only thing that comes to mind is:

1. If (and only if) that external JBOD is truly a JBOD
2. You have another system that can house those drives AND can install FreeNAS on that
   • Talking about a real system here; like at least a Dell T20 (which is like $180.00)...
   • Connecting the drives to SATA or a HBA... NO RAID Controllers...
3. Then maybe (just maybe) you can
   • Backup your current Configuration
     
   • Install a fresh copy of FreeNAS on the "real" machine
   • Connect all the drives
   • Import the Pool
   • Upload your Configuration

Otherwise, I honestly don't have much assistance to offer. Maybe one of the other contributors can assist though... /Fingers Crossed....

 

[silverbullettruck2001]

I guess my fingers are crossed as well. I promise if someone can help me get to the data, I will get a better storage device. I have backups, but the backup missed some files that I will lose without being able to recover. If there is anything you can provide so I can even attempt at recovering my files, I would appreciate it.

 

[silverbullettruck2001]

Mirfster,

If I did find a server to hook the hard drives into, would it matter what order the drives were connected to the SATA bus? Also, are there any precautions I need to take so I don't accidentally lose anything or make it worse?

 

[Mirfster]

If I did find a server to hook the hard drives into, would it matter what order the drives were connected to the SATA bus? Also, are there any precautions I need to take so I don't accidentally lose anything or make it worse?

As long as FreeNAS has direct access to the drives and they were not being served up as individual Raid0 (that is a zero at the end) drives then it does not matter.

Only concern is if those drives were being served to FreeNAS via Hardware Raid as individual Raid0 drives they are not going to want to work properly without the Raid Card.

Do you know exactly Card is in that JBOD unit and how it is configured?

 

[silverbullettruck2001]

Mirfster, I want to let you know that your solution worked! I scrounged up some hardware I had that would allow me to hook up the drives directly to 4 on-board SATA ports and I was delighted to see that the array showed up correctly in FreeNAS on the new server. The hardest part about the entire ordeal was getting the ISO to burn to a flash drive. The FreeNAS documentation says to use the Windows USB/DVD Download Tool, but the ISO that FreeNAS distributes won't work with that tool anymore. I had to use Win32 Disk Imager instead, which worked flawlessly.

Thank you VERY much for the suggestion. I am now in the process of backing up the array and purchasing replacement equipment that WILL NOT use a USB enclosure!

Thanks again!

 

[Mirfster]

Glad to hear things worked out. Seems you dodged a bullet on that one. :)

Once you are up and running on real hardware, make sure you have:

• Regularly scheduled SMART (Short and Long) Test
• Scrubs
• E-Mail notifications setup (for both "root" and SMART)
• Keep regularly scheduled backups (Data and Config)
• Check out the links in my sig under "Recommended Reading"; especially the last two to get things setup nicely

As for creating the media, I normally use IPMI to mount a Virtual CD and install from there (not sure if your next system will have that ability or not). Otherwise, I use Rufus to create a bootable USB to run the install from that.

Cheers to getting back your data! :D