silverbullettruck2001
Dabbler
- Joined
- Jul 20, 2014
- Messages
- 22
I am running FreeNAS-9.10-STABLE-201606270534 (dd17351). I haven't made any recent changes, but today FreeNAS started showing failures. It boots up fine since I boot from a separate drive, but I am getting errors when it is trying to load the ZFS disk from my RAID enclosure. I have the contents of the DMESG and GMULTIPATH STATUS results below. Specifically I see there are failures on da1, da2, and da3 with the GPT table. The RAID enclosure is not reporting any failed disks so it is puzzling why these are failing. I have researched this but a lot of the posts are suggesting destructive resolution, I would really like to know if there is anything I can do to resolve this that would allow me to get the data off of the drives. Any help would be appreciated. If there is any other information needed to assist, please let me know and I will gladly provide it.
Code:
Copyright (c) 1992-2016 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.3-STABLE #0 455d13d(9.10-STABLE): Sun Jun 26 22:47:03 PDT 2016 root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/trueos/sys/FreeNAS.amd64 amd64 FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512 VT(vga): resolution 640x480 module_register: cannot register pci/xhci from kernel; already loaded from xhci.ko Module pci/xhci failed to register: 17 CPU: Intel(R) Core(TM) i3-3120M CPU @ 2.50GHz (2494.39-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306a9 Family=0x6 Model=0x3a Stepping=9 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x3dbae3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX,F16C> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x1<LAHF> Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS> XSAVE Features=0x1<XSAVEOPT> VT-x: (disabled in BIOS) PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 9116319744 (8694 MB) avail memory = 7961489408 (7592 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <ALASKA A M I> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 random: <Software, Yarrow> initialized WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. ioapic0 <Version 2.0> irqs 0-23 on motherboard kbd1 at kbdmux0 module_register_init: MOD_LOAD (vesa, 0xffffffff80e73aa0, 0) error 19 cryptosoft0: <software crypto> on motherboard aesni0: No AESNI support. padlock0: No ACE support. acpi0: <ALASKA A M I> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 cpu2: <ACPI CPU> on acpi0 cpu3: <ACPI CPU> on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 Event timer "HPET1" frequency 14318180 Hz quality 440 Event timer "HPET2" frequency 14318180 Hz quality 440 Event timer "HPET3" frequency 14318180 Hz quality 440 Event timer "HPET4" frequency 14318180 Hz quality 440 atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 agp0: <IvyBridge mobile GT2 IG> on vgapci0 agp0: aperture size is 256M, detected 262140k stolen memory vgapci0: Boot video device xhci0: <Intel Panther Point USB 3.0 controller> mem 0xf7d00000-0xf7d0ffff irq 16 at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA xhci0: Port routing mask set to 0xffffffff usbus0 on xhci0 pci0: <simple comms> at device 22.0 (no driver attached) ehci0: <Intel Panther Point USB 2.0 controller> mem 0xf7d18000-0xf7d183ff irq 16 at device 26.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 pci0: <multimedia, HDA> at device 27.0 (no driver attached) pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0 pci1: <ACPI PCI bus> on pcib1 re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 16 at device 0.0 on pci1 re0: Using 1 MSI-X message re0: turning off MSI enable bit. re0: Chip rev. 0x2c800000 re0: MAC rev. 0x00100000 miibus0: <MII bus> on re0 rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:01:2e:4d:69:44 pcib2: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0 pci2: <ACPI PCI bus> on pcib2 pci2: <network> at device 0.0 (no driver attached) ehci1: <Intel Panther Point USB 2.0 controller> mem 0xf7d17000-0xf7d173ff irq 23 at device 29.0 on pci0 usbus2: EHCI version 1.0 usbus2 on ehci1 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 ahci0: <Intel Panther Point AHCI SATA controller> port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xf7d16000-0xf7d167ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich4: <AHCI channel> at channel 4 on ahci0 acpi_button0: <Power Button> on acpi0 ichwd0: <Intel Panther Point watchdog timer> on isa0 wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at port 0x2e-0x2f on isa0 orm0: <ISA Option ROM> at iomem 0xc0000-0xcefff on isa0 uart0: <16550 or compatible> at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 coretemp0: <CPU On-Die Thermal Sensors> on cpu0 est0: <Enhanced SpeedStep Frequency Control> on cpu0 coretemp1: <CPU On-Die Thermal Sensors> on cpu1 est1: <Enhanced SpeedStep Frequency Control> on cpu1 coretemp2: <CPU On-Die Thermal Sensors> on cpu2 est2: <Enhanced SpeedStep Frequency Control> on cpu2 coretemp3: <CPU On-Die Thermal Sensors> on cpu3 est3: <Enhanced SpeedStep Frequency Control> on cpu3 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled random: unblocking device. usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 480Mbps High Speed USB v2.0 usbus2: 480Mbps High Speed USB v2.0 ugen0.1: <0x8086> at usbus0 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen1.1: <Intel> at usbus1 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 ugen2.1: <Intel> at usbus2 uhub2: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2 uhub0: 8 ports with 8 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen2.2: <vendor 0x8087> at usbus2 uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus2 ugen1.2: <vendor 0x8087> at usbus1 uhub4: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1 ugen0.2: <Generic> at usbus0 umass0: <Generic USB Storage, class 0/0, rev 2.00/2.72, addr 1> on usbus0 umass0: SCSI over Bulk-Only; quirks = 0xc100 umass0:2:0:-1: Attached to scbus2 uhub4: 6 ports with 6 removable, self powered uhub3: 6 ports with 6 removable, self powered ugen1.3: <vendor 0x8087> at usbus1 ugen1.4: <vendor 0x0409> at usbus1 uhub5: <vendor 0x0409 product 0x005a, class 9/0, rev 2.00/1.00, addr 4> on usbus1 uhub5: 4 ports with 4 removable, self powered ugen1.5: <HP> at usbus1 ugen1.6: <Microsoft> at usbus1 ukbd0: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 6> on usbus1 kbd2 at ukbd0 ugen1.7: <Generic> at usbus1 ugen0.3: <JMicron> at usbus0 umass1: <JMicron USB to ATAATAPI Bridge, class 0/0, rev 3.00/28.08, addr 2> on usbus0 umass1: SCSI over Bulk-Only; quirks = 0x4000 umass1:3:1:-1: Attached to scbus3 (probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 (probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error (probe0:umass-sim1:1:0:0): SCSI status: Check Condition (probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe0:umass-sim1:1:0:0): Error 22, Unretryable error da0 at umass-sim0 bus 0 scbus2 target 0 lun 0 da0: <Generic STORAGE DEVICE 0272> Removable Direct Access SCSI device da0: Serial Number 000000000272 da0: 40.000MB/s transfers da0: 30040MB (61521920 512 byte sectors) da0: quirks=0x3<NO_SYNC_CACHE,NO_6_BYTE> da1 at umass-sim1 bus 1 scbus3 target 0 lun 0 da1: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device da1: Serial Number 000000000000 da1: 400.000MB/s transfers da1: 2861588MB (5860533168 512 byte sectors) da1: quirks=0xa<NO_6_BYTE,4K> da2 at umass-sim1 bus 1 scbus3 target 0 lun 1 da2: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device da2: Serial Number 000000000000 da2: 400.000MB/s transfers da2: 2861588MB (5860533168 512 byte sectors) da2: quirks=0xa<NO_6_BYTE,4K> da3 at umass-sim1 bus 1 scbus3 target 0 lun 2 da3: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device da3: Serial Number 000000000000 da3: 400.000MB/s transfers da3: 2861588MB (5860533168 512 byte sectors) da3: quirks=0xa<NO_6_BYTE,4K> da4 at umass-sim1 bus 1 scbus3 target 0 lun 3 da4: <WDC WD30 EFRX-68AX9N0 0X08> Fixed Direct Access SPC-4 SCSI device da4: Serial Number 000000000000 da4: 400.000MB/s transfers da4: 2861588MB (5860533168 512 byte sectors) da4: quirks=0xa<NO_6_BYTE,4K> SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! Timecounter "TSC-low" frequency 1247196182 Hz quality 1000 GEOM: da1: the secondary GPT table is corrupt or invalid. GEOM: da1: using the primary only -- recovery suggested. GEOM: da2: the secondary GPT table is corrupt or invalid. GEOM: da2: using the primary only -- recovery suggested. GEOM: da3: the secondary GPT table is corrupt or invalid. GEOM: da3: using the primary only -- recovery suggested. Trying to mount root from zfs:freenas-boot/ROOT/9.10-STABLE-201606270534 []... GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7) GEOM_MULTIPATH: disk3 created GEOM_MULTIPATH: da3 added to disk3 GEOM_MULTIPATH: da3 is now active path in disk3 GEOM_MULTIPATH: disk2 created GEOM_MULTIPATH: da2 added to disk2 GEOM_MULTIPATH: da2 is now active path in disk2 GEOM_MULTIPATH: disk1 created GEOM_MULTIPATH: da1 added to disk1 GEOM_MULTIPATH: da1 is now active path in disk1 GEOM: multipath/disk3: corrupt or invalid GPT detected. GEOM: multipath/disk3: GPT rejected -- may not be recoverable. GEOM: multipath/disk2: corrupt or invalid GPT detected. GEOM: multipath/disk2: GPT rejected -- may not be recoverable. GEOM: multipath/disk1: corrupt or invalid GPT detected. GEOM: multipath/disk1: GPT rejected -- may not be recoverable. re0: link state changed to UP hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> re0: link state changed to DOWN re0: link state changed to UP ums0: <HP HP USB Laser Mouse, class 0/0, rev 2.00/31.00, addr 5> on usbus1 ums0: 3 buttons and [XYZT] coordinates ID=0 uhid0: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 6> on usbus1 ums1: <Microsoft Natural Ergonomic Keyboard 4000, class 0/0, rev 2.00/1.73, addr 6> on usbus1 ums1: 5 buttons and [XYZ] coordinates ID=0 vboxdrv: fAsync=0 offMin=0x322 offMax=0xff4 pid 1406 (syslog-ng), uid 0: exited on signal 6 (core dumped) (probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 (probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error (probe0:umass-sim1:1:0:0): SCSI status: Check Condition (probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe0:umass-sim1:1:0:0): Error 22, Unretryable error ugen1.5: <HP> at usbus1 (disconnected) ums0: at uhub5, port 2, addr 5 (disconnected) (probe0:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 (probe0:umass-sim1:1:0:0): CAM status: SCSI Status Error (probe0:umass-sim1:1:0:0): SCSI status: Check Condition (probe0:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code) (probe0:umass-sim1:1:0:0): Error 22, Unretryable error
Code:
[root@cruze] ~# gmultipath status Name Status Components multipath/disk3 DEGRADED da3 (ACTIVE) multipath/disk2 DEGRADED da2 (ACTIVE) multipath/disk1 DEGRADED da1 (ACTIVE)