SOLVED CRITICAL: The volume MORPHIUS03 (ZFS) state is UNKNOWN:

[tstorzuk]

Umm,

A little help here please.

I am getting this error from one of my servers (email first, now when I log into the GUI in the Alert System too). Also, the drives on this pool don't show up when I view the disks. And lastly, when I view the volumes in the Used space is says 0 (Error) and in the Available it says Error getting available space and in the Status column is says UNKNOWN.

When I try to do a Zpool Clear, it tells me that there is no such pool.

I can't run smart tests or a scrub on the pool or the drives, as they aren't showing up anywhere.

From this information, I'm guessing that one of my M1015 cards might have crapped out?

Could someone confirm that this may be the problem, or have any other suggestions?

 

[gpsguy]

Just buy new hardware and restore your data from a backup.

Seriously, other that a mention about having one or more M1015 cards, we don't know anything about your hardware or the version of FreeNAS that you are running. See the forum rules in red at the top of the page and post the required information.

 

[tstorzuk]

See the forum rules in red at the top of the page and post the required information.

Ok, let's see what kind of help we get now;

Search before posting - Done. Could not find any posts that had the same error message or problem. Found similar posts, tried a bunch of suggestions but to no avail.

Proper location for new thread - The drives aren't visible, so yes; it's hardware. Done.

Title - the exact wording of the error message. Done.

Description of problem - thought I had covered that, but I guess not. I'll add pictures so that it's more understandable;

Here's the error message (Title of this thread).

Here's the volumes and what is says in each column.

Here are the disks, none of the ones for the the pool of concern are showing. Done.

Error information - original error message is already above.
dmesg;

Code:

ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ipmi0: IPMI device rev. 1, firmware rev. 1.86, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi1: <IPMI System Interface> on isa0
device_attach: ipmi1 attach returned 16
wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
GEOM_ELI: Device da8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da10p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da11p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da12p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da13p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da14p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da15p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da16p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da17p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da18p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da19p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da20p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da21p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da22p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da23p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.3-RELEASE-p13 #0 r281084+3bda974: Tue May 12 21:03:11 PDT 2015
    root@build3.ixsystems.com:/tank/home/jkh/build/FN/objs/os-base/amd64/tank/home/jkh/build/FN/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz (3300.10-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306a9  Family = 0x6  Model = 0x3a  Stepping = 9
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Standard Extended Features=0x281<GSFSBASE,SMEP,ENHMOVSB>
  TSC: P-state invariant, performance statistics
real memory  = 35165044736 (33536 MB)
avail memory = 33130385408 (31595 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3
cpu4 (AP): APIC ID:  4
cpu5 (AP): APIC ID:  5
cpu6 (AP): APIC ID:  6
cpu7 (AP): APIC ID:  7
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
ispfw: registered firmware <isp_2400_multi>
ispfw: registered firmware <isp_2500>
ispfw: registered firmware <isp_2500_multi>
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 67, 1 (4) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
mps0: <LSI SAS2008> port 0xe000-0xe0ff mem 0xdfc00000-0xdfc03fff,0xdfb80000-0xdfbbffff irq 17 at device 0.0 on pci2
mps0: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib3: <ACPI PCI-PCI bridge> irq 19 at device 6.0 on pci0
pci3: <ACPI PCI bus> on pcib3
mps1: <LSI SAS2008> port 0xd000-0xd0ff mem 0xdf600000-0xdf603fff,0xdf580000-0xdf5bffff irq 19 at device 0.0 on pci3
mps1: Firmware: 16.00.00.00, Driver: 16.00.00.00-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xf020-0xf03f mem 0xdfe00000-0xdfe1ffff,0xdfe25000-0xdfe25fff irq 20 at device 25.0 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: xx:xx:xx:xx:xx:xx
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xdfe24000-0xdfe243ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib4: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci5: <ACPI PCI bus> on pcib5
em1: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xc000-0xc01f mem 0xdfd00000-0xdfd1ffff,0xdfd20000-0xdfd23fff irq 16 at device 0.0 on pci5
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: xx:xx:xx:xx:xx:xx
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xdfe23000-0xdfe233ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci6: <ACPI PCI bus> on pcib6
vgapci0: <VGA-compatible display> mem 0xdd000000-0xddffffff,0xdf000000-0xdf003fff,0xde800000-0xdeffffff irq 23 at device 3.0 on pci6
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xdfe22000-0xdfe227ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
ichwd0 on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
uhub2: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
ugen0.3: <Winbond Electronics Corp> at usbus0
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
ugen0.4: <Sony> at usbus0
umass0: <Sony Storage Media, class 0/0, rev 2.00/1.00, addr 4> on usbus0
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:9:0:-1: Attached to scbus9
da7 at mps0 bus 0 scbus0 target 7 lun 0
da7: <ATA ST2000DM001-1CH1 CC26> Fixed Direct Access SCSI-6 device
da7: Serial Number             Z1E3V1M9
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da7: quirks=0x8<4K>
da16 at umass-sim0 bus 0 scbus9 target 0 lun 0
da16: <Sony Storage Media 0100> Removable Direct Access SCSI-4 device
da16: Serial Number 9B4001408050012610
da16: 40.000MB/s transfers
da16: 14904MB (30523392 512 byte sectors: 255H 63S/T 1899C)
da16: quirks=0x2<NO_6_BYTE>
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da0: Serial Number S2H7J90B813572
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da0: quirks=0x8<4K>
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da1: Serial Number S2H7J90B813571
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da1: quirks=0x8<4K>
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da2: Serial Number S2H7J90B813577
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da2: quirks=0x8<4K>
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da3: Serial Number S2H7J9EB308528
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da3: quirks=0x8<4K>
da4 at mps0 bus 0 scbus0 target 4 lun 0
da4: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da4: Serial Number S2H7J9EB308520
da4: 300.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da4: quirks=0x8<4K>
da5 at mps0 bus 0 scbus0 target 5 lun 0
da5: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da5: Serial Number S2H7J90B812887
da5: 300.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da5: quirks=0x8<4K>
da6 at mps0 bus 0 scbus0 target 6 lun 0
da6: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da6: Serial Number S2H7J90B812883
da6: 300.000MB/s transfers
da6: Command Queueing enabled
da6: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da6: quirks=0x8<4K>
da8 at mps1 bus 0 scbus1 target 0 lun 0
da8: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da8: Serial Number S2H7JD1B110189
da8: 300.000MB/s transfers
da8: Command Queueing enabled
da8: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da8: quirks=0x8<4K>
da9 at mps1 bus 0 scbus1 target 1 lun 0
da9: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da9: Serial Number S2H7J9EB308472
da9: 300.000MB/s transfers
da9: Command Queueing enabled
da9: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da9: quirks=0x8<4K>
da10 at mps1 bus 0 scbus1 target 2 lun 0
da10: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da10: Serial Number S2H7J9EB308471
da10: 300.000MB/s transfers
da10: Command Queueing enabled
da10: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da10: quirks=0x8<4K>
da11 at mps1 bus 0 scbus1 target 3 lun 0
da11: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da11: Serial Number S2H7J9EB308527
da11: 300.000MB/s transfers
da11: Command Queueing enabled
da11: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da11: quirks=0x8<4K>
da12 at mps1 bus 0 scbus1 target 4 lun 0
da12: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da12: Serial Number S2H7JD1B110188
da12: 300.000MB/s transfers
da12: Command Queueing enabled
da12: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da12: quirks=0x8<4K>
da13 at mps1 bus 0 scbus1 target 5 lun 0
da13: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da13: Serial Number S2H7JD1B110192
da13: 300.000MB/s transfers
da13: Command Queueing enabled
da13: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da13: quirks=0x8<4K>
da14 at mps1 bus 0 scbus1 target 6 lun 0
da14: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da14: Serial Number S2H7JD1B110191
da14: 300.000MB/s transfers
da14: Command Queueing enabled
da14: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da14: quirks=0x8<4K>
da15 at mps1 bus 0 scbus1 target 7 lun 0
da15: <ATA SAMSUNG HD204UI 0001> Fixed Direct Access SCSI-6 device
da15: Serial Number S2H7JD1B110190
da15: 300.000MB/s transfers
da15: Command Queueing enabled
da15: 1907729MB (3907029168 512 byte sectors: 255H 63S/T 243201C)
da15: quirks=0x8<4K>
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1650048432 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-9.3-STABLE-201505130355 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ipmi0: IPMI device rev. 1, firmware rev. 1.86, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi1: <IPMI System Interface> on isa0
device_attach: ipmi1 attach returned 16
wbwd0: DevID 0xc3 DevRev 0x33, will not attach, please report this.
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da10p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da11p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da12p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da13p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da14p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device da15p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
vboxdrv: fAsync=0 offMin=0x22e offMax=0xe5e

pciconf -lv;

Code:

hostb0@pci0:0:0:0:      class=0x060000 card=0x062415d9 chip=0x01588086 rev=0x09 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'Ivy Bridge DRAM Controller'
    class      = bridge
    subclass   = HOST-PCI
pcib1@pci0:0:1:0:       class=0x060400 card=0x062415d9 chip=0x01518086 rev=0x09 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Ivy Bridge PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:1:1:       class=0x060400 card=0x062415d9 chip=0x01558086 rev=0x09 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Ivy Bridge PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:6:0:       class=0x060400 card=0x062415d9 chip=0x015d8086 rev=0x09 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = 'Ivy Bridge PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
em0@pci0:0:25:0:        class=0x020000 card=0x150215d9 chip=0x15028086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82579LM Gigabit Network Connection'
    class      = network
    subclass   = ethernet
ehci0@pci0:0:26:0:      class=0x0c0320 card=0x062415d9 chip=0x1c2d8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
pcib4@pci0:0:28:0:      class=0x060400 card=0x062415d9 chip=0x1c108086 rev=0xb5 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:28:4:      class=0x060400 card=0x062415d9 chip=0x1c188086 rev=0xb5 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family PCI Express Root Port 5'
    class      = bridge
    subclass   = PCI-PCI
ehci1@pci0:0:29:0:      class=0x0c0320 card=0x062415d9 chip=0x1c268086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family USB Enhanced Host Controller'
    class      = serial bus
    subclass   = USB
pcib6@pci0:0:30:0:      class=0x060401 card=0x062415d9 chip=0x244e8086 rev=0xa5 hdr=0x01
    vendor     = 'Intel Corporation'
    device     = '82801 PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
isab0@pci0:0:31:0:      class=0x060100 card=0x062415d9 chip=0x1c548086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'C204 Chipset Family LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:      class=0x010601 card=0x062415d9 chip=0x1c028086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family 6 port SATA AHCI Controller'
    class      = mass storage
    subclass   = SATA
none0@pci0:0:31:3:      class=0x0c0500 card=0x062415d9 chip=0x1c228086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '6 Series/C200 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
mps0@pci0:2:0:0:        class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
    class      = mass storage
    subclass   = SAS
mps1@pci0:3:0:0:        class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    vendor     = 'LSI Logic / Symbios Logic'
    device     = 'SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]'
    class      = mass storage
    subclass   = SAS
em1@pci0:5:0:0: class=0x020000 card=0x000015d9 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
vgapci0@pci0:6:3:0:     class=0x030000 card=0x062415d9 chip=0x0532102b rev=0x0a hdr=0x00
    vendor     = 'Matrox Graphics, Inc.'
    device     = 'MGA G200eW WPCM450'
    class      = display
    subclass   = VGA

Debug;
Attached - and I hope that it's accessbile here, otherwise I'll have to post it in code brackets in another post.

Done.

System Configuration Information
FreeNAS version: FreeNAS-9.3-STABLE-201505130355
Hardware:
MOBO - Supermicro MBD-X9SCM-F-B
CPU - Intel 4 Core Xeon E-1230v2 3.3GHz 5.0GT/s 8MB LGA 1155
MEMORY - Kingston KVR16E11K4/32 DDR3-1600 32GB(4x8GB)/1Gx72 ECC CL11 Server Memory Kit
ZPOOL 3 CARD - IBM ServeRAID M1015 / LSI SAS9220-8i PCI-Express PCIe 2-port (4 drives per port via break-out cable) 6Gb/s SAS+SATA Controller (flashed to IT Mode)
ZPOOL 3 HDDs - (8) Western Digital 4TB Red NAS Hard Drive, SATA III w/ 64MB Cache (WD40EFRX) - set in ZFS RAIDZ2

 

[Bidule0hm]

Oh god, the other two pools are full to 96 %, you shouldn't fill the pools to more than 80 %. You don't have an alert in the GUI about that?

Regarding the original problem I only see two M1015 in the pciconf command output so it looks like the third one has failed. But I'll let a more experienced member about that to confirm or not ;)

 

[tstorzuk]

Yes, the other 2 pools were used to back up another server. I had just recreated the 3rd pool and was moving files off the other 2 to get them back down to a reasonable capacity. Yes, there is an alert in the GUI, but I had unchecked them as I was going to be moving files off them.

Yeah, I figure that the 3rd M1015 is toast. I've got extras, but it's a pain in the ass to get into the server to swap them out. I would appreciate to hear from more experienced members if they confirm the failure as well. It'll have to wait anyway, as this weekend I'm going out of town. But I will shutdown that server before I leave. Just in case.

Thanks for taking a look.

 

[Darren Myers]

Oh god, the other two pools are full to 96 %, you shouldn't fill the pools to more than 80 %. You don't have an alert in the GUI about that?

I was just thinking that too, it does indeed look like the third M1015 is toast, i would swap it and see if the pools dont come back.