Disk 'dropping out' can't see which one, error messages need more detail.

[diskdiddler]

Hello,

I've woken up to an email that one of my drives has dropped out, but unfortunately the email alert does not name the drive or port which did this.


You can see the critical alert does not state the name of the device which had the issue either.

The only indication I have of which disk it is, is one of my disks has "reset" it's settings.

I hate to criticize you guys because, a. I've never "paid" for the product and really like it and b. I never know if I'm doing something wrong?
However,..... to the layman this seems cut and dry, more information would be really helpful to an end user here, for diagnosing issues.



Can anyone please advise?
I've got a copy and paste of the console log (which isn't on by default either, I've turned it on) - however most users probably wouldn't have this.

Code:

Sep  2 00:18:53 freenas kernel: bge0: link state changed to DOWN
Sep  2 00:18:53 freenas kernel: bge0: link state changed to DOWN
Sep  2 00:18:59 freenas kernel: bge0: link state changed to UP
Sep  2 00:18:59 freenas kernel: bge0: link state changed to UP
Sep  2 00:19:10 freenas kernel: bge0: link state changed to DOWN
Sep  2 00:19:10 freenas kernel: bge0: link state changed to DOWN
Sep  2 00:19:40 freenas nmbd[2628]: [2015/09/02 00:19:40.787169,  0] ../source3/libsmb/nmblib.c:873(send_udp)
Sep  2 00:19:40 freenas nmbd[2628]:   Packet send failed to 192.168.0.255(138) ERRNO=No route to host
Sep  2 00:19:51 freenas kernel: bge0: link state changed to UP
Sep  2 00:19:51 freenas kernel: bge0: link state changed to UP
Sep  2 00:20:12 freenas dhclient: New IP Address (bge0): 192.168.0.9
Sep  2 00:20:12 freenas dhclient: New Subnet Mask (bge0): 255.255.255.0
Sep  2 00:20:12 freenas dhclient: New Broadcast Address (bge0): 192.168.0.255
Sep  2 00:20:12 freenas dhclient: New Routers (bge0): 192.168.0.254
Sep  2 01:43:03 freenas smbd[93521]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionshtpc (ipv4:192.168.0.15:56087) closed connection to service data
Sep  2 02:06:34 freenas ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
Sep  2 02:06:34 freenas ada5:  s/n 447EK3WUFS9A detached
Sep  2 02:06:34 freenas GEOM_ELI: Device ada5p1.eli destroyed.
Sep  2 02:06:34 freenas GEOM_ELI: Detached ada5p1.eli on last close.
Sep  2 02:06:34 freenas swap_pager: I/O error - pagein failed; blkno 2621777,size 4096, error 6
Sep  2 02:06:34 freenas vm_fault: pager read error, pid 1374 (devd)
Sep  2 02:06:34 freenas (ada5:ahcich5:0:0:0): Periph destroyed
Sep  2 02:06:34 freenas swap_pager: I/O error - pagein failed; blkno 2621777,size 4096, error 6
Sep  2 02:06:34 freenas vm_fault: pager read error, pid 1374 (devd)
Sep  2 02:06:34 freenas kernel: Failed to write core file for process devd (error 14)
Sep  2 02:06:34 freenas kernel: Failed to write core file for process devd (error 14)
Sep  2 02:06:34 freenas kernel: pid 1374 (devd), uid 0: exited on signal 11
Sep  2 02:06:37 freenas ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
Sep  2 02:06:37 freenas ada5:  ATA-8 SATA 3.x device
Sep  2 02:06:37 freenas ada5: Serial Number 447EK3WUFS9A
Sep  2 02:06:37 freenas ada5: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 02:06:37 freenas ada5: Command Queueing enabled
Sep  2 02:06:37 freenas ada5: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 02:06:37 freenas ada5: Previously was known as ad14
Sep  2 02:36:45 freenas smbd[87526]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionshtpc (ipv4:192.168.0.15:53993) closed connection to service data
Sep  2 02:40:00 freenas smbd[1862]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionshtpc (ipv4:192.168.0.15:57916) closed connection to service data
Sep  2 03:05:21 freenas syslog-ng[1753]: syslog-ng starting up; version='3.5.6'
Sep  2 03:05:21 freenas swap_pager: I/O error - pagein failed; blkno 2622012,size 4096, error 6
Sep  2 03:05:21 freenas panic: vm_page_free: freeing wired page 0xfffff8021be87da0
Sep  2 03:05:21 freenas cpuid = 1
Sep  2 03:05:21 freenas KDB: stack backtrace:
Sep  2 03:05:21 freenas db_trace_self_wrapper() at db_trace_self_wrapper+0x2a/frame 0xfffffe024f20a4d0
Sep  2 03:05:21 freenas kdb_backtrace() at kdb_backtrace+0x37/frame 0xfffffe024f20a590
Sep  2 03:05:21 freenas panic() at panic+0x1ce/frame 0xfffffe024f20a690
Sep  2 03:05:21 freenas vm_page_free_toq() at vm_page_free_toq+0x26e/frame 0xfffffe024f20a6b0
Sep  2 03:05:21 freenas tmpfs_nocacheread() at tmpfs_nocacheread+0x1ff/frame 0xfffffe024f20a700
Sep  2 03:05:21 freenas tmpfs_read() at tmpfs_read+0x35a/frame 0xfffffe024f20a7c0
Sep  2 03:05:21 freenas VOP_READ_APV() at VOP_READ_APV+0x6e/frame 0xfffffe024f20a7f0
Sep  2 03:05:21 freenas vn_read() at vn_read+0xed/frame 0xfffffe024f20a880
Sep  2 03:05:21 freenas vn_io_fault() at vn_io_fault+0x90/frame 0xfffffe024f20aa20
Sep  2 03:05:21 freenas dofileread() at dofileread+0xa1/frame 0xfffffe024f20aa70
Sep  2 03:05:21 freenas kern_preadv() at kern_preadv+0x68/frame 0xfffffe024f20aac0
Sep  2 03:05:21 freenas sys_pread() at sys_pread+0x68/frame 0xfffffe024f20ab10
Sep  2 03:05:21 freenas amd64_syscall() at amd64_syscall+0x5ea/frame 0xfffffe024f20ac30
Sep  2 03:05:21 freenas Xfast_syscall() at Xfast_syscall+0xf7/frame 0xfffffe024f20ac30
Sep  2 03:05:21 freenas --- syscall (475, FreeBSD ELF64, sys_pread), rip = 0x8025b3cdc, rsp = 0x7fffffffc138, rbp = 0x400 ---
Sep  2 03:05:21 freenas KDB: enter: panic
Sep  2 03:05:21 freenas Copyright (c) 1992-2014 The FreeBSD Project.
Sep  2 03:05:21 freenas Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Sep  2 03:05:21 freenas     The Regents of the University of California. All rights reserved.
Sep  2 03:05:21 freenas FreeBSD is a registered trademark of The FreeBSD Foundation.
Sep  2 03:05:21 freenas FreeBSD 9.3-RELEASE-p23 #2 r281084+e56b349: Fri Aug 21 21:59:07 PDT 2015
Sep  2 03:05:21 freenas root@build3.ixsystems.com:/tank/home/jkh/build/FN/objs/os-base/amd64/tank/home/jkh/build/FN/FreeBSD/src/sys/FREENAS.amd64 amd64
Sep  2 03:05:21 freenas gcc version 4.2.1 20070831 patched [FreeBSD]
Sep  2 03:05:21 freenas CPU: AMD Turion(tm) II Neo N54L Dual-Core Processor (2196.39-MHz K8-class CPU)
Sep  2 03:05:21 freenas Origin = "AuthenticAMD"  Id = 0x100f63  Family = 0x10  Model = 0x6  Stepping = 3
Sep  2 03:05:21 freenas Features=0x178bfbff
Sep  2 03:05:21 freenas Features2=0x802009
Sep  2 03:05:21 freenas AMD Features=0xee500800
Sep  2 03:05:21 freenas AMD Features2=0x837ff
Sep  2 03:05:21 freenas TSC: P-state invariant
Sep  2 03:05:21 freenas real memory  = 9126805504 (8704 MB)
Sep  2 03:05:21 freenas avail memory = 8204574720 (7824 MB)
Sep  2 03:05:21 freenas Event timer "LAPIC" quality 400
Sep  2 03:05:21 freenas ACPI APIC Table:
Sep  2 03:05:21 freenas FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
Sep  2 03:05:21 freenas FreeBSD/SMP: 1 package(s) x 2 core(s)
Sep  2 03:05:21 freenas cpu0 (BSP): APIC ID:  0
Sep  2 03:05:21 freenas cpu1 (AP): APIC ID:  1
Sep  2 03:05:21 freenas WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Sep  2 03:05:21 freenas ioapic0  irqs 0-23 on motherboard
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas ispfw: registered firmware
Sep  2 03:05:21 freenas kbd1 at kbdmux0
Sep  2 03:05:21 freenas cryptosoft0:  on motherboard
Sep  2 03:05:21 freenas aesni0: No AESNI support.
Sep  2 03:05:21 freenas padlock0: No ACE support.
Sep  2 03:05:21 freenas acpi0:  on motherboard
Sep  2 03:05:21 freenas acpi0: Power Button (fixed)
Sep  2 03:05:21 freenas acpi0: reservation of fee00000, 1000 (3) failed
Sep  2 03:05:21 freenas acpi0: reservation of ffb80000, 80000 (3) failed
Sep  2 03:05:21 freenas acpi0: reservation of fec10000, 20 (3) failed
Sep  2 03:05:21 freenas acpi0: reservation of fed80000, 1000 (3) failed
Sep  2 03:05:21 freenas acpi0: reservation of 0, a0000 (3) failed
Sep  2 03:05:21 freenas acpi0: reservation of 100000, ddf00000 (3) failed
Sep  2 03:05:21 freenas cpu0:  on acpi0
Sep  2 03:05:21 freenas cpu1:  on acpi0
Sep  2 03:05:21 freenas attimer0:  port 0x40-0x43 irq 0 on acpi0
Sep  2 03:05:21 freenas Timecounter "i8254" frequency 1193182 Hz quality 0
Sep  2 03:05:21 freenas Event timer "i8254" frequency 1193182 Hz quality 100
Sep  2 03:05:21 freenas atrtc0:  port 0x70-0x71 irq 8 on acpi0
Sep  2 03:05:21 freenas Event timer "RTC" frequency 32768 Hz quality 0
Sep  2 03:05:21 freenas hpet0:  iomem 0xfed00000-0xfed003ff on acpi0
Sep  2 03:05:21 freenas Timecounter "HPET" frequency 14318180 Hz quality 950
Sep  2 03:05:21 freenas Event timer "HPET" frequency 14318180 Hz quality 550
Sep  2 03:05:21 freenas Event timer "HPET1" frequency 14318180 Hz quality 450
Sep  2 03:05:21 freenas Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Sep  2 03:05:21 freenas acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
Sep  2 03:05:21 freenas pcib0:  port 0xcf8-0xcff on acpi0
Sep  2 03:05:21 freenas pci0:  on pcib0
Sep  2 03:05:21 freenas pcib1:  at device 1.0 on pci0
Sep  2 03:05:21 freenas pci1:  on pcib1
Sep  2 03:05:21 freenas vgapci0:  port 0xe000-0xe0ff mem 0xfc000000-0xfdffffff,0xfe8f0000-0xfe8fffff,0xfe700000-0xfe7fffff irq 18 at device 5.0 on pci1
Sep  2 03:05:21 freenas vgapci0: Boot video device
Sep  2 03:05:21 freenas pcib2:  irq 18 at device 6.0 on pci0
Sep  2 03:05:21 freenas pci2:  on pcib2
Sep  2 03:05:21 freenas bge0:  mem 0xfe9f0000-0xfe9fffff irq 18 at device 0.0 on pci2
Sep  2 03:05:21 freenas bge0: CHIP ID 0x05784100; ASIC REV 0x5784; CHIP REV 0x57841; PCI-E
Sep  2 03:05:21 freenas miibus0:  on bge0
Sep  2 03:05:21 freenas brgphy0:  PHY 1 on miibus0
Sep  2 03:05:21 freenas brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
Sep  2 03:05:21 freenas bge0: Ethernet address: 28:92:4a:2d:5a:ca
Sep  2 03:05:21 freenas ahci0:  port 0xd000-0xd007,0xc000-0xc003,0xb000-0xb007,0xa000-0xa003,0x9000-0x900f mem 0xfe6ffc00-0xfe6fffff irq 19 at device 17.0 on pci0
Sep  2 03:05:21 freenas ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier supported
Sep  2 03:05:21 freenas ahcich0:  at channel 0 on ahci0
Sep  2 03:05:22 freenas ahcich1:  at channel 1 on ahci0
Sep  2 03:05:22 freenas ahcich2:  at channel 2 on ahci0
Sep  2 03:05:22 freenas ahcich3:  at channel 3 on ahci0
Sep  2 03:05:22 freenas ahcich4:  at channel 4 on ahci0
Sep  2 03:05:22 freenas ahcich5:  at channel 5 on ahci0
Sep  2 03:05:22 freenas ohci0:  mem 0xfe6fe000-0xfe6fefff irq 18 at device 18.0 on pci0
Sep  2 03:05:22 freenas usbus0 on ohci0
Sep  2 03:05:22 freenas ehci0:  mem 0xfe6ff800-0xfe6ff8ff irq 17 at device 18.2 on pci0
Sep  2 03:05:22 freenas usbus1: EHCI version 1.0
Sep  2 03:05:22 freenas usbus1 on ehci0
Sep  2 03:05:22 freenas ohci1:  mem 0xfe6fd000-0xfe6fdfff irq 18 at device 19.0 on pci0
Sep  2 03:05:22 freenas usbus2 on ohci1
Sep  2 03:05:22 freenas ehci1:  mem 0xfe6ff400-0xfe6ff4ff irq 17 at device 19.2 on pci0
Sep  2 03:05:22 freenas usbus3: EHCI version 1.0
Sep  2 03:05:22 freenas usbus3 on ehci1
Sep  2 03:05:22 freenas isab0:  at device 20.3 on pci0
Sep  2 03:05:22 freenas isa0:  on isab0
Sep  2 03:05:22 freenas pcib3:  at device 20.4 on pci0
Sep  2 03:05:22 freenas pci3:  on pcib3
Sep  2 03:05:22 freenas ohci2:  mem 0xfe6fc000-0xfe6fcfff irq 18 at device 22.0 on pci0
Sep  2 03:05:22 freenas usbus4 on ohci2
Sep  2 03:05:22 freenas ehci2:  mem 0xfe6ff000-0xfe6ff0ff irq 17 at device 22.2 on pci0
Sep  2 03:05:22 freenas usbus5: EHCI version 1.0
Sep  2 03:05:22 freenas usbus5 on ehci2
Sep  2 03:05:22 freenas amdtemp0:  on hostb4
Sep  2 03:05:22 freenas acpi_button0:  on acpi0
Sep  2 03:05:22 freenas sc0:  at flags 0x100 on isa0
Sep  2 03:05:22 freenas sc0: VGA <16 virtual consoles, flags=0x300>
Sep  2 03:05:22 freenas vga0:  at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Sep  2 03:05:22 freenas wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
Sep  2 03:05:22 freenas hwpstate0:  on cpu0
Sep  2 03:05:22 freenas ZFS filesystem version: 5
Sep  2 03:05:22 freenas ZFS storage pool version: features support (5000)
Sep  2 03:05:22 freenas Timecounters tick every 1.000 msec
Sep  2 03:05:22 freenas ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
Sep  2 03:05:22 freenas usbus0: 12Mbps Full Speed USB v1.0
Sep  2 03:05:22 freenas usbus1: 480Mbps High Speed USB v2.0
Sep  2 03:05:22 freenas usbus2: 12Mbps Full Speed USB v1.0
Sep  2 03:05:22 freenas usbus3: 480Mbps High Speed USB v2.0
Sep  2 03:05:22 freenas usbus4: 12Mbps Full Speed USB v1.0
Sep  2 03:05:22 freenas usbus5: 480Mbps High Speed USB v2.0
Sep  2 03:05:22 freenas ugen0.1:  at usbus0
Sep  2 03:05:22 freenas uhub0:  on usbus0
Sep  2 03:05:22 freenas ugen1.1:  at usbus1
Sep  2 03:05:22 freenas uhub1:  on usbus1
Sep  2 03:05:22 freenas ugen2.1:  at usbus2
Sep  2 03:05:22 freenas uhub2:  on usbus2
Sep  2 03:05:22 freenas ugen3.1:  at usbus3
Sep  2 03:05:22 freenas uhub3:  on usbus3
Sep  2 03:05:22 freenas ugen4.1:  at usbus4
Sep  2 03:05:22 freenas uhub4:  on usbus4
Sep  2 03:05:22 freenas ugen5.1:  at usbus5
Sep  2 03:05:22 freenas uhub5:  on usbus5
Sep  2 03:05:22 freenas uhub4: 4 ports with 4 removable, self powered
Sep  2 03:05:22 freenas uhub0: 5 ports with 5 removable, self powered
Sep  2 03:05:22 freenas uhub2: 5 ports with 5 removable, self powered
Sep  2 03:05:22 freenas uhub5: 4 ports with 4 removable, self powered
Sep  2 03:05:22 freenas uhub1: 5 ports with 5 removable, self powered
Sep  2 03:05:22 freenas uhub3: 5 ports with 5 removable, self powered
Sep  2 03:05:22 freenas ugen1.2:  at usbus1
Sep  2 03:05:22 freenas umass0:  on usbus1
Sep  2 03:05:22 freenas umass0:  SCSI over Bulk-Only; quirks = 0xc100
Sep  2 03:05:22 freenas umass0:7:0:-1: Attached to scbus7
Sep  2 03:05:22 freenas ugen3.2:  at usbus3
Sep  2 03:05:22 freenas umass1:  on usbus3
Sep  2 03:05:22 freenas umass1:  SCSI over Bulk-Only; quirks = 0xc100
Sep  2 03:05:22 freenas umass1:8:1:-1: Attached to scbus8
Sep  2 03:05:22 freenas (probe0:umass-sim0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Sep  2 03:05:22 freenas (probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
Sep  2 03:05:22 freenas (probe0:umass-sim0:0:0:0): SCSI status: Check Condition
Sep  2 03:05:22 freenas (probe0:umass-sim0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
Sep  2 03:05:22 freenas (probe0:umass-sim0:0:0:0): Error 22, Unretryable error
Sep  2 03:05:22 freenas (probe1:umass-sim1:1:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00
Sep  2 03:05:22 freenas (probe1:umass-sim1:1:0:0): CAM status: SCSI Status Error
Sep  2 03:05:22 freenas (probe1:umass-sim1:1:0:0): SCSI status: Check Condition
Sep  2 03:05:22 freenas (probe1:umass-sim1:1:0:0): SCSI sense: ILLEGAL REQUEST asc:20,0 (Invalid command operation code)
Sep  2 03:05:22 freenas (probe1:umass-sim1:1:0:0): Error 22, Unretryable error
Sep  2 03:05:22 freenas ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
Sep  2 03:05:22 freenas ada0:  ATA-8 SATA 3.x device
Sep  2 03:05:22 freenas ada0: Serial Number 448DK2UQFS9A
Sep  2 03:05:22 freenas ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 03:05:22 freenas ada0: Command Queueing enabled
Sep  2 03:05:22 freenas ada0: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 03:05:22 freenas ada0: Previously was known as ad4
Sep  2 03:05:22 freenas ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
Sep  2 03:05:22 freenas ada1:  ATA-8 SATA 3.x device
Sep  2 03:05:22 freenas ada1: Serial Number 447EK3WTFS9A
Sep  2 03:05:22 freenas ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 03:05:22 freenas ada1: Command Queueing enabled
Sep  2 03:05:22 freenas ada1: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 03:05:22 freenas ada1: Previously was known as ad6
Sep  2 03:05:22 freenas ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
Sep  2 03:05:22 freenas ada2:  ATA-8 SATA 3.x device
Sep  2 03:05:22 freenas ada2: Serial Number 7492K3N4FS9A
Sep  2 03:05:22 freenas ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 03:05:22 freenas ada2: Command Queueing enabled
Sep  2 03:05:22 freenas ada2: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 03:05:22 freenas ada2: Previously was known as ad8
Sep  2 03:05:22 freenas ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
Sep  2 03:05:22 freenas ada3:  ATA-8 SATA 3.x device
Sep  2 03:05:22 freenas ada3: Serial Number 448HK2TGFS9A
Sep  2 03:05:22 freenas ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 03:05:22 freenas ada3: Command Queueing enabled
Sep  2 03:05:22 freenas ada3: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 03:05:22 freenas ada3: Previously was known as ad10
Sep  2 03:05:22 freenas ada4 at ahcich4 bus 0 scbus4 target 0 lun 0
Sep  2 03:05:22 freenas ada4:  ATA-8 SATA 3.x device
Sep  2 03:05:22 freenas ada4: Serial Number 5485K0LCFS9A
Sep  2 03:05:22 freenas ada4: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 03:05:22 freenas ada4: Command Queueing enabled
Sep  2 03:05:22 freenas ada4: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 03:05:22 freenas ada4: Previously was known as ad12
Sep  2 03:05:22 freenas ada5 at ahcich5 bus 0 scbus5 target 0 lun 0
Sep  2 03:05:22 freenas ada5:  ATA-8 SATA 3.x device
Sep  2 03:05:22 freenas ada5: Serial Number 447EK3WUFS9A
Sep  2 03:05:22 freenas ada5: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
Sep  2 03:05:22 freenas ada5: Command Queueing enabled
Sep  2 03:05:22 freenas ada5: 4769307MB (9767541168 512 byte sectors: 16H 63S/T 16383C)
Sep  2 03:05:22 freenas ada5: Previously was known as ad14
Sep  2 03:05:22 freenas SMP: AP CPU #1 Launched!
Sep  2 03:05:22 freenas Timecounter "TSC-low" frequency 1098197287 Hz quality 800
Sep  2 03:05:22 freenas da1 at umass-sim1 bus 1 scbus8 target 0 lun 0
Sep  2 03:05:22 freenas da1:  Removable Direct Access SCSI-5 device
Sep  2 03:05:22 freenas da1: Serial Number 09021000000000000000000107
Sep  2 03:05:22 freenas da1: 40.000MB/s transfers
Sep  2 03:05:22 freenas da1: 15288MB (31309824 512 byte sectors: 255H 63S/T 1948C)
Sep  2 03:05:22 freenas da1: quirks=0x2
Sep  2 03:05:22 freenas da0 at umass-sim0 bus 0 scbus7 target 0 lun 0
Sep  2 03:05:22 freenas da0:  Removable Direct Access SCSI-5 device
Sep  2 03:05:22 freenas da0: Serial Number 09021000000000000000001223
Sep  2 03:05:22 freenas da0: 40.000MB/s transfers
Sep  2 03:05:22 freenas da0: 14800MB (30310400 512 byte sectors: 255H 63S/T 1886C)
Sep  2 03:05:22 freenas da0: quirks=0x2
Sep  2 03:05:22 freenas Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-9.3-STABLE-201508250051 []...
Sep  2 03:05:22 freenas GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Sep  2 03:05:22 freenas ipmi0:  on isa0
Sep  2 03:05:22 freenas ipmi0: KCS mode found at mem 0x0 alignment 0x1 on isa
Sep  2 03:05:22 freenas ipmi0: couldn't configure I/O resource
Sep  2 03:05:22 freenas device_attach: ipmi0 attach returned 6
Sep  2 03:05:22 freenas wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
Sep  2 03:05:24 freenas GEOM_ELI: Device ada0p1.eli created.
Sep  2 03:05:24 freenas GEOM_ELI: Encryption: AES-XTS 256
Sep  2 03:05:24 freenas GEOM_ELI:     Crypto: software
Sep  2 03:05:24 freenas GEOM_ELI: Device ada1p1.eli created.
Sep  2 03:05:24 freenas GEOM_ELI: Encryption: AES-XTS 256
Sep  2 03:05:24 freenas GEOM_ELI:     Crypto: software
Sep  2 03:05:24 freenas GEOM_ELI: Device ada2p1.eli created.
Sep  2 03:05:24 freenas GEOM_ELI: Encryption: AES-XTS 256
Sep  2 03:05:24 freenas GEOM_ELI:     Crypto: software
Sep  2 03:05:24 freenas GEOM_ELI: Device ada3p1.eli created.
Sep  2 03:05:24 freenas GEOM_ELI: Encryption: AES-XTS 256
Sep  2 03:05:24 freenas GEOM_ELI:     Crypto: software
Sep  2 03:05:25 freenas GEOM_ELI: Device ada4p1.eli created.
Sep  2 03:05:25 freenas GEOM_ELI: Encryption: AES-XTS 256
Sep  2 03:05:25 freenas GEOM_ELI:     Crypto: software
Sep  2 03:05:25 freenas GEOM_ELI: Device ada5p1.eli created.
Sep  2 03:05:25 freenas GEOM_ELI: Encryption: AES-XTS 256
Sep  2 03:05:25 freenas GEOM_ELI:     Crypto: software
Sep  2 03:05:29 freenas root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Sep  2 03:05:29 freenas vboxdrv: fAsync=0 offMin=0x3d6 offMax=0xa3f
Sep  2 03:05:41 freenas ntpd[2358]: ntpd 4.2.4p5-a (1)
Sep  2 03:05:49 freenas ntpd[2359]: time reset -1.119126 s
Sep  2 03:05:50 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/net getlocalsid
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /sbin/sysctl -n 'kern.maxfilesperproc'
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: zfs list -H -o mountpoint
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: zfs list -H -o mountpoint
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: zfs list -H -o mountpoint
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/net getlocalsid
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/net setlocalsid S-1-5-21-2311848168-2374712123-2337797664
Sep  2 03:05:52 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/pdbedit -d 0 -i smbpasswd:/tmp/tmpzCfMma -s /usr/local/etc/smb4.conf -e tdbsam:/var/etc/private/passdb.tdb
Sep  2 03:05:54 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/smbpasswd -e 'root'
Sep  2 03:05:54 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/pdbedit -L
Sep  2 03:05:54 freenas generate_smb4_conf.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/net sam rights grant root SeTakeOwnershipPrivilege SeBackupPrivilege SeRestorePrivilege
Sep  2 03:05:55 freenas nmbd[2631]: [2015/09/02 03:05:55.166454,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Sep  2 03:05:55 freenas winbindd[2637]: [2015/09/02 03:05:55.395587,  0] ../source3/winbindd/winbindd_cache.c:3196(initialize_winbindd_cache)
Sep  2 03:05:55 freenas winbindd[2637]:   initialize_winbindd_cache: clearing cache and re-creating with version number 2
Sep  2 03:05:55 freenas winbindd[2637]: [2015/09/02 03:05:55.420539,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Sep  2 03:05:56 freenas smbd[2634]: [2015/09/02 03:05:56.271004,  0] ../lib/util/become_daemon.c:136(daemon_ready)
Sep  2 03:05:56 freenas smbd[2634]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Sep  2 03:05:57 freenas smbd[2634]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Sep  2 03:05:58 freenas smbd[2634]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Sep  2 03:05:59 freenas smbd[2634]: dnssd_clientstub ConnectToServer: connect() failed Socket:32 Err:-1 Errno:2 No such file or directory
Sep  2 03:06:18 freenas nmbd[2631]:   STATUS=daemon 'nmbd' finished starting up and ready to serve connections*****
Sep  2 03:06:18 freenas nmbd[2631]:  
Sep  2 03:06:18 freenas nmbd[2631]:   Samba name server FREENAS is now a local master browser for workgroup WORKGROUP on subnet 192.168.0.9
Sep  2 03:06:18 freenas nmbd[2631]:  
Sep  2 03:06:18 freenas nmbd[2631]:   *****
Sep  2 03:06:32 freenas bridge0: Ethernet address: 02:f5:d5:ae:e2:00
Sep  2 03:06:32 freenas kernel: bridge0: link state changed to UP
Sep  2 03:06:32 freenas kernel: bridge0: link state changed to UP
Sep  2 03:06:32 freenas kernel: bge0: promiscuous mode enabled
Sep  2 03:06:32 freenas epair0a: Ethernet address: 02:96:63:00:0b:0a
Sep  2 03:06:32 freenas epair0b: Ethernet address: 02:96:63:00:0c:0b
Sep  2 03:06:32 freenas kernel: epair0a: link state changed to UP
Sep  2 03:06:32 freenas kernel: epair0a: link state changed to UP
Sep  2 03:06:32 freenas kernel: epair0b: link state changed to UP
Sep  2 03:06:32 freenas kernel: epair0b: link state changed to UP
Sep  2 03:06:32 freenas kernel: epair0a: promiscuous mode enabled
Sep  2 03:06:32 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair0b
Sep  2 03:06:32 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair0b
Sep  2 03:07:01 freenas epair1a: Ethernet address: 02:b2:9e:00:0c:0a
Sep  2 03:07:01 freenas epair1b: Ethernet address: 02:b2:9e:00:0d:0b
Sep  2 03:07:01 freenas kernel: epair1a: link state changed to UP
Sep  2 03:07:01 freenas kernel: epair1a: link state changed to UP
Sep  2 03:07:01 freenas kernel: epair1b: link state changed to UP
Sep  2 03:07:01 freenas kernel: epair1b: link state changed to UP
Sep  2 03:07:01 freenas kernel: epair1a: promiscuous mode enabled
Sep  2 03:07:01 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair1b
Sep  2 03:07:01 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair1b
Sep  2 03:07:21 freenas epair2a: Ethernet address: 02:b3:71:00:0d:0a
Sep  2 03:07:21 freenas epair2b: Ethernet address: 02:b3:71:00:0e:0b
Sep  2 03:07:21 freenas kernel: epair2a: link state changed to UP
Sep  2 03:07:21 freenas kernel: epair2a: link state changed to UP
Sep  2 03:07:21 freenas kernel: epair2b: link state changed to UP
Sep  2 03:07:21 freenas kernel: epair2b: link state changed to UP
Sep  2 03:07:21 freenas kernel: epair2a: promiscuous mode enabled
Sep  2 03:07:21 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair2b
Sep  2 03:07:21 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair2b
Sep  2 03:07:39 freenas linux: pid 7291 (java): syscall inotify_init not implemented
Sep  2 03:07:41 freenas linux: pid 9272 (java): syscall inotify_init not implemented
Sep  2 03:07:53 freenas epair3a: Ethernet address: 02:78:54:00:0e:0a
Sep  2 03:07:53 freenas epair3b: Ethernet address: 02:78:54:00:0f:0b
Sep  2 03:07:53 freenas kernel: epair3a: link state changed to UP
Sep  2 03:07:53 freenas kernel: epair3a: link state changed to UP
Sep  2 03:07:53 freenas kernel: epair3b: link state changed to UP
Sep  2 03:07:53 freenas kernel: epair3b: link state changed to UP
Sep  2 03:07:53 freenas kernel: epair3a: promiscuous mode enabled
Sep  2 03:07:53 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair3b
Sep  2 03:07:53 freenas kernel: ng_ether_ifnet_arrival_event: can't re-name node epair3b
Sep  2 08:18:05 freenas manage.py: [common.pipesubr:71] Popen()ing: /usr/local/bin/ixdiagnose -d /var/db/system/ixdiagnose -s -F
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs list -H -o name '/mnt/ARRAY/jails/.warden-template-Pluginjail'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/sickbeard_1'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/htpc-manager_1'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/sabnzbd_1'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/qBittorrent_1'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/plex'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/qbittorent'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/test'
Sep  2 08:20:13 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/couchpotato_1'
Sep  2 08:20:14 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/crashplan_1'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs list -H -o name '/mnt/ARRAY/jails/.warden-template-Pluginjail'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/sickbeard_1'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/htpc-manager_1'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/sabnzbd_1'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/qBittorrent_1'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/plex'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/qbittorent'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/test'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/couchpotato_1'
Sep  2 08:20:23 freenas manage.py: [common.pipesubr:71] Popen()ing: /sbin/zfs get -H origin '/mnt/ARRAY/jails/crashplan_1'
Sep  2 08:20:39 freenas manage.py: [common.pipesubr:71] Popen()ing: /bin/rm -r -f /var/db/system/ixdiagnose
Sep  2 08:23:36 freenas smbd[39078]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionshomedesktop (ipv4:192.168.0.3:65406) closed connection to service media

Stop refresh

 

[diskdiddler]

I have also typed this into the console, again another command which only intermediate users and higher might know about?

It would be nice to be able to more easily identify the health of disks in a system and if they've been "trouble" before.
I am going to assume (?) that the 6'th and final disk I've outlined in the pic is the one with the fault, since it is the only one with a higher number there.
Also in the previous screenshot, you can see ADA5 (disk #6) was the one which had its settings reset. However the zpool status command doesn't reveal the ADA designation, which is also,... unfortunate and inconsistent

 

[DrKK]

First of all, no disk has dropped out. Your array is still online, just degraded. 97% chance that you will have to replace the one drive. As you have a RAID-Z2, this is not an emergency necessarily, but let's deal with it quickly.

Can you run "camcontrol devlist", which should tell you what the disk id's are (probably ada0, ada1, etc, or da0, da1, etc), and then give us the output of "smartctl -x /dev/ada5" (replacing ada5 with whichever one is the one throwing errors?

 

[diskdiddler]

First of all, no disk has dropped out. Your array is still online, just degraded. 97% chance that you will have to replace the one drive. As you have a RAID-Z2, this is not an emergency necessarily, but let's deal with it quickly.

Can you run "camcontrol devlist", which should tell you what the disk id's are (probably ada0, ada1, etc, or da0, da1, etc), and then give us the output of "smartctl -x /dev/ada5" (replacing ada5 with whichever one is the one throwing errors?

I noticed in the text log, it says
"Sep 2 02:06:34 freenas ada5: s/n 447EK3WUFS9A detached"
Plus the wording of the email
"The volume ARRAY (ZFS) state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state."
Perhaps "dropped out" isn't the right term but a disk has gone offline when it's not meant to?

Code:

[root@freenas] ~# camcontrol devlist
<TOSHIBA MD04ACA500 FP1A>          at scbus0 target 0 lun 0 (pass0,ada0)
<TOSHIBA MD04ACA500 FP1A>          at scbus1 target 0 lun 0 (pass1,ada1)
<TOSHIBA MD04ACA500 FP1A>          at scbus2 target 0 lun 0 (pass2,ada2)
<TOSHIBA MD04ACA500 FP1A>          at scbus3 target 0 lun 0 (pass3,ada3)
<TOSHIBA MD04ACA500 FP1A>          at scbus4 target 0 lun 0 (pass4,ada4)
<TOSHIBA MD04ACA500 FP1A>          at scbus5 target 0 lun 0 (pass5,ada5)
<General USB Flash Disk 1.00>      at scbus7 target 0 lun 0 (pass6,da0)
<General USB Flash Disk 1.00>      at scbus8 target 0 lun 0 (pass7,da1)
[root@freenas] ~#

Code:

[root@freenas] ~# smartctl -x /dev/ada5
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p23 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Toshiba 3.5" MD04ACA... Enterprise HDD
Device Model:     TOSHIBA MD04ACA500
Serial Number:    447EK3WUFS9A
LU WWN Device Id: 5 000039 57bd80a72
Firmware Version: FP1A
User Capacity:    5,000,981,078,016 bytes [5.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Sep  2 10:05:32 2015 AEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM feature is:   Disabled
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x85) Offline data collection activity
                                        was aborted by an interrupting command f                                                                                                                                                                                                                                              rom host.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  120) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off supp                                                                                                                                                                                                                                              ort.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 546) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   050    -    0
  2 Throughput_Performance  P-S---   100   100   050    -    0
  3 Spin_Up_Time            POS--K   100   100   001    -    580
  4 Start_Stop_Count        -O--CK   100   100   000    -    42
  5 Reallocated_Sector_Ct   PO--CK   100   100   050    -    0
  7 Seek_Error_Rate         PO-R--   100   100   050    -    0
  8 Seek_Time_Performance   P-S---   100   100   050    -    0
  9 Power_On_Hours          -O--CK   076   076   000    -    9779
10 Spin_Retry_Count        PO--CK   100   100   030    -    0
12 Power_Cycle_Count       -O--CK   100   100   000    -    42
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    564
192 Power-Off_Retract_Count -O--CK   100   100   000    -    17
193 Load_Cycle_Count        -O--CK   100   100   000    -    86
194 Temperature_Celsius     -O---K   100   100   000    -    42 (Min/Max 20/68)
196 Reallocated_Event_Count -O--CK   100   100   000    -    0
197 Current_Pending_Sector  -O--CK   100   100   000    -    0
198 Offline_Uncorrectable   ----CK   100   100   000    -    0
199 UDMA_CRC_Error_Count    -O--CK   200   253   000    -    0
220 Disk_Shift              -O----   100   100   000    -    0
222 Loaded_Hours            -O--CK   076   076   000    -    9776
223 Load_Retry_Count        -O--CK   100   100   000    -    0
224 Load_Friction           -O---K   100   100   000    -    0
226 Load-in_Time            -OS--K   100   100   000    -    207
240 Head_Flying_Hours       P-----   100   100   001    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O     51  Comprehensive SMART error log
0x03       GPL     R/O     64  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters
0x24       GPL     R/O  12288  Current Device Internal Status Data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa8       GPL     VS       8  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (64 sectors)
No Errors Logged

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA                                                                                                                                                                                                                                              _of_first_error
# 1  Short offline       Completed without error       00%      9778         -
# 2  Extended offline    Completed without error       00%      9418         -
# 3  Extended offline    Completed without error       00%      8675         -
# 4  Extended offline    Completed without error       00%      7963         -
# 5  Extended offline    Completed without error       00%      7954         -
# 6  Extended offline    Completed without error       00%      7925         -
# 7  Extended offline    Completed without error       00%      7908         -
# 8  Short offline       Completed without error       00%      7886         -
# 9  Short offline       Completed without error       00%      7838         -
#10  Short offline       Completed without error       00%      7790         -
#11  Short offline       Completed without error       00%      7742         -
#12  Short offline       Completed without error       00%      7694         -
#13  Short offline       Completed without error       00%      7646         -
#14  Short offline       Completed without error       00%      7598         -
#15  Short offline       Completed without error       00%      7550         -
#16  Short offline       Completed without error       00%      7526         -
#17  Short offline       Completed without error       00%      7478         -
#18  Short offline       Completed without error       00%      7430         -
#19  Short offline       Completed without error       00%      7383         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       1 (0x0001)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    42 Celsius
Power Cycle Min/Max Temperature:     42/49 Celsius
Lifetime    Min/Max Temperature:     20/68 Celsius
Under/Over Temperature Limit Count:   0/13265

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      5/55 Celsius
Min/Max Temperature Limit:            5/55 Celsius
Temperature History Size (Index):    478 (138)

Index    Estimated Time   Temperature Celsius
139    2015-09-02 02:08    47  ****************************
140    2015-09-02 02:09    48  *****************************
141    2015-09-02 02:10    48  *****************************
142    2015-09-02 02:11    47  ****************************
143    2015-09-02 02:12    48  *****************************
144    2015-09-02 02:13    47  ****************************
145    2015-09-02 02:14    48  *****************************
...    ..( 12 skipped).    ..  *****************************
158    2015-09-02 02:27    48  *****************************
159    2015-09-02 02:28    47  ****************************
160    2015-09-02 02:29    48  *****************************
...    ..(  4 skipped).    ..  *****************************
165    2015-09-02 02:34    48  *****************************
166    2015-09-02 02:35    47  ****************************
...    ..(  2 skipped).    ..  ****************************
169    2015-09-02 02:38    47  ****************************
170    2015-09-02 02:39    48  *****************************
171    2015-09-02 02:40    47  ****************************
...    ..(  4 skipped).    ..  ****************************
176    2015-09-02 02:45    47  ****************************
177    2015-09-02 02:46    48  *****************************
178    2015-09-02 02:47    47  ****************************
...    ..(  2 skipped).    ..  ****************************
181    2015-09-02 02:50    47  ****************************
182    2015-09-02 02:51    48  *****************************
183    2015-09-02 02:52     ?  -
184    2015-09-02 02:53    48  *****************************
185    2015-09-02 02:54    47  ****************************
186    2015-09-02 02:55    48  *****************************
187    2015-09-02 02:56    48  *****************************
188    2015-09-02 02:57    48  *****************************
189    2015-09-02 02:58    49  ******************************
190    2015-09-02 02:59    48  *****************************
191    2015-09-02 03:00    48  *****************************
192    2015-09-02 03:01    49  ******************************
193    2015-09-02 03:02    49  ******************************
194    2015-09-02 03:03    48  *****************************
...    ..( 11 skipped).    ..  *****************************
206    2015-09-02 03:15    48  *****************************
207    2015-09-02 03:16    47  ****************************
208    2015-09-02 03:17    48  *****************************
209    2015-09-02 03:18    47  ****************************
...    ..(  9 skipped).    ..  ****************************
219    2015-09-02 03:28    47  ****************************
220    2015-09-02 03:29    46  ***************************
221    2015-09-02 03:30    47  ****************************
...    ..(  9 skipped).    ..  ****************************
231    2015-09-02 03:40    47  ****************************
232    2015-09-02 03:41    46  ***************************
...    ..(  6 skipped).    ..  ***************************
239    2015-09-02 03:48    46  ***************************
240    2015-09-02 03:49    45  **************************
241    2015-09-02 03:50    46  ***************************
242    2015-09-02 03:51    45  **************************
...    ..( 15 skipped).    ..  **************************
258    2015-09-02 04:07    45  **************************
259    2015-09-02 04:08    44  *************************
...    ..( 28 skipped).    ..  *************************
288    2015-09-02 04:37    44  *************************
289    2015-09-02 04:38    43  ************************
290    2015-09-02 04:39    44  *************************
291    2015-09-02 04:40    43  ************************
292    2015-09-02 04:41    44  *************************
293    2015-09-02 04:42    44  *************************
294    2015-09-02 04:43    43  ************************
...    ..(  3 skipped).    ..  ************************
298    2015-09-02 04:47    43  ************************
299    2015-09-02 04:48    44  *************************
300    2015-09-02 04:49    43  ************************
...    ..(  2 skipped).    ..  ************************
303    2015-09-02 04:52    43  ************************
304    2015-09-02 04:53    44  *************************
305    2015-09-02 04:54    43  ************************
...    ..( 71 skipped).    ..  ************************
377    2015-09-02 06:06    43  ************************
378    2015-09-02 06:07    44  *************************
379    2015-09-02 06:08    43  ************************
380    2015-09-02 06:09    44  *************************
...    ..(  4 skipped).    ..  *************************
385    2015-09-02 06:14    44  *************************
386    2015-09-02 06:15    43  ************************
...    ..( 75 skipped).    ..  ************************
462    2015-09-02 07:31    43  ************************
463    2015-09-02 07:32    42  ***********************
464    2015-09-02 07:33    43  ************************
465    2015-09-02 07:34    43  ************************
466    2015-09-02 07:35    42  ***********************
467    2015-09-02 07:36    43  ************************
...    ..(  5 skipped).    ..  ************************
473    2015-09-02 07:42    43  ************************
474    2015-09-02 07:43    42  ***********************
475    2015-09-02 07:44    43  ************************
...    ..(  2 skipped).    ..  ************************
   0    2015-09-02 07:47    43  ************************
   1    2015-09-02 07:48    42  ***********************
...    ..(  2 skipped).    ..  ***********************
   4    2015-09-02 07:51    42  ***********************
   5    2015-09-02 07:52    43  ************************
   6    2015-09-02 07:53    42  ***********************
   7    2015-09-02 07:54    43  ************************
   8    2015-09-02 07:55    42  ***********************
...    ..(  4 skipped).    ..  ***********************
  13    2015-09-02 08:00    42  ***********************
  14    2015-09-02 08:01    43  ************************
  15    2015-09-02 08:02    43  ************************
  16    2015-09-02 08:03    42  ***********************
  17    2015-09-02 08:04    42  ***********************
  18    2015-09-02 08:05    43  ************************
...    ..(  8 skipped).    ..  ************************
  27    2015-09-02 08:14    43  ************************
  28    2015-09-02 08:15    42  ***********************
  29    2015-09-02 08:16    42  ***********************
  30    2015-09-02 08:17    42  ***********************
  31    2015-09-02 08:18    43  ************************
...    ..( 26 skipped).    ..  ************************
  58    2015-09-02 08:45    43  ************************
  59    2015-09-02 08:46    44  *************************
  60    2015-09-02 08:47    43  ************************
  61    2015-09-02 08:48    43  ************************
  62    2015-09-02 08:49    44  *************************
  63    2015-09-02 08:50    43  ************************
  64    2015-09-02 08:51    44  *************************
  65    2015-09-02 08:52    43  ************************
  66    2015-09-02 08:53    44  *************************
  67    2015-09-02 08:54    43  ************************
  68    2015-09-02 08:55    44  *************************
  69    2015-09-02 08:56    43  ************************
  70    2015-09-02 08:57    44  *************************
  71    2015-09-02 08:58    44  *************************
  72    2015-09-02 08:59    43  ************************
  73    2015-09-02 09:00    44  *************************
  74    2015-09-02 09:01    43  ************************
  75    2015-09-02 09:02    44  *************************
  76    2015-09-02 09:03    43  ************************
...    ..(  6 skipped).    ..  ************************
  83    2015-09-02 09:10    43  ************************
  84    2015-09-02 09:11    44  *************************
  85    2015-09-02 09:12    43  ************************
...    ..(  8 skipped).    ..  ************************
  94    2015-09-02 09:21    43  ************************
  95    2015-09-02 09:22    42  ***********************
  96    2015-09-02 09:23    43  ************************
  97    2015-09-02 09:24    43  ************************
  98    2015-09-02 09:25    43  ************************
  99    2015-09-02 09:26    42  ***********************
100    2015-09-02 09:27    43  ************************
101    2015-09-02 09:28    43  ************************
102    2015-09-02 09:29    43  ************************
103    2015-09-02 09:30    42  ***********************
104    2015-09-02 09:31    43  ************************
105    2015-09-02 09:32    42  ***********************
106    2015-09-02 09:33    42  ***********************
107    2015-09-02 09:34    43  ************************
...    ..(  2 skipped).    ..  ************************
110    2015-09-02 09:37    43  ************************
111    2015-09-02 09:38    42  ***********************
112    2015-09-02 09:39    43  ************************
...    ..(  2 skipped).    ..  ************************
115    2015-09-02 09:42    43  ************************
116    2015-09-02 09:43    42  ***********************
117    2015-09-02 09:44    42  ***********************
118    2015-09-02 09:45    43  ************************
119    2015-09-02 09:46    42  ***********************
120    2015-09-02 09:47    43  ************************
121    2015-09-02 09:48    42  ***********************
122    2015-09-02 09:49    42  ***********************
123    2015-09-02 09:50    42  ***********************
124    2015-09-02 09:51    43  ************************
125    2015-09-02 09:52    42  ***********************
126    2015-09-02 09:53    43  ************************
127    2015-09-02 09:54    42  ***********************
...    ..( 10 skipped).    ..  ***********************
138    2015-09-02 10:05    42  ***********************

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP Log 0x04)
Page Offset Size         Value  Description
  1  =====  =                =  == General Statistics (rev 2) ==
  1  0x008  4               42  Lifetime Power-On Resets
  1  0x010  4             9779  Power-on Hours
  1  0x018  6      28162587126  Logical Sectors Written
  1  0x020  6        430380431  Number of Write Commands
  1  0x028  6     121173223522  Logical Sectors Read
  1  0x030  6        673549530  Number of Read Commands
  2  =====  =                =  == Free-Fall Statistics (rev 1) ==
  2  0x010  4              564  Overlimit Shock Events
  3  =====  =                =  == Rotating Media Statistics (rev 1) ==
  3  0x008  4             9779  Spindle Motor Power-on Hours
  3  0x010  4             9776  Head Flying Hours
  3  0x018  4               86  Head Load Events
  3  0x020  4                0  Number of Reallocated Logical Sectors
  3  0x028  4                0  Read Recovery Attempts
  3  0x030  4                0  Number of Mechanical Start Failures
  4  =====  =                =  == General Errors Statistics (rev 1) ==
  4  0x008  4                0  Number of Reported Uncorrectable Errors
  4  0x010  4              256  Resets Between Cmd Acceptance and Completion
  5  =====  =                =  == Temperature Statistics (rev 1) ==
  5  0x008  1               42  Current Temperature
  5  0x010  1               41~ Average Short Term Temperature
  5  0x018  1               40~ Average Long Term Temperature
  5  0x020  1               68  Highest Temperature
  5  0x028  1               20  Lowest Temperature
  5  0x030  1               61~ Highest Average Short Term Temperature
  5  0x038  1               36~ Lowest Average Short Term Temperature
  5  0x040  1               48~ Highest Average Long Term Temperature
  5  0x048  1               38~ Lowest Average Long Term Temperature
  5  0x050  4            13848  Time in Over-Temperature
  5  0x058  1               55  Specified Maximum Operating Temperature
  5  0x060  4                0  Time in Under-Temperature
  5  0x068  1                5  Specified Minimum Operating Temperature
  6  =====  =                =  == Transport Statistics (rev 1) ==
  6  0x008  4            32140  Number of Hardware Resets
  6  0x018  4                0  Number of Interface CRC Errors
  7  =====  =                =  == Solid State Device Statistics (rev 1) ==
                              |_ ~ normalized value

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  4            0  Command failed due to ICRC error
0x0002  4            0  R_ERR response for data FIS
0x0003  4            0  R_ERR response for device-to-host data FIS
0x0004  4            0  R_ERR response for host-to-device data FIS
0x0005  4            0  R_ERR response for non-data FIS
0x0006  4            0  R_ERR response for device-to-host non-data FIS
0x0007  4            0  R_ERR response for host-to-device non-data FIS
0x0008  4            0  Device-to-host non-data FIS retries
0x0009  4            6  Transition from drive PhyRdy to drive PhyNRdy
0x000a  4            5  Device-to-host register FISes sent due to a COMRESET
0x000b  4            0  CRC errors within host-to-device FIS
0x000d  4            0  Non-CRC errors within host-to-device FIS
0x000f  4            0  R_ERR response for host-to-device data FIS, CRC
0x0010  4            0  R_ERR response for host-to-device data FIS, non-CRC
0x0012  4            0  R_ERR response for host-to-device non-data FIS, CRC
0x0013  4            0  R_ERR response for host-to-device non-data FIS, non-CRC

[root@freenas] ~#

I strongly suspect disk#6 (ADA5) is the issue, based on the information I have.
NOTE: I appreciate the help!
I do feel though, that the information provided in the GUI is still,.. insufficient.
The initial email I received would be nice if it listed the disk in some capacity, the port, the S/N the name, whatever - as it stands I got nothing and need to hunt and peck for the information. Thanks to you and what little knowledge I have, I've probably found the disk but it should be easier for a layman.

 

[cyberjock]

glabel status will give you the gptid->device conversion.

ada5 looks fine according to SMART info though. :P

So do the glabel status to see which disk has 3 errors, then provide the SMART info for that disk. It's probably already failing though.

 

[DrKK]

Yeah that drive looks ok. Look at the glabel's as Cyberjock says.

Also, your drives are running a bit warmer than I'd suggest. Especially those Toshiba's should be kept in the low 30's if you can. This drive seems to spend more of its life in the low 40's.

 

[diskdiddler]

Code:

[root@freenas] ~# glabel status
                                      Name  Status  Components
gptid/92ffaf88-b686-11e4-b873-28924a2d5aca     N/A  da0p1
gptid/93096e78-b686-11e4-b873-28924a2d5aca     N/A  da0p2
gptid/ad7e6723-b687-11e4-ba56-28924a2d5aca     N/A  da1p1
gptid/ad8cafcc-b687-11e4-ba56-28924a2d5aca     N/A  da1p2
gptid/7a3a09f4-10cb-11e4-bc4d-441ea13cac6b     N/A  ada0p2
gptid/7aaa3c1a-10cb-11e4-bc4d-441ea13cac6b     N/A  ada1p2
gptid/2a07e64b-9fc5-11e4-8140-28924a2d5aca     N/A  ada2p2
gptid/7b8d77f7-10cb-11e4-bc4d-441ea13cac6b     N/A  ada3p2
gptid/90c05956-38d3-11e4-b55a-28924a2d5aca     N/A  ada4p2
gptid/7c75735a-10cb-11e4-bc4d-441ea13cac6b     N/A  ada5p2
[root@freenas] ~#

Disk beginning with "7c75" (final disk) does appear to map to ada5p2 ? So would I be right, thinking that it's definitely disk #6 (ada5?)

 

[diskdiddler]

Yeah that drive looks ok. Look at the glabel's as Cyberjock says.

Also, your drives are running a bit warmer than I'd suggest. Especially those Toshiba's should be kept in the low 30's if you can. This drive seems to spend more of its life in the low 40's.

That topic has come up before and my intention is to keep them as cool as possible, but I'm in Australia, it's physically impossible to keep them under 40c in summer.
I "pushed it" last summer by trying to avoid my external fan, hence the big 68c spike in the history. It's very likely my "fault" about this problem - I just take issue with the ..... difficulty in ensuring the data is correct from FreeNAS UI / logging so I can replace the right disk.
FWIW: This disk has done this multiple times (this is the first time with my external cooling fan though) I suspect it truly has a minor fault :/
NOTE: just to state again, I really appreciate the help, this thread is not just about my problem, it's about "how can we slightly adjust a few emails, logs, reports, menus, to ensure people KNOW which disk caused a problem" (and avoid threads like this in the future)

 

[cyberjock]

If I had to guess what happened, here's my guess:

*something* happened to disconnect the disk from the zpool (obvious that this happened from the emails). Overheating drives can cause them to simply "drop dead". Crappy PSU (overheating PSU?) and SATA cables seem highly likely too.

The system kernel panic'd (obvious from your posts) an hour or so later. (this could mean the system was overheating?) Normally I'd expect a kernel panic from hardware failure.. so there may be a hardware failure that isn't the disk.

On reboot the disk was detected by the system and put back in the pool (typical behavior for a zpool import). The CHKSUM errors are because there are "holes" where data is missing between when the disk was detached from the system and when it crashed (and then rebooted).

So you have your work cut out for you. :P

 

[diskdiddler]

I agree with your analysis.
I'd be inclined to say, hey I have a major system issue but this has occurred 3 times now, each time, always ADA5 - (each time I had to "hunt and peck" to isolate it being ADA5 btw)
I intend to replace the disk, regardless of it passing tests. When I'm inside the system I'll give it a good clean out to boot.

If other disks had decided to drop out / I'd blame the PSU but as it stands, seems likely the HDD is the first course of action.
Would you agree though, it would be nice if the email from the system informed the admin, WHICH disk dropped out? I'd like to think that actually wouldn't be hugely difficult to add as a feature / fix.

One last thing, I think she was in the middle of the monthly scrub, higher CPU, disk, PSU, usage of course, hence the heat / drop out? :/

 

[cyberjock]

I think you've totally oversimplified the issue by replacing ada5. It could very well be a voltage drop from the PSU causing that disk to drop because it's last in the chain. There's TONS of possibilities. I think the chances of it being the disk itself is nearly zero based on the SMART. Not a single indicator shows anything out of place.

The data cable is special to the drive and could easily be to blame.

You could even claim that when the SATA chipset overheats it fails on the channel that happens to be ada5. I could easily dream up a dozen more very possible scenarios. :p


Of course, you could definitely replace the disk and claim victory only because you changed the disk order for the cabling, etc.

Sure, it would be nice to add that feature. But it would also mean keeping a database of what exists and then looking for what is missing. Lots of room for error, lots of room for special hardware (aka RAID controllers) to bone you, etc. It is a great feature that I would love to see happen in theory, but in practice I'd prefer FreeNAS NOT to try to interpret the information for me. Let me, the human being, be the decision maker of what is actually going on. We also don't have any solid info that the drive didn't disappear and reappear a second later. That's all it takes for ZFS to drop a disk, but the middleware could be confused because it typically takes 1-4 seconds... per disk.. to query SMART data. So you're playing a race game. :P

 

[diskdiddler]

I think you've totally oversimplified the issue by replacing ada5. It could very well be a voltage drop from the PSU causing that disk to drop because it's last in the chain. There's TONS of possibilities. I think the chances of it being the disk itself is nearly zero based on the SMART. Not a single indicator shows anything out of place.

The data cable is special to the drive and could easily be to blame.

You could even claim that when the SATA chipset overheats it fails on the channel that happens to be ada5. I could easily dream up a dozen more very possible scenarios. :p


Of course, you could definitely replace the disk and claim victory only because you changed the disk order for the cabling, etc.

I will also change the sata / power cable. Considering that disk has hit 68c though.......... I can't see any harm in replacing it (it's free for me except a trip into the city for a swap out)
Who knows the cause but it does feel like a logical step #1 to begin with.
Ultimately I am glad that I took your advice and went for 6 disk / 2 disk redundancy just in case. I've only lost 1 disk 3 times now but it's enough to shake me.

Sure, it would be nice to add that feature. But it would also mean keeping a database of what exists and then looking for what is missing. Lots of room for error, lots of room for special hardware (aka RAID controllers) to bone you, etc. It is a great feature that I would love to see happen in theory, but in practice I'd prefer FreeNAS NOT to try to interpret the information for me. Let me, the human being, be the decision maker of what is actually going on. We also don't have any solid info that the drive didn't disappear and reappear a second later. That's all it takes for ZFS to drop a disk, but the middleware could be confused because it typically takes 1-4 seconds... per disk.. to query SMART data. So you're playing a race game. :p

This doesn't make sense to me based on my knowledge of software / FreeNAS.
How hard is it to change

The volume ARRAY (ZFS) state is DEGRADED: One or more devices has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

to

The volume ARRAY (ZFS) state is DEGRADED: %DEVICETHATDROPPED% has been removed by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state.

?
I've filed a bug report about it regardless, it seems such a no brainer piece of information personally.

 

[cyberjock]

Actually, its exactly what I explained above. You have to track all that info somewhere, keep it up to date appropriately, and be able to find the "missing" entry. FreeNAS used to spit out the gptid that was missing back in the early days, but that was taken out because people didn't WANT the information.

What one person wants another doesn't. :P

 

[diskdiddler]

Actually, its exactly what I explained above. You have to track all that info somewhere, keep it up to date appropriately, and be able to find the "missing" entry. FreeNAS used to spit out the gptid that was missing back in the early days, but that was taken out because people didn't WANT the information.

What one person wants another doesn't. :p

So outlandishly ridiculous and stupid, I'd almost accuse you of lying. I sincerely doubt you are of course.
There's no hope for some idiot users. This kind of information is DEFINITELY something which should be included. The more information the better.
I can't fathom why anyone would not want it.

 

[diskdiddler]

Well I've replaced the MOLEX -> SATA adapter cable, I've replaced the SATA cable and I've replaced the disk (as well as cleaned out the case of dust)
Let's see how she goes.

I wish I was on like 200k or something, I'd gladly spec out a system with 6 of these instead. http://www.dpreview.com/articles/5938341907/samsung-introduces-pm1633a-world-first-2-5-16tb-ssd quiet, cool, fast.
EDIT:
Just looked up the price of those............ Ok maybe I'd need to be on 500k before I'd consider using 6 of those in an array.

 

[cyberjock]

How has the server been?

 

[diskdiddler]

Thus far she's behaving - but I need to see scrub and smart checks which are only during the end of month - that's when we get full thrash
I oiled the fan too (desk fan keeping the server cool - if you saw my setup you would.... make unpleasant comments... it's very makeshift looking!)

Thanks