SCSI Status Error and MCA Memory Error

[ggoldfingerd]

I'm currently having two issues with my server, on being this SCSI Status Error, and another being an MCA Error. I've done what research and testing that I could do, now it is time to ask for some help. I'll fist discuss the SCSI Status Error.

On Jan 5th 2015, I received the following error:

Code:

(da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d7 b8 00 00 40 00 length 32768 SMID 608 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d8 f8 00 00 40 00 length 32768 SMID 92 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d7 f8 00 00 40 00 length 32768 SMID 309 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 90 ef 48 60 00 00 08 00
(da2:mps0:0:11:0): CAM status: SCSI Status Error
(da2:mps0:0:11:0): SCSI status: Check Condition
(da2:mps0:0:11:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
(da2:mps0:0:11:0): Info: 0x90ef4860
(da2:mps0:0:11:0): Error 22, Unretryable error

Zpool status:

Code:

Checking status of zfs pools:
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot    29G  2.56G  26.4G         -      -     8%  1.00x  ONLINE  -
matrix          65T  22.1T  42.9T         -     4%    34%  1.00x  ONLINE  /mnt

  pool: matrix
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 8K in 0h0m with 0 errors on Tue Jan  5 18:55:32 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        matrix                                          ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/661a8152-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/66cabe0c-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/6784e9c3-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/68330778-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/68e4afb8-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/69b9d1f7-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/79ba13b6-6ef1-11e5-a52e-0025900806ac  ONLINE       0     1     0
            gptid/7a96dded-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7b725ecf-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7c49da3b-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7d2e8867-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7e0b5030-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0

errors: No known data errors

I performed a SMART short, conveyance, and long test:

Code:

smartctl -a /dev/da2
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX31DC4CKV1K
LU WWN Device Id: 5 0014ee 2b6711437
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Jan  6 17:56:37 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 4064) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 694) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   197   196   021    Pre-fail  Always       -       9116
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       21
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   095   095   000    Old_age   Always       -       4222
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       21
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       18
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1794
194 Temperature_Celsius     0x0022   122   112   000    Old_age   Always       -       30
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      4215         -
# 2  Conveyance offline  Completed without error       00%      4201         -
# 3  Short offline       Completed without error       00%      4201         -
# 4  Short offline       Completed without error       00%      3443         -
# 5  Short offline       Completed without error       00%      3228         -
# 6  Extended offline    Completed without error       00%      3145         -
# 7  Short offline       Completed without error       00%      3060         -
# 8  Short offline       Completed without error       00%      2892         -
# 9  Extended offline    Completed without error       00%      2810         -
#10  Short offline       Completed without error       00%      2724         -
#11  Short offline       Completed without error       00%      2483         -
#12  Extended offline    Completed without error       00%      2400         -
#13  Short offline       Completed without error       00%      2315         -
#14  Short offline       Completed without error       00%      2148         -
#15  Extended offline    Completed without error       00%      1865         -
#16  Conveyance offline  Completed without error       00%       953         -
#17  Short offline       Completed without error       00%       953         -
#18  Extended offline    Completed without error       00%       129         -
#19  Extended offline    Completed without error       00%        12         -
#20  Conveyance offline  Completed without error       00%         0         -
#21  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

The only interesting thing that I saw was that the Load_Cycle_Count was at 1794 when the other drives in this dev that I checked were around 100. IDK if that really means anything, but that was the main difference that I saw.

I post my hardware and software version at the bottom of this post, but my phase version for my M1015 is:

Code:

mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 28 at device 0.0 on pci9
mps0: Firmware: 20.00.04.00, Driver: 20.00.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>

 

[ggoldfingerd]

My dmesg is:

Code:

Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.3-RELEASE-p28 #0 r288272+f229c79: Sat Dec 12 11:58:01 PST 2015
    root@build3.ixsystems.com:/tank/home/stable-builds/FN/objs/os-base/amd64/tank/home/stable-builds/FN/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU           L5630  @ 2.13GHz (2128.04-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x206c2  Family = 0x6  Model = 0x2c  Stepping = 2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ee3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant, performance statistics
real memory  = 138512695296 (132096 MB)
avail memory = 132995047424 (126833 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <062310 APIC1640>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3
cpu4 (AP): APIC ID: 18
cpu5 (AP): APIC ID: 19
cpu6 (AP): APIC ID: 20
cpu7 (AP): APIC ID: 21
cpu8 (AP): APIC ID: 32
cpu9 (AP): APIC ID: 33
cpu10 (AP): APIC ID: 34
cpu11 (AP): APIC ID: 35
cpu12 (AP): APIC ID: 50
cpu13 (AP): APIC ID: 51
cpu14 (AP): APIC ID: 52
cpu15 (AP): APIC ID: 53
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0: Changing APIC ID to 6
ioapic1: Changing APIC ID to 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ispfw: registered firmware <isp_1040>
ispfw: registered firmware <isp_1040_it>
ispfw: registered firmware <isp_1080>
ispfw: registered firmware <isp_1080_it>
ispfw: registered firmware <isp_12160>
ispfw: registered firmware <isp_12160_it>
ispfw: registered firmware <isp_2100>
ispfw: registered firmware <isp_2200>
ispfw: registered firmware <isp_2300>
ispfw: registered firmware <isp_2322>
ispfw: registered firmware <isp_2400>
ispfw: registered firmware <isp_2400_multi>
ispfw: registered firmware <isp_2500>
ispfw: registered firmware <isp_2500_multi>
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
padlock0: No ACE support.
acpi0: <NEC > on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, bff00000 (3) failed
cpu0: <ACPI CPU> on acpi0
ACPI Warning: Incorrect checksum in table [OEMB] - 0x8E, should be 0x83 (20111123/tbutils-290)
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci9: <ACPI PCI bus> on pcib1
mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem 0xfaf3c000-0xfaf3ffff,0xfaf40000-0xfaf7ffff irq 28 at device 0.0 on pci9
mps0: Firmware: 20.00.04.00, Driver: 20.00.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci8: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci7: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci6: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xbe80-0xbe9f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xbe20-0xbe3f irq 21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xbdc0-0xbddf irq 19 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xfacf6000-0xfacf63ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib6: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib7
em0: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xcf80-0xcf9f mem 0xfade0000-0xfadfffff,0xfaddc000-0xfaddffff irq 16 at device 0.0 on pci3
em0: Using MSIX interrupts with 3 vectors
em0: Ethernet address: 00:25:90:08:06:ac
pcib8: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
pci4: <ACPI PCI bus> on pcib8
em1: <Intel(R) PRO/1000 Network Connection 7.4.2> port 0xdf80-0xdf9f mem 0xfaee0000-0xfaefffff,0xfaedc000-0xfaedffff irq 17 at device 0.0 on pci4
em1: Using MSIX interrupts with 3 vectors
em1: Ethernet address: 00:25:90:08:06:ad
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xbf00-0xbf1f irq 23 at device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus4 on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xbec0-0xbedf irq 19 at device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus5 on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xbea0-0xbebf irq 18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6 on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfacf8000-0xfacf83ff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib9
vgapci0: <VGA-compatible display> mem 0xf8000000-0xf8ffffff,0xf9ffc000-0xf9ffffff,0xfa000000-0xfa7fffff irq 16 at device 3.0 on pci1
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH10 SATA300 controller> port 0xbff0-0xbff7,0xbf8c-0xbf8f,0xbfe0-0xbfe7,0xbf88-0xbf8b,0xbfa0-0xbfaf,0xbf90-0xbf9f irq 19 at device 31.2 on pci0
ata2: <ATA channel> at channel 0 on atapci0
ata3: <ATA channel> at channel 1 on atapci0
atapci1: <Intel ICH10 SATA300 controller> port 0xbf80-0xbf87,0xbf7c-0xbf7f,0xbf28-0xbf2f,0xbf24-0xbf27,0xbf60-0xbf6f,0xbf30-0xbf3f irq 19 at device 31.5 on pci0
ata4: <ATA channel> at channel 0 on atapci1
ata5: <ATA channel> at channel 1 on atapci1
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
fdc0: <floppy drive controller (FDE)> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef,0x3f7 irq 5 drq 2 on acpi0
qpi0: <QPI system bus> on motherboard
pcib10: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci255: <PCI bus> on pcib10
pcib11: <QPI Host-PCI bridge> pcibus 254 on qpi0
pci254: <PCI bus> on pcib11
ichwd0 on isa0
wbwd0: <Winbond 83627HF/F/HG/G Rev. UD-A> at port 0x2e-0x2f on isa0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
est8: <Enhanced SpeedStep Frequency Control> on cpu8
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
est9: <Enhanced SpeedStep Frequency Control> on cpu9
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
est10: <Enhanced SpeedStep Frequency Control> on cpu10
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
est11: <Enhanced SpeedStep Frequency Control> on cpu11
coretemp12: <CPU On-Die Thermal Sensors> on cpu12
est12: <Enhanced SpeedStep Frequency Control> on cpu12
coretemp13: <CPU On-Die Thermal Sensors> on cpu13
est13: <Enhanced SpeedStep Frequency Control> on cpu13
coretemp14: <CPU On-Die Thermal Sensors> on cpu14
est14: <Enhanced SpeedStep Frequency Control> on cpu14
coretemp15: <CPU On-Die Thermal Sensors> on cpu15
est15: <Enhanced SpeedStep Frequency Control> on cpu15
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <Intel> at usbus4
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub3: 6 ports with 6 removable, self powered
uhub7: 6 ports with 6 removable, self powered
mps0: SAS Address for SATA device = 4875474000d4bd8b
mps0: SAS Address for SATA device = 4875474ef7c9ab7f
mps0: SAS Address for SATA device = 4867484ce0d5cf86
mps0: SAS Address for SATA device = d2625b3cd9c6bb8f
mps0: SAS Address for SATA device = 48684855f1deb180
mps0: SAS Address for SATA device = 4875474000d5b890
mps0: SAS Address for SATA device = 4875474ef7e6a886
mps0: SAS Address for SATA device = 4867484ce0dac26f
mps0: SAS Address for SATA device = d2625b3cebcad273
mps0: SAS Address for SATA device = 4873474cdcd1d471
mps0: SAS Address for SATA device = 4875474fe1daaa86
mps0: SAS Address for SATA device = 4867484ce0c7ab85
mps0: SAS Address from SATA device = 4875474000d4bd8b
mps0: SAS Address from SATA device = 4875474ef7c9ab7f
mps0: SAS Address from SATA device = 4867484ce0d5cf86
mps0: SAS Address from SATA device = d2625b3cd9c6bb8f
mps0: SAS Address from SATA device = 48684855f1deb180
mps0: SAS Address from SATA device = 4875474000d5b890
mps0: SAS Address from SATA device = 4875474ef7e6a886
mps0: SAS Address from SATA device = 4867484ce0dac26f
mps0: SAS Address from SATA device = d2625b3cebcad273
mps0: SAS Address from SATA device = 4873474cdcd1d471
mps0: SAS Address from SATA device = 4875474fe1daaa86
mps0: SAS Address from SATA device = 4867484ce0c7ab85
mps0: SAS Address for SATA device = 4875473df1c7a887
mps0: SAS Address for SATA device = 48734751dcd5cd95
mps0: SAS Address for SATA device = 486b473dddd1c985
mps0: SAS Address from SATA device = 4875473df1c7a887
mps0: SAS Address from SATA device = 48734751dcd5cd95
mps0: SAS Address from SATA device = 486b473dddd1c985
ugen4.2: <CP1500PFCLCD> at usbus4
ugen2.2: <American Megatrends Inc.> at usbus2
ukbd0: <Keyboard Interface> on usbus2
kbd2 at ukbd0
ums0: <Mouse Interface> on usbus2
ugen7.2: <SanDisk> at usbus7
umass0: <SanDisk Cruzer Fit, class 0/0, rev 2.00/1.27, addr 2> on usbus7
umass0:  SCSI over Bulk-Only; quirks = 0x8100
umass0:6:0:-1: Attached to scbus6
ugen7.3: <SanDisk> at usbus7
umass1: <SanDisk Cruzer Fit, class 0/0, rev 2.00/1.27, addr 3> on usbus7
umass1:  SCSI over Bulk-Only; quirks = 0x8100
umass1:7:1:-1: Attached to scbus7
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da0: Serial Number      WD-WX31DC45TCFP
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da0: quirks=0x8<4K>
da1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da1: Serial Number      WD-WX31DC45TDAU
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da1: quirks=0x8<4K>
da2 at mps0 bus 0 scbus0 target 11 lun 0
da2: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da2: Serial Number      WD-WX31DC4CKV1K
da2: 300.000MB/s transfers
da2: Command Queueing enabled
da2: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da2: quirks=0x8<4K>
da3 at mps0 bus 0 scbus0 target 13 lun 0
da3: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da3: Serial Number      WD-WX71DA4A0AY6
da3: 300.000MB/s transfers
da3: Command Queueing enabled
da3: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da3: quirks=0x8<4K>
da4 at mps0 bus 0 scbus0 target 14 lun 0
da4: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da4: Serial Number      WD-WX21DC42E72L
da4: 300.000MB/s transfers
da4: Command Queueing enabled
da4: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da4: quirks=0x8<4K>
da5 at mps0 bus 0 scbus0 target 15 lun 0
da5: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da5: Serial Number      WD-WX21DC4D5J4K
da5: 300.000MB/s transfers
da5: Command Queueing enabled
da5: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da5: quirks=0x8<4K>
da6 at mps0 bus 0 scbus0 target 16 lun 0
da6: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da6: Serial Number      WD-WX61DA4F0ESZ
da6: 300.000MB/s transfers
da6: Command Queueing enabled
da6: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da6: quirks=0x8<4K>
da7 at mps0 bus 0 scbus0 target 18 lun 0
da7: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da7: Serial Number      WD-WX31DC4CK94D
da7: 300.000MB/s transfers
da7: Command Queueing enabled
da7: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da7: quirks=0x8<4K>
da8 at mps0 bus 0 scbus0 target 19 lun 0
da8: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SCSI-6 device
da8: Serial Number      WD-WMC4N0K9144R
da8: 300.000MB/s transfers
da8: Command Queueing enabled
da8: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da8: quirks=0x8<4K>
da9 at mps0 bus 0 scbus0 target 20 lun 0
da9: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SCSI-6 device
da9: Serial Number      WD-WMC4N0K9C8K6
da9: 300.000MB/s transfers
da9: Command Queueing enabled
da9: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da9: quirks=0x8<4K>
da10 at mps0 bus 0 scbus0 target 25 lun 0
da10: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da10: Serial Number      WD-WX21D9421ASJ
da10: 300.000MB/s transfers
da10: Command Queueing enabled
da10: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da10: quirks=0x8<4K>
da11 at mps0 bus 0 scbus0 target 26 lun 0
da11: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da11: Serial Number      WD-WX31D55A474J
da11: 300.000MB/s transfers
da11: Command Queueing enabled
da11: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da11: quirks=0x8<4K>
da12 at mps0 bus 0 scbus0 target 27 lun 0
da12: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da12: Serial Number      WD-WX31D55A4JK4
da12: 300.000MB/s transfers
da12: Command Queueing enabled
da12: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da12: quirks=0x8<4K>
da13 at mps0 bus 0 scbus0 target 28 lun 0
da13: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da13: Serial Number      WD-WX31D55A4EXK
da13: 300.000MB/s transfers
da13: Command Queueing enabled
da13: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da13: quirks=0x8<4K>
da15 at umass-sim0 bus 0 scbus6 target 0 lun 0
da15: <SanDisk Cruzer Fit 1.27> Removable Direct Access SCSI-6 device
da15: Serial Number 4C530301240325113484
da15: 40.000MB/s transfers
da15: 29812MB (61056064 512 byte sectors: 255H 63S/T 3800C)
da15: quirks=0x2<NO_6_BYTE>
da14 at mps0 bus 0 scbus0 target 29 lun 0
da14: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
da14: Serial Number      WD-WX71D65JEN6E
da14: 300.000MB/s transfers
da14: Command Queueing enabled
da14: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
da14: quirks=0x8<4K>
da16 at umass-sim1 bus 1 scbus7 target 0 lun 0
da16: <SanDisk Cruzer Fit 1.27> Removable Direct Access SCSI-6 device
da16: Serial Number 4C530101330325108030
da16: 40.000MB/s transfers
da16: 29812MB (61056064 512 byte sectors: 255H 63S/T 3800C)
da16: quirks=0x2<NO_6_BYTE>
ses0 at mps0 bus 0 scbus0 target 12 lun 0
ses0: <LSILOGIC SASX36 A.1 7015> Fixed Enclosure Services SCSI-3 device
ses0: 300.000MB/s transfers
ses0: Command Queueing enabled
ses0: SCSI-3 ENC Device
ses0: da0,pass0: Element descriptor: '000'
ses0: da0,pass0: SAS Device Slot Element: 1 Phys at Slot 0, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917844
ses0: da7,pass8: Element descriptor: '001'
ses0: da7,pass8: SAS Device Slot Element: 1 Phys at Slot 1, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917845
ses0: da13,pass14: Element descriptor: '002'
ses0: da13,pass14: SAS Device Slot Element: 1 Phys at Slot 2, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917846
ses0: da8,pass9: Element descriptor: '003'
ses0: da8,pass9: SAS Device Slot Element: 1 Phys at Slot 3, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917847
ses0: da14,pass15: Element descriptor: '004'
ses0: da14,pass15: SAS Device Slot Element: 1 Phys at Slot 4, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917848
ses0: da1,pass1: Element descriptor: '006'
ses0: da1,pass1: SAS Device Slot Element: 1 Phys at Slot 6, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 500304800091784a
ses0: da2,pass2: Element descriptor: '007'
ses0: da2,pass2: SAS Device Slot Element: 1 Phys at Slot 7, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 500304800091784b
ses0: da12,pass13: Element descriptor: '008'
ses0: da12,pass13: SAS Device Slot Element: 1 Phys at Slot 8, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 500304800091784c
ses0: da9,pass10: Element descriptor: '009'
ses0: da9,pass10: SAS Device Slot Element: 1 Phys at Slot 9, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 500304800091784d
ses0: da3,pass4: Element descriptor: '012'
ses0: da3,pass4: SAS Device Slot Element: 1 Phys at Slot 12, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917850
ses0: da5,pass6: Element descriptor: '013'
ses0: da5,pass6: SAS Device Slot Element: 1 Phys at Slot 13, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917851
ses0: da11,pass12: Element descriptor: '014'
ses0: da11,pass12: SAS Device Slot Element: 1 Phys at Slot 14, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917852
ses0: da4,pass5: Element descriptor: '018'
ses0: da4,pass5: SAS Device Slot Element: 1 Phys at Slot 18, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917856
ses0: da6,pass7: Element descriptor: '019'
ses0: da6,pass7: SAS Device Slot Element: 1 Phys at Slot 19, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917857
ses0: da10,pass11: Element descriptor: '020'
ses0: da10,pass11: SAS Device Slot Element: 1 Phys at Slot 20, Not All Phys
ses0:  phy 0: SATA device
ses0:  phy 0: parent 500304800091787f addr 5003048000917858
SMP: AP CPU #1 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #14 Launched!
Timecounter "TSC" frequency 2128040652 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/FreeNAS-9.3-STABLE-201512121950 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ipmi0: <IPMI System Interface> on isa0
ipmi0: KCS mode found at io 0xca2 alignment 0x1 on isa
ipmi0: IPMI device rev. 1, firmware rev. 2.22, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da10p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da11p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da12p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device da13p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
vboxdrv: fAsync=0 offMin=0x252 offMax=0x2c2e
bridge0: Ethernet address: 02:58:01:16:26:00
em0: promiscuous mode enabled
bridge0: link state changed to UP
epair0a: Ethernet address: 02:f5:06:00:0e:0a
epair0b: Ethernet address: 02:f5:06:00:0f:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: promiscuous mode enabled
em0: link state changed to DOWN
ng_ether_ifnet_arrival_event: can't re-name node epair0b
em0: link state changed to UP
epair1a: Ethernet address: 02:60:0c:00:0f:0a
epair1b: Ethernet address: 02:60:0c:00:10:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair1b
epair2a: Ethernet address: 02:69:0c:00:10:0a
epair2b: Ethernet address: 02:69:0c:00:11:0b
epair2a: link state changed to UP
epair2b: link state changed to UP
epair2a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair2b
epair3a: Ethernet address: 02:07:c1:00:11:0a
epair3b: Ethernet address: 02:07:c1:00:12:0b
epair3a: link state changed to UP
epair3b: link state changed to UP
epair3a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair3b
epair4a: Ethernet address: 02:f2:0c:00:12:0a
epair4b: Ethernet address: 02:f2:0c:00:13:0b
epair4a: link state changed to UP
epair4b: link state changed to UP
epair4a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair4b
epair5a: Ethernet address: 02:c8:5c:00:13:0a
epair5b: Ethernet address: 02:c8:5c:00:14:0b
epair5a: link state changed to UP
epair5b: link state changed to UP
epair5a: promiscuous mode enabled
ng_ether_ifnet_arrival_event: can't re-name node epair5b
(da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d7 b8 00 00 40 00 length 32768 SMID 608 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d8 f8 00 00 40 00 length 32768 SMID 92 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d7 f8 00 00 40 00 length 32768 SMID 309 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 90 ef 48 60 00 00 08 00
(da2:mps0:0:11:0): CAM status: SCSI Status Error
(da2:mps0:0:11:0): SCSI status: Check Condition
(da2:mps0:0:11:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
(da2:mps0:0:11:0): Info: 0x90ef4860
(da2:mps0:0:11:0): Error 22, Unretryable error

I decided to clear the error and see if it would occur again.

 

[ggoldfingerd]

On Jan 8th 2015, I received the following error:

Code:

(da6:mps0:0:16:0): READ(10). CDB: 28 00 8d 1b 55 e8 00 00 40 00 length 32768 SMID 964 terminated ioc 804b scsi 0 state 0 xfer 0
(da6:mps0:0:16:0): WRITE(16). CDB: 8a 00 00 00 00 02 a5 e4 59 b0 00 00 00 08 00 00
(da6:mps0:0:16:0): CAM status: SCSI Status Error
(da6:mps0:0:16:0): SCSI status: Check Condition
(da6:mps0:0:16:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
(da6:mps0:0:16:0): Info: 0x2a5e459b0
(da6:mps0:0:16:0): Error 22, Unretryable error

Zpool status:

Code:

Checking status of zfs pools:
NAME           SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
freenas-boot    29G  2.56G  26.4G         -      -     8%  1.00x  ONLINE  -
matrix          65T  23.1T  41.9T         -     5%    35%  1.00x  ONLINE  /mnt

  pool: matrix
state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: resilvered 4K in 0h0m with 0 errors on Fri Jan  8 13:11:05 2016
config:

        NAME                                            STATE     READ WRITE CKSUM
        matrix                                          ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/661a8152-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/66cabe0c-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/6784e9c3-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/68330778-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/68e4afb8-1d06-11e5-a95c-0025900806ac  ONLINE       0     0     0
            gptid/69b9d1f7-1d06-11e5-a95c-0025900806ac  ONLINE       0     1     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/79ba13b6-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7a96dded-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7b725ecf-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7c49da3b-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7d2e8867-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0
            gptid/7e0b5030-6ef1-11e5-a52e-0025900806ac  ONLINE       0     0     0

errors: No known data errors

Here is the updated SMART information:

Code:

smartctl -a /dev/da6
smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p28 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD60EFRX-68MYMN1
Serial Number:    WD-WX61DA4F0ESZ
LU WWN Device Id: 5 0014ee 2b61e0669
Firmware Version: 82.00A82
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5700 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Jan 11 19:53:39 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                ( 4244) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 696) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   206   205   021    Pre-fail  Always       -       8666
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       28
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   094   094   000    Old_age   Always       -       4440
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       28
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       25
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       100
194 Temperature_Celsius     0x0022   121   115   000    Old_age   Always       -       31
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%      4424         -
# 2  Short offline       Completed without error       00%      3539         -
# 3  Short offline       Completed without error       00%      3323         -
# 4  Extended offline    Completed without error       00%      3241         -
# 5  Short offline       Completed without error       00%      3155         -
# 6  Short offline       Completed without error       00%      2988         -
# 7  Extended offline    Completed without error       00%      2905         -
# 8  Short offline       Completed without error       00%      2820         -
# 9  Short offline       Completed without error       00%      2579         -
#10  Extended offline    Completed without error       00%      2496         -
#11  Short offline       Completed without error       00%      2411         -
#12  Short offline       Completed without error       00%      2243         -
#13  Short offline       Completed without error       00%       941         -
#14  Extended offline    Completed without error       00%       123         -
#15  Extended offline    Completed without error       00%       109         -
#16  Conveyance offline  Completed without error       00%         0         -
#17  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I also wanted to post the console messages from the Web GUI since it includes time stamps.

Code:

Jan  1 00:00:00 freenas newsyslog[44036]: logfile turned over
Jan  1 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  1 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160101.0100-3w"
Jan  1 01:00:04 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151211.0100-3w"
Jan  2 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  2 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160102.0100-3w"
Jan  2 01:00:04 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151212.0100-3w"
Jan  2 02:08:18 freenas update_check.py: [freenasOS.Configuration:597] Unable to load http://update.freenas.org/FreeNAS/trains.txt: <urlopen error [Errno 4] Non-recoverable failure in name resolution>
Jan  3 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  3 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160103.0100-3w"
Jan  3 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151213.0100-3w"
Jan  4 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  4 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160104.0100-3w"
Jan  4 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151214.0100-3w"
Jan  5 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  5 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160105.0100-3w"
Jan  5 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151215.0100-3w"
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d7 b8 00 00 40 00 length 32768 SMID 608 terminated ioc 804b scsi 0 state 0 xfer 0
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d8 f8 00 00 40 00 length 32768 SMID 92 terminated ioc 804b scsi 0 state 0 xfer 0
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): READ(10). CDB: 28 00 3d 96 d7 f8 00 00 40 00 length 32768 SMID 309 terminated ioc 804b scsi 0 state 0 xfer 0
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 90 ef 48 60 00 00 08 00
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): CAM status: SCSI Status Error
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): SCSI status: Check Condition
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): Info: 0x90ef4860
Jan  5 18:55:26 freenas (da2:mps0:0:11:0): Error 22, Unretryable error
Jan  5 19:11:06 freenas smbd[6078]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionsi7_quad_shit (ipv4:192.168.1.145:8225) closed connection to service Windows Backups
Jan  5 20:30:31 freenas mps0: SAS Address for SATA device = 48684842e0e0ce7e
Jan  5 20:30:31 freenas mps0: SAS Address from SATA device = 48684842e0e0ce7e
Jan  5 20:30:32 freenas da17 at mps0 bus 0 scbus0 target 30 lun 0
Jan  5 20:30:32 freenas da17: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
Jan  5 20:30:32 freenas da17: Serial Number      WD-WX61D6574PTC
Jan  5 20:30:32 freenas da17: 300.000MB/s transfers
Jan  5 20:30:32 freenas da17: Command Queueing enabled
Jan  5 20:30:32 freenas da17: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
Jan  5 20:30:32 freenas da17: quirks=0x8<4K>
Jan  5 20:30:32 freenas ses0: da17,pass18: Element descriptor: '015'
Jan  5 20:30:32 freenas ses0: da17,pass18: SAS Device Slot Element: 1 Phys at Slot 15, Not All Phys
Jan  5 20:30:32 freenas ses0:  phy 0: SATA device
Jan  5 20:30:32 freenas ses0:  phy 0: parent 500304800091787f addr 5003048000917853
Jan  5 20:30:39 freenas mps0: SAS Address for SATA device = 48684850dfc8bd87
Jan  5 20:30:39 freenas mps0: SAS Address from SATA device = 48684850dfc8bd87
Jan  5 20:30:40 freenas da18 at mps0 bus 0 scbus0 target 31 lun 0
Jan  5 20:30:40 freenas da18: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
Jan  5 20:30:40 freenas da18: Serial Number      WD-WXA1D65E388L
Jan  5 20:30:40 freenas da18: 300.000MB/s transfers
Jan  5 20:30:40 freenas da18: Command Queueing enabled
Jan  5 20:30:40 freenas da18: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
Jan  5 20:30:40 freenas da18: quirks=0x8<4K>
Jan  5 20:30:40 freenas ses0: da18,pass19: Element descriptor: '021'
Jan  5 20:30:40 freenas ses0: da18,pass19: SAS Device Slot Element: 1 Phys at Slot 21, Not All Phys
Jan  5 20:30:40 freenas ses0:  phy 0: SATA device
Jan  5 20:30:40 freenas ses0:  phy 0: parent 500304800091787f addr 5003048000917859
Jan  5 20:30:46 freenas mps0: SAS Address for SATA device = 4868484fefc2b585
Jan  5 20:30:46 freenas mps0: SAS Address from SATA device = 4868484fefc2b585
Jan  5 20:30:47 freenas da19 at mps0 bus 0 scbus0 target 32 lun 0
Jan  5 20:30:47 freenas da19: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
Jan  5 20:30:47 freenas da19: Serial Number      WD-WX91D65DC28J
Jan  5 20:30:47 freenas da19: 300.000MB/s transfers
Jan  5 20:30:47 freenas da19: Command Queueing enabled
Jan  5 20:30:47 freenas da19: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
Jan  5 20:30:47 freenas da19: quirks=0x8<4K>
Jan  5 20:30:47 freenas ses0: da19,pass20: Element descriptor: '022'
Jan  5 20:30:47 freenas ses0: da19,pass20: SAS Device Slot Element: 1 Phys at Slot 22, Not All Phys
Jan  5 20:30:47 freenas ses0:  phy 0: SATA device
Jan  5 20:30:47 freenas ses0:  phy 0: parent 500304800091787f addr 500304800091785a
Jan  5 20:30:53 freenas mps0: SAS Address for SATA device = 48684850dfd8ca93
Jan  5 20:30:53 freenas mps0: SAS Address from SATA device = 48684850dfd8ca93
Jan  5 20:30:54 freenas da20 at mps0 bus 0 scbus0 target 33 lun 0
Jan  5 20:30:54 freenas da20: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
Jan  5 20:30:54 freenas da20: Serial Number      WD-WXA1D65E3HEX
Jan  5 20:30:54 freenas da20: 300.000MB/s transfers
Jan  5 20:30:54 freenas da20: Command Queueing enabled
Jan  5 20:30:54 freenas da20: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
Jan  5 20:30:54 freenas da20: quirks=0x8<4K>
Jan  5 20:30:54 freenas ses0: da20,pass21: Element descriptor: '016'
Jan  5 20:30:54 freenas ses0: da20,pass21: SAS Device Slot Element: 1 Phys at Slot 16, Not All Phys
Jan  5 20:30:54 freenas ses0:  phy 0: SATA device
Jan  5 20:30:54 freenas ses0:  phy 0: parent 500304800091787f addr 5003048000917854
Jan  5 20:31:00 freenas mps0: SAS Address for SATA device = 48684855f1e9d16b
Jan  5 20:31:00 freenas mps0: SAS Address from SATA device = 48684855f1e9d16b
Jan  5 20:31:01 freenas da21 at mps0 bus 0 scbus0 target 34 lun 0
Jan  5 20:31:01 freenas da21: <ATA WDC WD60EFRX-68M 0A82> Fixed Direct Access SCSI-6 device
Jan  5 20:31:01 freenas da21: Serial Number      WD-WX71D65JEYV0
Jan  5 20:31:01 freenas da21: 300.000MB/s transfers
Jan  5 20:31:01 freenas da21: Command Queueing enabled
Jan  5 20:31:01 freenas da21: 5723166MB (11721045168 512 byte sectors: 255H 63S/T 729601C)
Jan  5 20:31:01 freenas da21: quirks=0x8<4K>
Jan  5 20:31:01 freenas ses0: da21,pass22: Element descriptor: '010'
Jan  5 20:31:01 freenas ses0: da21,pass22: SAS Device Slot Element: 1 Phys at Slot 10, Not All Phys
Jan  5 20:31:01 freenas ses0:  phy 0: SATA device
Jan  5 20:31:01 freenas ses0:  phy 0: parent 500304800091787f addr 500304800091784e
Jan  5 21:54:32 freenas smbd[19748]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionsFailed to fetch record!
Jan  5 22:16:03 freenas kernel: Limiting closed port RST response from 204 to 200 packets/sec
Jan  5 22:16:03 freenas kernel: Limiting closed port RST response from 204 to 200 packets/sec
Jan  6 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  6 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160106.0100-3w"
Jan  6 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151216.0100-3w"
Jan  6 13:35:37 freenas MCA: Bank 8, Status 0x8c0000400001009f
Jan  6 13:35:37 freenas MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
Jan  6 13:35:37 freenas MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
Jan  6 13:35:37 freenas MCA: CPU 13 COR (1) RD channel ?? memory error
Jan  6 13:35:37 freenas MCA: Address 0x13461befc0
Jan  6 13:35:37 freenas MCA: Misc 0xc9224b0400010585
Jan  6 13:35:37 freenas MCA: Bank 8, Status 0x8c0000400001009f
Jan  6 13:35:37 freenas MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
Jan  6 13:35:37 freenas MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
Jan  6 13:35:37 freenas MCA: CPU 12 COR (1) RD channel ?? memory error
Jan  6 13:35:37 freenas MCA: Address 0x13461befc0
Jan  6 13:35:37 freenas MCA: Misc 0xc9224b0400010585
Jan  6 13:35:48 freenas MCA: Bank 8, Status 0x8c0000400001009f
Jan  6 13:35:48 freenas MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
Jan  6 13:35:48 freenas MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
Jan  6 13:35:48 freenas MCA: CPU 13 COR (1) RD channel ?? memory error
Jan  6 13:35:48 freenas MCA: Address 0x13461befc0
Jan  6 13:35:48 freenas MCA: Misc 0xc9224b0400011481
Jan  6 13:35:48 freenas MCA: Bank 8, Status 0x8c0000400001009f
Jan  6 13:35:48 freenas MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
Jan  6 13:35:48 freenas MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
Jan  6 13:35:48 freenas MCA: CPU 12 COR (1) RD channel ?? memory error
Jan  6 13:35:48 freenas MCA: Address 0x13461befc0
Jan  6 13:35:48 freenas MCA: Misc 0xc9224b0400011481
Jan  6 21:26:41 freenas smbd[84832]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionsi7_quad_shit (ipv4:192.168.1.145:4637) closed connection to service Media
Jan  7 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  7 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160107.0100-3w"
Jan  7 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151217.0100-3w"
Jan  8 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  8 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160108.0100-3w"
Jan  8 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151218.0100-3w"
Jan  8 01:46:41 freenas smbd[65523]:   STATUS=daemon 'smbd' finished starting up and ready to serve connectionsi7_quad_shit (ipv4:192.168.1.145:5045) closed connection to service Media
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): READ(10). CDB: 28 00 8d 1b 55 e8 00 00 40 00 length 32768 SMID 964 terminated ioc 804b scsi 0 state 0 xfer 0
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): WRITE(16). CDB: 8a 00 00 00 00 02 a5 e4 59 b0 00 00 00 08 00 00
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): CAM status: SCSI Status Error
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): SCSI status: Check Condition
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): Info: 0x2a5e459b0
Jan  8 13:10:59 freenas (da6:mps0:0:16:0): Error 22, Unretryable error
Jan  9 00:00:00 freenas syslog-ng[2193]: Configuration reload request received, reloading configuration;
Jan  9 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs snapshot -r "matrix@auto-20160109.0100-3w"
Jan  9 01:00:03 freenas autosnap.py: [tools.autosnap:71] Popen()ing: /sbin/zfs destroy -r -d "matrix@auto-20151219.0100-3w"
Jan  9 02:08:04 freenas update_check.py: [freenasOS.Configuration:593] Got http error HTTP Error 404: Not Found
Jan  9 02:08:04 freenas update_check.py: [freenasOS.Manifest:386] Could not get CRL file https://web.ixsystems.com/updates/ix_crl.pem

I thought the console message was imported because I was also in the process of installed more drives. The SCSI Status Error occurred before I pulled them in. That same night, I performed the SMART short and conveyance test, then started the long test. On Jan 6th, I enabled the kernel geometry debug flags and performed badblocks testing on the new drives. This console log shows that the error occurred before any of this testing. The badblocks test is still running on the new drives as it takes about 100 hours to complete.

Hardware:

Code:

MB: Supermicro Motherboard X8DTE-F
CPU: 2x L5630 Quad Core Xeon Cpu's
RAM: Micron 128 GB ECC
DISKS: 12x WD Red WD60EFRX
PSU: 2x Supermicro PWS-920P-SQ 920W
CASE: Supermicro 24 Bay 846E1-R900B
SAS/SATA Controller: IBM Serveraid M1015 flashed to IT Mode
UPS: CyberPower CP1500PFCLCD PFC

I just became a FreeNAS user about 7 months ago so I am still new to this. I find it unlikely that two drives with SCSI Status Error are failing based SMART tests. I'll see what happens when the SMART test finished on da6. I'm not sure how to proceed from here, sounds like there could possibly be a hardware issue. Does anyone have any recommendations or see any issues with the data I posted?

I feel that the only questionable hardware that I have are the SFF-8087 Male to Mini SAS cables. My case came with Supermicro cables, but they were not long enough for the M1015. I purchased Monoprice ones off Amazon because they were in stock and could be shipped fast. I still have the Supermicro ones, but I would have to get a different SAS/SATA Controller with the connections on the back of the card. I also have the MCA memory issue, but I think that its unrelated since ECC is correcting the errors.

 

[ggoldfingerd]

This post is for the MCA Memory Error.

I originally built my server with 64GB of memory. The 64GB was from 8x Micron MT36KSZF1G72PZ-1G4D1 sticks. I used MemTest to ensure there were no issues. I never encountered any memory issue in FreeNAS. Before I added my second VDev, I needed to increase my memory to meet minimum requirements of FreeNAS. I added 4x Micron MT36KSF2G72PZ-1G4E1HF and perform MemTest again. No memory issues were found. I eventually started to receive MCA errors in my security run output. I'll post a few of the 10 different occurrences below.

9/30/15:

Code:

> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400017050
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400017050
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400011083
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400011083
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400011080
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400011080
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400010280
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400010280

10/3/15:

Code:

> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400016545
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400016040
> MCA: Bank 8, Status 0x88000040000200cf
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) MS channel ?? memory error
> MCA: Misc 0xc9224b0400013040

11/12/15:

Code:

> MCA: Bank 8, Status 0x88000040000200cf
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) MS channel ?? memory error
> MCA: Misc 0xc9224b0400010042
> MCA: Bank 8, Status 0x88000040000200cf
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) MS channel ?? memory error
> MCA: Misc 0xc9224b0400010042

Every occurrence was on BANK 8. I was able to locate the memory slot using dmidecode:

Code:

# dmidecode 2.12
SMBIOS 2.6 present.
87 structures occupying 3396 bytes.
Table at 0x0009D000.

...
Handle 0x004C, DMI type 17, 28 bytes
Memory Device
        Array Handle: 0x0046
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 8192 MB
        Form Factor: DIMM
        Set: None
        Locator: P2-DIMM2A
        Bank Locator: BANK8
        Type: DDR3
        Type Detail: Other
        Speed: 1066 MHz
        Manufacturer: Micron
        Serial Number: AE008AE0
        Asset Tag: AssetTagNum8
        Part Number: 36KSZF1G72PZ-1G4D1
        Rank: Unknown
...

I've attached the entire dmidecode if needed. I switched BANK8 memory with another BANK. I still received MCA errors for the same BANK. I rearranged all sticks. I stopped receiving the MCA error for about one month until 01/07/15:

Code:

> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400010585
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400010585
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 51
> MCA: CPU 13 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400011481
> MCA: Bank 8, Status 0x8c0000400001009f
> MCA: Global Cap 0x0000000000001c09, Status 0x0000000000000000
> MCA: Vendor "GenuineIntel", ID 0x206c2, APIC ID 50
> MCA: CPU 12 COR (1) RD channel ?? memory error
> MCA: Address 0x13461befc0
> MCA: Misc 0xc9224b0400011481

This seems to tell me that the problem is not in the RAM. Is the only thing hardware item left the motherboard, or is it possible for the CPU to cause this?

I could leave the BANK8 blank and see if that resolves the issue. FreeNAS seems to leave about 20GB free on my server with 128GB of RAM.

 

[ggoldfingerd]

I updated the SMART information for da6. I don't see any issues with the results.

 

[jgreco]

It's certainly possible that it is a bad board. Or a bad CPU. Or a mis-seated CPU. We recently had a build go awry here where one DIMM was POST'ing as failed, but upon inspection and swapping, it appeared the DIMM slot was bad. Pulling the CPU and resocketing it solved the issue.

The first post indicates an LBA out-of-range error. This is extremely unusual and is suggestive of a hardware issue. Obviously it isn't causing constant errors, so it'll be a real joy to track down, but I'd strongly suggest you start taking a serious look at the hardware.

 

[ggoldfingerd]

It's certainly possible that it is a bad board. Or a bad CPU. Or a mis-seated CPU. We recently had a build go awry here where one DIMM was POST'ing as failed, but upon inspection and swapping, it appeared the DIMM slot was bad. Pulling the CPU and resocketing it solved the issue.

The first post indicates an LBA out-of-range error. This is extremely unusual and is suggestive of a hardware issue. Obviously it isn't causing constant errors, so it'll be a real joy to track down, but I'd strongly suggest you start taking a serious look at the hardware.

Thanks for your information. I am going to try and reseat the CPU's. I pulled BANK8 RAM and the error still occurred. This either means that something else is wrong or that BANK8 is not listed correctly. I would go with the former statement.

I will start working on the hardware. I might get a different SAS controller which will allow me to use my Supermico SFF-8087 Male to Mini SAS cables. I am going to consider a newer motherboard and CPU since I also have the MCA error. If I can fix the MCA error by reseating the CPU's, then I will try and keep the current motherboard.

 

[jgreco]

Well, we definitely wish you good luck. ;-)

 

[ggoldfingerd]

I'm continuing to get SCSI Status Errors. They continue to alternate between da2 and da6. I finally got a different error code on the 16th.

Jan 9th 2016:

Code:

(da6:mps0:0:16:0): READ(10). CDB: 28 00 8d 1b 55 e8 00 00 40 00 length 32768 SMID 964 terminated ioc 804b scsi 0 state 0 xfer 0
(da6:mps0:0:16:0): WRITE(16). CDB: 8a 00 00 00 00 02 a5 e4 59 b0 00 00 00 08 00 00
(da6:mps0:0:16:0): CAMstatus: SCSI Status Error
(da6:mps0:0:16:0): SCSI status: Check Condition
(da6:mps0:0:16:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
(da6:mps0:0:16:0): Info: 0x2a5e459b0
(da6:mps0:0:16:0): Error 22, Unretryable error

Jan 14th 2016:

Code:

(da2:mps0:0:11:0): READ(10). CDB: 28 00 ac 87 05 50 00 00 40 00 length 32768 SMID 738 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 d1 ac 28 58 00 00 40 00 length 32768 SMID 939 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 d1 ac 28 98 00 00 40 00 length 32768 SMID 682 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 d1 ac 28 d8 00 00 40 00 length 32768 SMID 803 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 d1 ac 29 18 00 00 40 00 length 32768 SMID 172 terminated ioc 804b scsi 0 state 0 xfer 0
(da2:mps0:0:11:0): WRITE(10). CDB: 2a 00 d1 18 7c 78 00 00 18 00
(da2:mps0:0:11:0): CAMstatus: SCSI Status Error
(da2:mps0:0:11:0): SCSI status: Check Condition
(da2:mps0:0:11:0): SCSI sense: ILLEGAL REQUEST asc:21,0 (Logical block address out of range)
(da2:mps0:0:11:0): Info: 0xd1187c78
(da2:mps0:0:11:0): Error 22, Unretryable error

Jan 16th 2016:

Code:

(da6:mps0:0:16:0): READ(16). CDB: 88 00 00 00 00 02 6e bf 63 30 00 00 00 40 00 00 length 32768 SMID 790 terminated ioc 804b scsi 0 state 0 xfer 0
(da6:mps0:0:16:0): READ(16). CDB: 88 00 00 00 00 02 6e bf 63 b0 00 00 00 40 00 00
(da6:mps0:0:16:0): CAMstatus: SCSI Status Error
(da6:mps0:0:16:0): SCSI status: Check Condition
(da6:mps0:0:16:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)
(da6:mps0:0:16:0): Info: 0x26ebf63b0
(da6:mps0:0:16:0): Error 5, Unretryable error

SMART status still shows no new errors for da2 and da6. I'm going to switch my SFF-8087 Male to Mini SAS cables with each other since that is all I can really do without have other hardware to try at this point.

 

[fx24]

I'm continuing to get SCSI Status Errors. They continue to alternate between da2 and da6. I finally got a different error code on the 16th.

If you figure out your problem, I would certainly like to know the cause. I have had this problem for a year now.

 

[ggoldfingerd]

If you figure out your problem, I would certainly like to know the cause. I have had this problem for a year now.

I definitely will post the final fix when I figure it out. I've tried many different things with no luck. I'm going to replace one of the HDDs to see what that does. Sadly this would have happened by now but the new drive failed BadBlocks testing. I had to RMA and it will probably be a few weeks before I to replace it.

 

[ggoldfingerd]

Well I stopped getting the SCSI errors for over a month. I ended up switching one HDD out. Keep in mind I had two HDD's reporting the SCSI errors. Since I haven't received anymore SCSI errors, I am going to put the HDD I took out into my backup pool. I really doubt the HDD was bad.

I am still getting the MCA errors when my server has higher activity. I just ordered a replacement motherboard since it was not expensive. Hopefully this will solve my issues. I'll update this post again once I know for sure.

 

[fx24]

I also have multiple hard drives with errors; it is intermittent.

So are you saying that your SCSI Status Error went away after swapping out one of the multiple HDDs showing errors?

 

[ggoldfingerd]

I also have multiple hard drives with errors; it is intermittent.

So are you saying that your SCSI Status Error went away after swapping out one of the multiple HDDs showing errors?

No, I cannot say this. I would be confident if I was still get errors right before I switched it out. In fact, I would do an ABA comparison to be 100% sure. It is possible that something is temperature sensitive. This was occurring during the winter months. I will definitely update this thread if I do get it. I have seen multiple SCSI Status Error threads where the final solution isn't posted.

 

[fx24]

I added a fan to cool my HBA and the problem still persists. Gonna try a CPU re-seat next following by swapping out the mobo and lastly swapping out the backplane. Perhaps a whole new case altogether. This is getting old.

 

[jgreco]

I'd almost be suspicious of a knock-off HBA. We've seen them, but not all that frequently in recent years.

 

[fx24]

I'd almost be suspicious of a knock-off HBA. We've seen them, but not all that frequently in recent years.

Thanks for the idea. I'll put that on the list of things to swap out.

 

[ggoldfingerd]

I thought I would provide a quick update. I dug deeper into my server and discovered that my backplane was a BPN-SAS-846EL1, so SAS1. I've been running with 19 6TB hard drives for well over a year now. I have two pools, one pool with two vdevs (zraid2, 6 drives each), and one pool with 7 striped disks. It is possible that my SCSI is related to the backplane since it really isn't suppose to support hard drivers larger than 4TB (or 2TB ?).

I just replaced my backplane with a BPN-SAS2-846EL1. I will have to wait and see if the error occurs due to its random nature. There is still a possibility of the SCSI issue being caused my something else. In fact, I have not had the SCSI error for the two previous months with the SAS1 backplane.

The MCA memory error is another story. I still need to fix this issue!

 

[fx24]

Concerning my error, I actually haven't had it in months. I am unsure if this is due to an update of FreeNAS or if it's because I no longer hammer the pool with heavy usage. Your update has piqued my curiosity though so I think I'll emulate how I used to use it to see if I can replicate the error.

Thanks for the update.

 

[ggoldfingerd]

Well I ended up removing one RAM module at a time until I stopped seeing the error. I just left it out without replacement. I still have 11464MB which is enough.

I will post back soon about my SCSI Status Error. I actually still have the issue and will describe it in detail. I just found a few threads about "WD60EFRX-68L0BN1" HDD issue. I have one of these out of my 19 HDD's. My this has been the issue all along.....