mrichardson03
Dabbler
- Joined
- Oct 27, 2014
- Messages
- 22
I've got 6x WD Red 3TB drives in a RAID-Z2 running off the onboard LSI 2308 (
20.00.04.00 IT firmware from SuperMicro) controller on a X10SL7-F, and it's running FreeNAS-9.10-STABLE-201606270534 (before the LSI driver nonsense). I've just had one drive fail, I know. However, I apparently had swap space in use on that drive while it was failing, and in order to get the Web UI back to do the drive replacement, I had to reboot the box. During that reboot, the drive that FreeNAS was telling me had failed changed. Now I'm not sure if I've got two failed drives, or a disk controller that's going on the motherboard.
Before Reboot
The vm_fault messages go on for quite some time. This tells me that da1 is toast.
After Reboot
After the reboot, da1 is back, but now da5 is gone. I've got a replacement disk for da5 getting reslivered in an eSATA dock, and the serial of the disk that was failing before the reboot isn't in the logs anymore. I'm not sure if the controller would rename the drives in this scenario. However, a short SMART test on da1 hasn't come back after 20+ minutes...
20.00.04.00 IT firmware from SuperMicro) controller on a X10SL7-F, and it's running FreeNAS-9.10-STABLE-201606270534 (before the LSI driver nonsense). I've just had one drive fail, I know. However, I apparently had swap space in use on that drive while it was failing, and in order to get the Web UI back to do the drive replacement, I had to reboot the box. During that reboot, the drive that FreeNAS was telling me had failed changed. Now I'm not sure if I've got two failed drives, or a disk controller that's going on the motherboard.
Before Reboot
Code:
Aug 8 12:54:45 freenas mps0: mpssas_prepare_remove: Sending reset for target ID 1 Aug 8 12:54:45 freenas da1 at mps0 bus 0 scbus0 target 1 lun 0 Aug 8 12:54:45 freenas da1: <ATA WDC WD30EFRX-68A 0A80> s/n WD-WMC1T1145005 detached Aug 8 12:54:46 freenas devd: Executing '[ -e /tmp/.sync_disk_done ] && LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/python /usr/local/www/freenasUI/tools/sync_disks.py && LD_LIBRARY_PATH=/usr/local/lib /usr/local/bin/python /usr/local/www/freenasUI/tools/smart_alert.py -d da1' Aug 8 12:54:47 freenas (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 26 c4 33 f0 00 00 40 00 length 32768 SMID 629 terminated ioc 804b scsi 0 state c xfer 0 Aug 8 12:54:47 freenas (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 26 c7 1e b8 00 00 18 00 length 12288 SMID 792 terminated ioc 804b scsi 0 state c xfer(da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 26 c4 33 f0 00 00 40 00 Aug 8 12:54:47 freenas 0 Aug 8 12:54:47 freenas (da1:mps0:0:1:0): CAM status: CCB request completed with an error Aug 8 12:54:47 freenas (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 26 c7 23 58 00 00 18 00 length 12288 SMID 382 terminated ioc 804b scsi 0 state c xfer(da1: 0 Aug 8 12:54:47 freenas mps0:0:1:0): mps0: Error 5, Periph was invalidated Aug 8 12:54:47 freenas Unfreezing devq for target ID 1 Aug 8 12:54:47 freenas (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 26 c7 1e b8 00 00 18 00 Aug 8 12:54:47 freenas (da1:mps0:0:1:0): CAM status: CCB request completed with an error Aug 8 12:54:47 freenas (da1:mps0:0:1:0): Error 5, Periph was invalidated Aug 8 12:54:47 freenas (da1:mps0:0:1:0): WRITE(10). CDB: 2a 00 26 c7 23 58 00 00 18 00 Aug 8 12:54:47 freenas (da1:mps0:0:1:0): CAM status: CCB request completed with an error Aug 8 12:54:47 freenas (da1:mps0:0:1:0): Error 5, Periph was invalidated Aug 8 12:54:47 freenas GEOM_ELI: Device da1p1.eli destroyed. Aug 8 12:54:47 freenas GEOM_ELI: Detached da1p1.eli on last close. Aug 8 12:54:47 freenas (da1:mps0:0:1:0): Periph destroyed Aug 8 12:54:47 freenas devd: Executing 'logger -p kern.notice -t ZFS 'vdev is removed, pool_guid=5301505203498458122 vdev_guid=13611148881476578051'' Aug 8 12:54:47 freenas ZFS: vdev is removed, pool_guid=5301505203498458122 vdev_guid=13611148881476578051 Aug 8 12:54:48 freenas swap_pager: I/O error - pagein failed; blkno 532148,size 12288, error 6 Aug 8 12:54:48 freenas vm_fault: pager read error, pid 15364 (zfsd) Aug 8 12:54:48 freenas swap_pager: I/O error - pagein failed; blkno 532148,size 12288, error 6 Aug 8 12:54:48 freenas vm_fault: pager read error, pid 15364 (zfsd)
The vm_fault messages go on for quite some time. This tells me that da1 is toast.
After Reboot
Code:
Aug 8 13:33:43 freenas da0 at mps0 bus 0 scbus0 target 0 lun 0 Aug 8 13:33:43 freenas da1 at mps0 bus 0 scbus0 target 1 lun 0 Aug 8 13:33:43 freenas da1: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device Aug 8 13:33:43 freenas da1: Serial Number WD-WCC4N0048718 Aug 8 13:33:43 freenas da1: 600.000MB/s transfers Aug 8 13:33:43 freenas da1: Command Queueing enabled Aug 8 13:33:43 freenas da1: 2861588MB (5860533168 512 byte sectors) Aug 8 13:33:43 freenas da1: quirks=0x8<4K> Aug 8 13:33:43 freenas da3 at mps0 bus 0 scbus0 target 3 lun 0 Aug 8 13:33:43 freenas da0: <ATA WDC WD30EFRX-68E 0A82> Fixed Direct Access SPC-4 SCSI device Aug 8 13:33:43 freenas da0: Serial Number WD-WMC4N0F4VZUV Aug 8 13:33:43 freenas da0: 600.000MB/s transfers Aug 8 13:33:43 freenas da0: Command Queueing enabled Aug 8 13:33:43 freenas da0: 2861588MB (5860533168 512 byte sectors) Aug 8 13:33:43 freenas da0: quirks=0x8<4K> Aug 8 13:33:43 freenas da2 at mps0 bus 0 scbus0 target 2 lun 0 Aug 8 13:33:43 freenas da2: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device Aug 8 13:33:43 freenas da2: Serial Number WD-WCC4NDA2NCC9 Aug 8 13:33:43 freenas da2: 600.000MB/s transfers Aug 8 13:33:43 freenas da2: Command Queueing enabled Aug 8 13:33:43 freenas da2: 2861588MB (5860533168 512 byte sectors) Aug 8 13:33:43 freenas da2: quirks=0x8<4K> Aug 8 13:33:43 freenas da4 at mps0 bus 0 scbus0 target 4 lun 0 Aug 8 13:33:43 freenas da3: <ATA WDC WD30EFRX-68A 0A80> Fixed Direct Access SPC-4 SCSI device Aug 8 13:33:43 freenas da4: <ATA WDC WD30EFRX-68E 0A80> Fixed Direct Access SPC-4 SCSI device Aug 8 13:33:43 freenas da4: Serial Number WD-WCC4N0299267 Aug 8 13:33:43 freenas da4: 600.000MB/s transfers Aug 8 13:33:43 freenas da4: Command Queueing enabled Aug 8 13:33:43 freenas da4: 2861588MB (5860533168 512 byte sectors) Aug 8 13:33:43 freenas da4: quirks=0x8<4K> Aug 8 13:33:43 freenas da3: Serial Number WD-WMC1T0860631 Aug 8 13:33:43 freenas da3: 600.000MB/s transfers Aug 8 13:33:43 freenas da3: Command Queueing enabled Aug 8 13:33:43 freenas da3: 2861588MB (5860533168 512 byte sectors) Aug 8 13:33:43 freenas da3: quirks=0x8<4K>
After the reboot, da1 is back, but now da5 is gone. I've got a replacement disk for da5 getting reslivered in an eSATA dock, and the serial of the disk that was failing before the reboot isn't in the logs anymore. I'm not sure if the controller would rename the drives in this scenario. However, a short SMART test on da1 hasn't come back after 20+ minutes...