DEGRADED POOL - Continuous Resilvering - Please help!!

Status
Not open for further replies.
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
I am thoroughly frustrated and cannot find the answer and need some additional help. I finally went ahead and killed my old setup because I was having bad disk issues constantly. So I bought a got a dedicated HBA, bought 6x 3TB drives, and did a totally clean install of FreeNAS 11. Created my pool, set services, and great. Things looking good. Then my pool started throwing errors and went degraded with a fault. I ordered replacement drive and resilvering began after setting the new drive in and selecting the old drive and replaced with the new drive. Resilvering went through and great. A few days later I had 2 drives go to fault. I got new drives and replaced those.

Maybe just bad luck and got some bad drives? Anyway, the resilver completed, but I noticed 2 "UNAVAIL" drives still lingering, I keep getting FreeNAS errors, and for some reason the disks present are all showing ONLINE and no errors, but I am having these unattached drives still showing and I cannot "detach" them at all. FreeNAS continually tries to resilver and throws errors because the drives are not present. I waited for the resilver to complete, and I rebooted. Still the same results after full reboot. I noticed an update to U3, so I went ahead and updated, rebooted, still showing and still resilvering! What do I do?? Here's my specs and ZPOOL STATUS.

Intel i5 Quad Core
16GB RAM
8GB Sandisk USB for OS
LSI 9211-8i SAS SATA 8-port PCI-E 6Gb/s (IT Mode)
6x HGST 3TB 7200RPM SATA

Code:
root@freenas:~ # zpool status
  pool: ZFS
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
		continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Fri Sep  8 14:01:21 2017
		178G scanned out of 3.84T at 204M/s, 5h13m to go
		2.09G resilvered, 4.53% done
config:

		NAME											  STATE	 READ WRITE CKSUM
		ZFS											   DEGRADED	 0	 0	 0
		  raidz2-0										DEGRADED	 0	 0	 0
			gptid/19063171-8b52-11e7-9590-90b11c7cdff3	ONLINE	   0	 0	 0  (resilvering)
			gptid/a4f21737-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0
			gptid/a8546b7d-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0
			replacing-3								   DEGRADED	 2	 0	 0
			  5049241458402641816						 UNAVAIL	  0	 0	 0  was /dev/gptid/ab79ad8f-81c8-11e7-8d73-90b11c7cdff3
			  gptid/94a1392e-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0  (resilvering)
			replacing-4								   DEGRADED	10	 0	 0
			  10615410235847933216						UNAVAIL	  0	 0	 0  was /dev/gptid/ad473d89-81c8-11e7-8d73-90b11c7cdff3
			  gptid/7029483a-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0  (resilvering)
			gptid/af2c42a6-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: none requested
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da6p2	 ONLINE	   0	 0	 0

errors: No known data errors
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
It is also worth noting I am seeing this type of output in the shell CONSTANTLY. Any insight would greatly appreciated.

Code:
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 52 da 88 00 00 40 00
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 52 da 90 00 00 40 00
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 52 da 90 00 00 40 00
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 52 da 50 00 00 40 00
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 52 da 50 00 00 40 00
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 52 d4 48 00 01 00 00
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 52 df 98 00 00 40 00
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 52 e7 60 00 00 40 00
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 52 e0 18 00 01 00 00
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 52 e7 60 00 00 40 00
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 52 e7 a0 00 00 80 00
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 52 ea 60 00 00 40 00
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 52 f3 50 00 00 80 00
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 52 f5 d8 00 00 40 00
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas	 (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 52 f7 88 00 00 f8 00 length 126976 SMID 276 terminated ioc 804b scsi 0 state 0 xfer 0
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 52 f7 88 00 00 f8 00
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): CAM status: CCB request completed with an error
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): Retrying command
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 52 fa 00 00 01 00 00
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 53 59 d8 00 01 00 00
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 53 57 d8 00 00 c0 00
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 53 e2 78 00 00 40 00
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas	 (da5:mps0:0:8:0): READ(10). CDB: 28 00 1e 24 8e c0 00 00 20 00 length 16384 SMID 154 terminated ioc 804b scsi 0 state 0 xfer (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas 0
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 53 e1 f0 00 00 40 00
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 53 e1 f8 00 00 40 00
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 53 e2 78 00 00 40 00
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1e 24 8e c0 00 00 20 00
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): CAM status: CCB request completed with an error
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): Retrying command
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 53 da f8 00 01 00 00
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 53 eb 58 00 00 80 00
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 53 eb 60 00 00 80 00
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 53 eb 60 00 00 80 00
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 53 ea e0 00 00 40 00
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da3:mps0:0:6:0): Retrying command (per sense data)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 53 ea d8 00 00 40 00
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:58 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas	 (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 53 f1 20 00 00 d0 00 length 106496 SMID 698 terminated ioc 804b scsi 0 state 0 xfer 0
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 53 f1 20 00 00 d0 00
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): Retrying command
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 53 f0 20 00 01 00 00
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 53 f8 70 00 00 80 00
Sep  8 16:38:59 freenas	 (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 53 f8 28 00 00 40 00 length 32768 SMID 151 terminated ioc 804b scsi 0 state 0 xfer (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas 0
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 53 f8 70 00 00 80 00
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 53 f8 28 00 00 40 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 53 f7 e8 00 00 40 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 54 46 d0 00 00 80 00
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 54 46 d0 00 00 80 00
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 54 46 d0 00 00 80 00
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): READ(10). CDB: 28 00 1f 54 46 50 00 00 40 00
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da5:mps0:0:8:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas	 (da1:mps0:0:2:0): READ(10). CDB: 28 00 1e c7 c9 48 00 00 08 00 length 4096 SMID 633 terminated ioc 804b scsi 0 state 0 xfer 0
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1e c7 c9 48 00 00 08 00
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): Retrying command
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): READ(10). CDB: 28 00 1f 54 58 28 00 00 f8 00
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da1:mps0:0:2:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas	 (da4:mps0:0:7:0): READ(10). CDB: 28 00 1e c7 c9 40 00 00 10 00 length 8192 SMID 953 terminated ioc 804b scsi 0 state 0 xfer 0
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1e c7 c9 40 00 00 10 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 54 58 60 00 00 40 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 54 5e 60 00 00 40 00
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 54 5e 68 00 00 40 00
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 54 61 a8 00 00 40 00
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas	 (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 54 63 60 00 00 80 00 length 65536 SMID 876 terminated ioc 804b scsi 0 state 0 xfer 0
Sep  8 16:38:59 freenas	 (da4:mps0:0:7:0): WRITE(10). CDB: 2a 00 55 a5 8d 20 00 00 08 00 length 4096 SMID 701 terminated ioc 804b scsi 0 state 0 xfer (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 54 63 60 00 00 80 00
Sep  8 16:38:59 freenas 0
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): WRITE(10). CDB: 2a 00 55 a5 8d 20 00 00 08 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 54 64 e8 00 00 40 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas	 (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 54 62 68 00 01 00 00 length 131072 SMID 134 terminated ioc 804b scsi 0 state 0 xfer 0
Sep  8 16:38:59 freenas	 (da3:mps0:0:6:0): WRITE(10). CDB: 2a 00 55 a5 8d 28 00 00 08 00 length 4096 SMID 723 terminated ioc 804b scsi 0 state 0 xfer (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 54 62 68 00 01 00 00
Sep  8 16:38:59 freenas 0
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): Retrying command
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): WRITE(10). CDB: 2a 00 55 a5 8d 28 00 00 08 00
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): CAM status: CCB request completed with an error
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): Retrying command
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): READ(10). CDB: 28 00 1f 54 63 68 00 00 c8 00
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da3:mps0:0:6:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 54 69 e8 00 00 80 00
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): READ(10). CDB: 28 00 1f 54 69 f0 00 00 80 00
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da0:mps0:0:1:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): READ(10). CDB: 28 00 1f 54 6a a8 00 00 40 00
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
Sep  8 16:38:59 freenas (da4:mps0:0:7:0): Retrying command (per sense data)
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): READ(10). CDB: 28 00 1f 54 75 a8 00 00 40 00
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): CAM status: SCSI Status Error
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI status: Check Condition
Sep  8 16:38:59 freenas (da2:mps0:0:5:0): SCSI sense: ABORTED COMMAND asc:47,3 (Information unit iuCRC error detected)
 
D

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
According to your zpool status, the drives are STILL resilvering. Those "UNAVAIL" drives should go away once resilvering is done. Zpool status continues to display that info until it's sure the replacement drives are fully resilvered.

As far as all the drive failures, I suspect it's the cabling. Those SCSI errors are usually due to bad physical connections to the HBA or to the drives. I've used the LSI HBA before and they are solid. And the HGST are probably the most reliable drives on the market. When the drives failed the second time, was it the same drive slots that were failing (i.e. It was always Drive #2 and #3) or was it different drive slot? If it's always the same drive slot (position), then it's definitely a cable issue. Get new cables, make sure they are all firmly connected and see if the problem goes away. They sell cables that have a latching mechanism. I would recommend using those.
 
D

DaveY

Contributor
Joined
Dec 1, 2014
Messages
141
Hmm, I use similar cables, but they were from Amazon https://www.amazon.com/gp/product/B008VLHTA4/

Looking at your SCSI errors again, it's definitely the HBA or the splitter cables and not the drives. You're getting SCSI errors on all 6 of your drives. That sounds like bad cables or bad HBA connection. The likelihood of you getting the worst batch of HGST drives in history is pretty slim.

Where did you buy the HBA? Was it also on Ebay? Also, you said you killed your old setup due to bad drives, what was the old setup specs? Having that many failures on your brand new drives is very unlikely.
 
I

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
Did you burn in your drives when you got them? You should.

Also rs225 might have a good point. Use a different power supply in case you are using the same one from the old setup.
 
P

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
What is your motherboard? What else do you have in your configuration that isn't listed? What happens when you attach the drives directly to the motherboard instead of the HBA?
 
joeschmuck

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,994
I agree with @pschatz100 , since you only have six hard drives, can you attach them to your motherboard directly and remove the HBA from the equation?
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
I appreciate all of the comments, suggestions, and help! My old setup consisted of a bunch of 1TB drives pulled from old systems, etc. I would test them for bad sectors or errors, and if they passed the test I would add them to my "good" stack. Rotate as they failed, etc. I ran out of extra drives and decided to pull the trigger on the new HGST drives. I also wanted to get faster, and more SATA ports so I got the LSI HBA and cables from eBay. The LSI card came pre-flashed in IT mode. Because of all the changes I backed up my array and wanted to start with a clean install on a new USB Stick (Sandisk Cruzer Blade 8GB) and create a new RAIDZ2 array with the new drives.

The motherboard is from a Dell OptiPlex 9020 Mid Tower. The system passes all diagnostic tests and memory tests (including extended tests). I took that motherboard and put it in a nicer case that supports more drives and has much better cooling, and also has a Thermaltake 850 WATT PSU. I don't believe the PSU is at fault as I have checked output with a PSU tester and it shows within normal ranges. This in my mind puts the fault back to the HBA and cables as I also agree that the likelihood of that many bad drives is very slim.

That being said, I will try the following, and if this fails I will try a different PSU as well.
  1. Move 2 drives directly to the motherboard and use my RocketRaid Card (in JBOD) mode for the other 4 drives. I know this card works.
  2. This move will also eliminate the cables and use standard SATA cables.
If I still have errors, I will assume it's the PSU/Mobo. If not, I will assume it's the Mini SAS cables.

Could the version of "IT Mode" firmware on that HBA cause this? Should I be looking at a specific version or distribution/variant of firmware? Is there a driver I need to install to FreeNAS for better support? I have only read that you need the LSI Card to be flashed in "IT Mode" but that was it. Let me know your thoughts.

Thank you!!!
Kevin
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
Also... The resilvering process would start over and go from something like 45% down to 2%, and the amount of GB resilvered would also drop down to the beginning. It FINALLY finished and says complete, however it still shows the old drives and DEGRADED status...

In reference to the files listed for PLEX, I have Plex setup in a Jail and seems to work just fine.

Code:
root@freenas:~ # zpool status -v
  pool: ZFS
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 3.45G in 3h29m with 9 errors on Sun Sep 10 20:58:54 2017
config:

		NAME											  STATE	 READ WRITE CKSUM
		ZFS											   DEGRADED	 0	 0	 9
		  raidz2-0										DEGRADED	 0	 0	18
			gptid/19063171-8b52-11e7-9590-90b11c7cdff3	ONLINE	   0	 0	 0
			gptid/a4f21737-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0
			gptid/a8546b7d-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0
			replacing-3								   DEGRADED	25	 0	 0
			  5049241458402641816						 UNAVAIL	  0	 0	 0  was /dev/gptid/ab79ad8f-81c8-11e7-8d73-90b11c7cdff3
			  gptid/94a1392e-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0
			replacing-4								   DEGRADED	13	 0	 0
			  10615410235847933216						UNAVAIL	  0	 0	 0  was /dev/gptid/ad473d89-81c8-11e7-8d73-90b11c7cdff3
			  gptid/7029483a-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/af2c42a6-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0

errors: Permanent errors have been detected in the following files:

		/mnt/ZFS/Share/Plex/localhost/b/7f03803e889a67e24b36c68fad137fb7e52bcd3.bundle/Contents/GoP-0.xml
		/mnt/ZFS/Share/Plex/localhost/c/a7e6206c66155aec916f1d2b039e698c768f8e9.bundle/Contents/GoP-0.xml

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h3m with 0 errors on Sat Sep  9 03:48:11 2017
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da6p2	 ONLINE	   0	 0	 0

errors: No known data errors
 
R

rs225

Guru
Joined
Jun 28, 2014
Messages
878
The resilver can't complete if there are checksum errors on the raidz or top level. If you can find and fix the hardware, do that. If you can connect your old drives while the new ones resilver, that might also help.
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
Okay, that makes sense. The cables recommended above will be here tomorrow. I will leave the HBA alone (probably re-seat it) and swap cables first before anything else to see if that clears things up. If not, I will move forward with the other proposed recommendations and try to isolate the issue further. I will post back results after the new cables are in. Thank you!
 
Stux

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
kjparenteau said:
Could the version of "IT Mode" firmware on that HBA cause this? Should I be looking at a specific version or distribution/variant of firmware? Is there a driver I need to install to FreeNAS for better support? I have only read that you need the LSI Card to be flashed in "IT Mode" but that was it. Let me know your thoughts.
Click to expand...

For an LSI2008 based board, as I believe the 9211-8i is, you should be using IT firmware 20.00.07.00 (at time of writing).

Earlier versions (for example, 20.00.04.00) did have issues which can show up as SAS errors.

sas2flash -list

Should show you the firmware version.

Also, is it possible your HBA is overheating? Try directing a fan at the heatsink and see if your issues go away.
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
I know for a fact it is not overheating as I mounted an 80mm fan specifically directed towards the HBA to ensure it is kept cool. It is also inside a Corsair Obsidian 650D case which should provide great cooling. All of the HDD's are directly behind a 200mm fan in the front of the chassis, then vented by another 200mm fan out of the top of the case, and the rear 120mm fan. Plus the CPU itself of course has a fan.

As far as firmware goes, I think you may be on to something... Am I able to flash new firmware via FreeNAS? Or do I need to boot do the card itself? Here's the output of the command you suggested, and it looks like I am running a very old version...

Code:
root@freenas:~ # sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		Adapter Selected is a LSI SAS: SAS2008(B2)

		Controller Number			  : 0
		Controller					 : SAS2008(B2)
		PCI Address					: 00:01:00:00
		SAS Address					: 500605b-0-02c8-b23a
		NVDATA Version (Default)	   : 14.00.00.07
		NVDATA Version (Persistent)	: 14.00.00.07
		Firmware Product ID			: 0x2213 (IT)
		Firmware Version			   : 20.00.00.00
		NVDATA Vendor				  : LSI
		NVDATA Product ID			  : SAS9200-8e
		BIOS Version				   : 07.39.00.00
		UEFI BSD Version			   : N/A
		FCODE Version				  : N/A
		Board Name					 : SAS9200-8e
		Board Assembly				 : N/A
		Board Tracer Number			: N/A

		Finished Processing Commands Successfully.
		Exiting SAS2Flash.


Thank you!
Kevin
 
Ericloewe

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,194
FreeNAS includes the tool, which you just used, but the paranoid among us are more comfortable if cards are flashed from an environment that isn't using the disks, such as the EFI shell or a temporary clean install of FreeNAS with all disks disconnected.
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
So partial good news, it's getting better. I ran the firmware upgrade and rebooted the system. The HBA is now on the latest version as suggested above by Stux. Upon reboot, it of course wanted to restart the resilvering process. It went MUCH faster this time as there were no SCSI errors like I was seeing before. However, it still did not complete and the UNAVAIL disks are still present and not officially replaced... Please see the SSH outputs below...

Also, I have the new Mini SAS to SATA cables coming in today from the previously suggested cable types over the generic ones. I will replace this evening and retry the process again and see if that clears things up. Is there something can can repair or clear those Checksum errors? Scrub? What about those PLEX errors on those files? Kill the Jail/Plugin and start over? The errors seem to be from before and not with the new drives because the drives don't show errors but the pool as a whole does, but maybe I am wrong.

Code:
root@freenas:~ # sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

		Adapter Selected is a LSI SAS: SAS2008(B2)

Num   Ctlr			FW Ver		NVDATA		x86-BIOS		 PCI Addr
----------------------------------------------------------------------------

0  SAS2008(B2)	 20.00.07.00	14.01.00.08	07.39.00.00	 00:01:00:00

		Finished Processing Commands Successfully.
		Exiting SAS2Flash.


Code:
root@freenas:~ # zpool status -v
  pool: ZFS
state: DEGRADED
status: One or more devices has experienced an error resulting in data
		corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
		entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 3.42G in 2h13m with 9 errors on Tue Sep 12 11:00:39 2017
config:

		NAME											  STATE	 READ WRITE CKSUM
		ZFS											   DEGRADED	 0	 0	 9
		  raidz2-0										DEGRADED	 0	 0	18
			gptid/19063171-8b52-11e7-9590-90b11c7cdff3	ONLINE	   0	 0	 0
			gptid/a4f21737-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0
			gptid/a8546b7d-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0
			replacing-3								   DEGRADED	 0	 0	 0
			  5049241458402641816						 UNAVAIL	  0	 0	 0  was /dev/gptid/ab79ad8f-81c8-11e7-8d73-90b11c7cdff3
			  gptid/94a1392e-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0
			replacing-4								   DEGRADED	 0	 0	 0
			  10615410235847933216						UNAVAIL	  0	 0	 0  was /dev/gptid/ad473d89-81c8-11e7-8d73-90b11c7cdff3
			  gptid/7029483a-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/af2c42a6-81c8-11e7-8d73-90b11c7cdff3	ONLINE	   0	 0	 0

errors: Permanent errors have been detected in the following files:

		/mnt/ZFS/Share/Plex/localhost/b/7f03803e889a67e24b36c68fad137fb7e52bcd3.bundle/Contents/GoP-0.xml
		/mnt/ZFS/Share/Plex/localhost/c/a7e6206c66155aec916f1d2b039e698c768f8e9.bundle/Contents/GoP-0.xml

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h3m with 0 errors on Sat Sep  9 03:48:11 2017
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da6p2	 ONLINE	   0	 0	 0

errors: No known data errors
 
K

kjparenteau

Explorer
Joined
Jul 13, 2012
Messages
70
Good news! The firmware update absolutely did the trick and saved the day. I was unaware that the firmware versioning was so specific, but the comment from Stux about upgrading the firmware was spot on. The firmware update cleared the constant stream of SCSI and ATA errors, etc., but I kept having problems with resilvering. The fix after the firmware update was to delete the PLEX library and associated Jail, AND to clear the path on the share itself from the media/storage integration with the Jail to clear the 2 errors listed as permanent errors.

I did a ZPOOL CLEAR, then scrubbed the pool. After the pool was scrubbed, it marked those blocks labeled with permanent errors as clear and now there are no more errors present, AND the 2 disks that were marked as UNAVAIL finally allowed it to resilver properly and be officially replaced by the new drives I put in.

Note: I have NOT yet replaced the generic branded Mini SAS to SATA Breakout cables, but I think I will still go ahead and do so later this evening to the branded ones to be safe.

THANK YOU ALL WHO HELPED TO GET MY ARRAY BACK ONLINE!!!

Code:
root@freenas:~ # zpool status -v
  pool: ZFS
state: ONLINE
  scan: scrub repaired 0 in 1h56m with 0 errors on Tue Sep 12 15:53:55 2017
config:

		NAME											STATE	 READ WRITE CKSUM
		ZFS											 ONLINE	   0	 0	 0
		  raidz2-0									  ONLINE	   0	 0	 0
			gptid/19063171-8b52-11e7-9590-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/a4f21737-81c8-11e7-8d73-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/a8546b7d-81c8-11e7-8d73-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/94a1392e-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/7029483a-9441-11e7-a405-90b11c7cdff3  ONLINE	   0	 0	 0
			gptid/af2c42a6-81c8-11e7-8d73-90b11c7cdff3  ONLINE	   0	 0	 0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h3m with 0 errors on Sat Sep  9 03:48:11 2017
config:

		NAME		STATE	 READ WRITE CKSUM
		freenas-boot  ONLINE	   0	 0	 0
		  da6p2	 ONLINE	   0	 0	 0

errors: No known data errors
 
Status
Not open for further replies.

Similar threads

arameen
  • Locked
Hard Drive Troubleshooting - Massive Failures - Need Help Isolating the Problem(s)
4 5 6 7 8
Replies
144
Views
20K
joeschmuck
joeschmuck
DanPrs
  • Locked
SOLVED Volume Degraded After 2 Offline Drives
Replies
18
Views
3K
DanPrs
DanPrs
L
  • Locked
Slow resilvering speed after replacing second disk
Replies
10
Views
2K
level3
L
V
Disk replaced for upgrade, now zpool degraded after resilvering
Replies
6
Views
2K
Krautmaster
K
g00mbs
SOLVED Degraded zpool -> Unavailable
2
Replies
29
Views
4K
g00mbs
g00mbs
Top