Driver Replacement in Restart Loop

[LostAtRedRock]

For the last week I have been trying to get a drive replacement to complete. I had a drive which had reported errors so I bought a replacement installed it and did a drive replace through the FreeNAS UI. Since then the drive resilver progresses until it reaches some random point and then restarts. I did some digging into the log and see that at the time of the latest restarts at least (I would assume a similar pattern for the past week but have not dug through the older logs) there are errors being reported from the drive I am trying to replace. It seems like this would be kind of an expected behavior right? That is why I am trying to replace the drive in the first place.

My thought it that I may be best just pulling the failing drive completely but except for catastrophic failures I have not gone that route in the past. Details on the output I am seeing are below.

Thoughts?

Output from zpool status:

Code:

  pool: Media

 state: ONLINE

status: One or more devices is currently being resilvered.  The pool will

continue to function, possibly in a degraded state.

action: Wait for the resilver to complete.

  scan: resilver in progress since Sat May 12 16:56:51 2018

		3.41T scanned out of 37.0T at 284M/s, 34h28m to go

		379G resilvered, 9.21% done

config:


NAME											  STATE	 READ WRITE CKSUM

Media											 ONLINE	   0	 0	 0

  raidz1-0										ONLINE	   0	 0	 0

	gptid/d90e1c28-3dc9-11e3-a713-0025908aa958	ONLINE	   0	 0	 0

	gptid/dae8b43e-3dc9-11e3-a713-0025908aa958	ONLINE	   0	 0	 0

	gptid/5abef3ab-4ec9-11e3-9125-0025908aa958	ONLINE	   0	 0	 0

	gptid/383844e8-28f8-11e5-8956-0025908aa958	ONLINE	   0	 0	 0

	replacing-4								   ONLINE	   0	 0	 0

	  gptid/e0c83090-3dc9-11e3-a713-0025908aa958  ONLINE	   0	 0	 0

	  gptid/b80ae62f-4da4-11e8-8a73-0025908aa958  ONLINE	   0	 0	 0  (resilvering)

  raidz1-1										ONLINE	   0	 0	 0

	gptid/4012ec09-55ff-11e6-8ddf-0025908aa958	ONLINE	   0	 0	 0

	gptid/96a5f123-52d8-11e6-bc2d-0025908aa958	ONLINE	   0	 0	 0

	gptid/8272d70e-5065-11e6-ad63-0025908aa958	ONLINE	   0	 0	 0

	gptid/d99b755d-4d4e-11e6-85e8-0025908aa958	ONLINE	   0	 0	 0

	gptid/e56b90fc-4ade-11e6-85e8-0025908aa958	ONLINE	   0	 0	 0

  raidz1-2										ONLINE	   0	 0	 0

	gptid/5e9e9e05-3f78-11e3-bbeb-0025908aa958	ONLINE	   0	 0	 0

	gptid/5f84c05d-3f78-11e3-bbeb-0025908aa958	ONLINE	   0	 0	 0

	gptid/606310ec-3f78-11e3-bbeb-0025908aa958	ONLINE	   0	 0	 0

	gptid/6146a1d3-3f78-11e3-bbeb-0025908aa958	ONLINE	   0	 0	 0

	gptid/622343af-3f78-11e3-bbeb-0025908aa958	ONLINE	   0	 0	 0


errors: No known data errors

The most recent resilver restarts from zpool history -i

Code:

2018-05-12.16:53:34 [txg:27637366] scan aborted, restarting errors=0

2018-05-12.16:53:34 [txg:27637366] scan setup func=2 mintxg=3 maxtxg=27516466

2018-05-12.16:54:49 [txg:27637378] scan aborted, restarting errors=0

2018-05-12.16:54:49 [txg:27637378] scan setup func=2 mintxg=3 maxtxg=27516466

2018-05-12.16:55:34 [txg:27637384] scan aborted, restarting errors=0

2018-05-12.16:55:34 [txg:27637384] scan setup func=2 mintxg=3 maxtxg=27516466

2018-05-12.16:56:51 [txg:27637397] scan aborted, restarting errors=0

2018-05-12.16:56:51 [txg:27637397] scan setup func=2 mintxg=3 maxtxg=27516466

And /var/log/messages contents showing the failures with timestamps similar to the resilver restart:

Code:

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 f4 2b 70 00 00 40 00 length 32768 SMID 155 terminated ioc 804b scsi 0 state 0 xfer 0

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 f4 2f 30 00 00 40 00 length 32768 SMID 896 terminated ioc 804b scsi 0 state 0 xfe(da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 f4 2b 70 00 00 40 00 

May 12 16:55:14 NagelNAS r 0

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): CAM status: CCB request completed with an error

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c1 a5 d3 c8 00 00 40 00 length 32768 SMID 410 terminated ioc 804b scsi 0 state 0 xfe(da11:r 0

May 12 16:55:14 NagelNAS mps1:0: (da11:mps1:0:19:0): READ(10). CDB: 28 00 c1 a5 d4 08 00 00 40 00 length 32768 SMID 258 terminated ioc 804b scsi 0 state 0 xfe19:r 0

May 12 16:55:14 NagelNAS 0): Retrying command

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 f4 2f 30 00 00 40 00 

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): CAM status: CCB request completed with an error

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): Retrying command

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c1 a5 d3 c8 00 00 40 00 

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): CAM status: CCB request completed with an error

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): Retrying command

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c1 a5 d4 08 00 00 40 00 

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): CAM status: CCB request completed with an error

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): Retrying command

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c1 a5 d2 08 00 00 40 00 

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): CAM status: SCSI Status Error

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): SCSI status: Check Condition

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): Info: 0xc1a5d208

May 12 16:55:14 NagelNAS (da11:mps1:0:19:0): Error 5, Unretryable error

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 e7 ff f0 00 00 08 00 length 4096 SMID 480 terminated ioc 804b scsi 0 state 0 xfer 0

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 fa 99 a0 00 01 00 00 length 131072 SMID 964 terminated ioc 804b scsi 0 state 0 xf(da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 e7 ff f0 00 00 08 00 

May 12 16:55:26 NagelNAS er 0

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): CAM status: CCB request completed with an error

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): Retrying command

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 fa 99 a0 00 01 00 00 

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): CAM status: CCB request completed with an error

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): Retrying command

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c1 a5 0d 98 00 00 40 00 

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): CAM status: SCSI Status Error

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): SCSI status: Check Condition

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): Info: 0xc1a50da0

May 12 16:55:26 NagelNAS (da11:mps1:0:19:0): Error 5, Unretryable error

May 12 16:56:43 NagelNAS (da11:mps1:0:19:0): READ(10). CDB: 28 00 c0 12 a1 08 00 01 00 00 

May 12 16:56:43 NagelNAS (da11:mps1:0:19:0): CAM status: SCSI Status Error

May 12 16:56:43 NagelNAS (da11:mps1:0:19:0): SCSI status: Check Condition

May 12 16:56:43 NagelNAS (da11:mps1:0:19:0): SCSI sense: MEDIUM ERROR asc:11,0 (Unrecovered read error)

May 12 16:56:43 NagelNAS (da11:mps1:0:19:0): Info: 0xc012a160

May 12 16:56:43 NagelNAS (da11:mps1:0:19:0): Error 5, Unretryable error

 

[Jailer]

Complete list of hardware please as well as what version of FreeNAS you are running.