New Error log - Anyone seen this before?

[evillone]

Any help with this is appreciated - Not sure what this output means.
Thanks in advance guys. !!

freenas.local kernel log messages:
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted

-- End of security output --

 

[hugovsky]

I'd say you have a failing drive. Have you tried smart tests on that drive? But let others comment.

EDIT: Maybe a failing sata cable?

 

[BigDave]

I'd say you have a failing drive. Have you tried smart tests on that drive? But let others comment.

EDIT: Maybe a failing sata cable?

^^^^^^^^^^^^^^^^^^^^As good of a place to start as any, I would also take a look at the CLI command <zpool status>
you can post the results if you need help with that output.

 

[evillone]

state: ONLINE
scan: scrub repaired 0 in 11h39m with 0 errors on Sun Nov 1 10:39:44 2015
config:

NAME STATE READ WRITE CKSUM
NAS ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/683b83af-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/68f0342d-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/698b53b0-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a114719-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a64799b-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6ae45631-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Mon Jan 18 03:46:09 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors
[root@freenas ~]#

 

[evillone]

attempting smart test on drive now - i posted this on my way home as i received the alert emails to my phone - so i was gonna post results,

thanks again guys.

 

[evillone]

so i just scheduled a long self test to run on the hour -

 

[joeschmuck]

You scheduled the SMART Long test to run one time, not every hour right? IMO the SMART Short test should be run nightly and the Long test should be run weekly. Right now you should run it via the shell "smartctl -t long /dev/ada5". But it could also be a SATA cable as indicated. Run the SMART test and post the output using the INSERT - CODE icons in the text bar above. Look in my signature for Decoding your SMART Data, look at the ID's in red. If you have high values (anything above zero is high) then you likely have a mechanical failure of the drive. If nothing looks bad then it could be the SATA cable or some other hardware issue.

 

[evillone]

Thx Joe, i was just contemplating that right now -
i went under services, turned on SMART and was trying to decide how to run the test,
once i did that ALERTS came up -
which is confusing as the original error was for ada5

• CRITICAL: Device: /dev/ada2, 296 Currently unreadable (pending) sectors
• CRITICAL: Device: /dev/ada2, 296 Offline uncorrectable sectors
  
  unless error was for the cables as you guys indicated, and this is just a coincidence?

rebooted, seems as though ada2 is dead :(

 

[Bidule0hm]

AFAICS ada2 is dead.

Device labels can change from reboot to reboot, the ada2 drive from now can be the ada5 drive from before a previous reboot. So if you rebooted it can explain that.

 

[evillone]

OK - gotcha thx!
but shouldn't zpool status be showing errors right now? it still shows nothing wrong.

 

[BigDave]

@evillone You might be a bit careful here as you have a pool configure as RAIDz1

 

[evillone]

Thx, BigDave, I have the whole ball 'o wax backed up on another system ;) worse case scenario,
but to be honest, im trying to find out what drive it is, as i think its the last of the "Bad SEAGATE" batch i had bought in 2013.

Update: yep i got 2 left of those bad seagate's and this is the 2nd last one, 1 left! off to buy a NAS drive to replace.

 

[BigDave]

Backup! Oh good! Post some smart results.

 

[evillone]

Ahh the pleasantries of re-silvering , ;)
- replaced with NAS Drive. Let's see how well these do over the long run, its my 3rd one now.

 

[Bhoot]

-snippy snip

Do run SMART (long and short) at regular intervals. Also I see from your

Code:

state: ONLINE
scan: scrub repaired 0 in 11h39m with 0 errors on Sun Nov 1 10:39:44 2015
config:

NAME STATE READ WRITE CKSUM
NAS ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/683b83af-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/68f0342d-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/698b53b0-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a114719-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a64799b-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6ae45631-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Mon Jan 18 03:46:09 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors

you are not running scrubs. I would highly recommend setting up both these things asap as per your convenience of schedule. If you still have issues figuring out a perfect table/schedule cyberjock has laid out a very good schedule for doing all this.

 

[joeschmuck]

@evillone
Ensure you run the SMART long test on all your drives because you may have more than one failure. When you check the results, look at the data I mentioned which is in my tagline. If you see any ID's 5, 197, 198, 199, etc... with any value other than zero, or the extended test failed with an error, post the results here for further advice. Also within the SMART output is the serial number of the drive, use that number when you pull a drive to replace it. Many people pull the wrong drive and kill their system.

 

[evillone]

Good point guys, I do have to set that stuff up, it's a little intimidating but i need some good pointers to get through it. I will try and find cyberjock's docs for it.
he has some great ones out there.