New Error log - Anyone seen this before?

Status
Not open for further replies.

evillone

Explorer
Joined
May 17, 2014
Messages
60
Any help with this is appreciated - Not sure what this output means.
Thanks in advance guys. !!

freenas.local kernel log messages:
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 85 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 50 85 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 10 8f 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 28 8f 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 98 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 f8 98 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 a8 a2 4e 40 e6 00 00 00 d0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 d0 a2 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Retrying command
> (ada5:ata7:0:1:0): READ_DMA48. ACB: 25 00 38 ac 4e 40 e6 00 00 00 a0 00
> (ada5:ata7:0:1:0): CAM status: ATA Status Error
> (ada5:ata7:0:1:0): ATA status: 51 (DRDY SERV ERR), error: 40 (UNC )
> (ada5:ata7:0:1:0): RES: 51 40 a8 ac 4e e6 e6 00 00 00 00
> (ada5:ata7:0:1:0): Error 5, Retries exhausted

-- End of security output --
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
I'd say you have a failing drive. Have you tried smart tests on that drive? But let others comment.

EDIT: Maybe a failing sata cable?
^^^^^^^^^^^^^^^^^^^^As good of a place to start as any, I would also take a look at the CLI command <zpool status>
you can post the results if you need help with that output.
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
state: ONLINE
scan: scrub repaired 0 in 11h39m with 0 errors on Sun Nov 1 10:39:44 2015
config:

NAME STATE READ WRITE CKSUM
NAS ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/683b83af-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/68f0342d-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/698b53b0-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a114719-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a64799b-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6ae45631-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Mon Jan 18 03:46:09 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors
[root@freenas ~]#
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
attempting smart test on drive now - i posted this on my way home as i received the alert emails to my phone - so i was gonna post results,

thanks again guys.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,996
You scheduled the SMART Long test to run one time, not every hour right? IMO the SMART Short test should be run nightly and the Long test should be run weekly. Right now you should run it via the shell "smartctl -t long /dev/ada5". But it could also be a SATA cable as indicated. Run the SMART test and post the output using the INSERT - CODE icons in the text bar above. Look in my signature for Decoding your SMART Data, look at the ID's in red. If you have high values (anything above zero is high) then you likely have a mechanical failure of the drive. If nothing looks bad then it could be the SATA cable or some other hardware issue.
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
Thx Joe, i was just contemplating that right now -
i went under services, turned on SMART and was trying to decide how to run the test,
once i did that ALERTS came up -
which is confusing as the original error was for ada5
  • CRITICAL: Device: /dev/ada2, 296 Currently unreadable (pending) sectors
  • CRITICAL: Device: /dev/ada2, 296 Offline uncorrectable sectors

    unless error was for the cables as you guys indicated, and this is just a coincidence?
rebooted, seems as though ada2 is dead :(
 

Bidule0hm

Server Electronics Sorcerer
Joined
Aug 5, 2013
Messages
3,710
AFAICS ada2 is dead.

Device labels can change from reboot to reboot, the ada2 drive from now can be the ada5 drive from before a previous reboot. So if you rebooted it can explain that.
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
OK - gotcha thx!
but shouldn't zpool status be showing errors right now? it still shows nothing wrong.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
@evillone You might be a bit careful here as you have a pool configure as RAIDz1
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
Thx, BigDave, I have the whole ball 'o wax backed up on another system ;) worse case scenario,
but to be honest, im trying to find out what drive it is, as i think its the last of the "Bad SEAGATE" batch i had bought in 2013.

Update: yep i got 2 left of those bad seagate's and this is the 2nd last one, 1 left! off to buy a NAS drive to replace.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Backup! Oh good! Post some smart results.
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
Ahh the pleasantries of re-silvering , ;)
- replaced with NAS Drive. Let's see how well these do over the long run, its my 3rd one now.
 

Bhoot

Patron
Joined
Mar 28, 2015
Messages
241
-snippy snip
Do run SMART (long and short) at regular intervals. Also I see from your
Code:
state: ONLINE
scan: scrub repaired 0 in 11h39m with 0 errors on Sun Nov 1 10:39:44 2015
config:

NAME STATE READ WRITE CKSUM
NAS ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/683b83af-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/68f0342d-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/698b53b0-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a114719-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6a64799b-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0
gptid/6ae45631-4cf1-11e5-8f38-6805ca32cdb3 ONLINE 0 0 0

errors: No known data errors

pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0h1m with 0 errors on Mon Jan 18 03:46:09 2016
config:

NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
da0p2 ONLINE 0 0 0

errors: No known data errors
you are not running scrubs. I would highly recommend setting up both these things asap as per your convenience of schedule. If you still have issues figuring out a perfect table/schedule cyberjock has laid out a very good schedule for doing all this.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,996
@evillone
Ensure you run the SMART long test on all your drives because you may have more than one failure. When you check the results, look at the data I mentioned which is in my tagline. If you see any ID's 5, 197, 198, 199, etc... with any value other than zero, or the extended test failed with an error, post the results here for further advice. Also within the SMART output is the serial number of the drive, use that number when you pull a drive to replace it. Many people pull the wrong drive and kill their system.
 

evillone

Explorer
Joined
May 17, 2014
Messages
60
Good point guys, I do have to set that stuff up, it's a little intimidating but i need some good pointers to get through it. I will try and find cyberjock's docs for it.
he has some great ones out there.
 
Status
Not open for further replies.
Top