ATA Status Errors

[terroreek]

Hi,

I am wondering if anyone has run into this issue, but all of a sudden my NAS is throwing out this error (see below from dmseg.today and console). It's just repeating and I can see that one of the drives activity lights is just on, not flickering at all. I am running FreeNAS-9.2.0-RELEASE-x64 (ab098f4), now in the status icon, I have a warning it reads as follows;

WARNING: The volume DATA (ZFS) status is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'.

When I take a look at the volume status (see attached), it shows everything as online. I know there is a scrub that's running. Any ideas what the issue might be?? I don't think its a drive failure, but it could be. Any insight would be appreciated. Should I just run zpool clear and see if that fixes it??

Code:

(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 d0 27 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 08 28 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 d0 27 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 10 28 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 d0 27 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 f0 27 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Error 5, Retries exhausted
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 f0 40 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 48 41 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 f0 40 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 48 41 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 f0 40 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 48 41 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 f0 40 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 48 41 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 f0 40 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 20 41 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Error 5, Retries exhausted
GEOM: ada3: the primary GPT table is corrupt or invalid.
GEOM: ada3: using the secondary instead -- recovery strongly advised.
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 4c 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 10 4d 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 4c 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 28 4d 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 4c 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 28 4d 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 4c 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 10 4d 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 4c 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 28 4d 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Error 5, Retries exhausted
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 57 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 f0 57 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 57 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 f0 57 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 57 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 f8 57 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 57 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 f8 57 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 a0 57 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 f0 57 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Error 5, Retries exhausted
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 50 58 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 b8 58 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command
(ada3:ahcich3:0:0:0): READ_FPDMA_QUEUED. ACB: 60 b0 50 58 71 40 04 00 00 00 00 00
(ada3:ahcich3:0:0:0): CAM status: ATA Status Error
(ada3:ahcich3:0:0:0): ATA status: 41 (DRDY ERR), error: 40 (UNC )
(ada3:ahcich3:0:0:0): RES: 41 40 c0 58 71 40 04 00 00 00 00
(ada3:ahcich3:0:0:0): Retrying command

 

[cyberjock]

Didn't try to search for that error, huh?

ada3 is failing and needs to be replaced.

 

[terroreek]

uhhh no, I didn't :(. I got the FreeNAS manual so I got instructions on swapping out the disks. Thank you for you help. However shouldn't the SMART reports I have scheduled tell me that as well?

 

[jgreco]

SMART reports contain whatever the drive manufacturer wants you to think about the state of the drive, and aren't too useful unless you've actually got SMART testing set up and configured.

 

[cyberjock]

Also, if you don't have the SMART monitoring setup so that it will email you when there's a problem, then it'll silent become aware of the problem but not be able to tell you. I can't tell you how many people don't setup the email settings then are astonished when FreeNAS can't email them when it finds a problem. /smh

 

[terroreek]

Ok so I got a replacement drive, I had a WD 2TB Green drive with another 2TB WD Green drive. When I click replace, I get an error that the replacement drive is smaller than the original disk. However it shows that the drive is a 2TB drive, it's not the exact same model, however they are both 2TB disks.

Code:

Error: Disk replacement failed: "cannot replace 12727161674229845451 with gptid/e5e3393c-ad89-11e3-9de2-00012e2cd7a0: device is too small, "

This is what I have in the /var/log/messages

Code:

Mar 17 00:40:42 yoda notifier: swapoff: /dev/ada3p1.eli: No such file or directory
Mar 17 00:40:42 yoda notifier: geli: No such device: /dev/ada3p1.
Mar 17 00:40:42 yoda manage.py: [middleware.exceptions:38] [MiddlewareError: Disk replacement failed: "cannot replace 12727161674229845451 with gptid/4b297cf6-ad8e-11e3-8ee7-00012e2cd7a0: device is too small, "]

In the console I got this message

Code:

GEOM_ELI: Device ada3p1.eli destroyed.
GEOM_ELI: Detached ada3p1.eli on last close.

Any think I can do from shell to replace this the drive??

 

[cyberjock]

So your new disk is slightly smaller than the current designs. My recommendation is to go and set the swapspace to zero in the GUI, then try doing the disk replacement, then set the swapspace to 2GB again.

If that doesn't work your disk is just small enough that it's not suitable for replacement.

Don't try to replace the drive from the CLI. That's just asking for trouble. We don't tell you to replace the disk from the CLI so you are on your own with any consequences now and/or in the future if you choose to do it from the CLI. And I say this because 90% of the time there ARE consequences. :)

 

[terroreek]

... really some days I hate hard drive manufacturers, maybe they should relabel their drives as roughly 2TB give or take. Sorry, I went through the manual I can't seem to find where I change the swap space size.

Also I would like to point out that the pool was created via CLI from Freenas .7, and I have upgraded Freenas along the way, and the upgrade from Freenas .7 to 8 was a full Freenas re-install but importing my ZFS pool. So weird to be told not to use CLI vs GUI, but I'll take that advise.

 

[cyberjock]

Didn't look in the GUI or manual very hard, did you? I grabbed the PDF and searched for "swap size" and it was the FIRST result.

Settings -> General...

Swap size on each drive in GiB, affects new disks only. Setting this to 0 disables swap creation completely (STRONGLY DISCOURAGED).

So couldn't search for your error, and couldn't search for the setting.. interesting!

/facepalm

 

[terroreek]

Didn't look in the GUI or manual very hard, did you? I grabbed the PDF and searched for "swap size" and it was the FIRST result.

Settings -> General...



So couldn't search for your error, and couldn't search for the setting.. interesting!

/facepalm

*sigh* well I don't know what to tell you other than I click the link in your signature and open the manual for my version in firefox on linux and search swap space and it takes me to page 236 of 274, and tells me what swap space nothing related to the gui settings. I went through the table of contents as well for about, to find it, couldn't find relevant information. Searched the terms swap, swapspace etc, as well. I thought it safer to shut my NAS down until I can get a fix. So no I didn't have a chance to look at the GUI since last night, and of course means I can't remotely access it from work. This post and the previous post is from work. So I apologize for not spending extra time on manually scrolling through the ~300 page document. I am not looking for an apology nor your sympathy but maybe understanding under normal circumstances I would be more diligent in my trouble shooting, I am a Sr. Sysadmin. However I have been really busy with work, and am running on very little sleep the last couple of weeks, the job can be 24x7. When I initially posted I was in the middle of what I actually get paid to do, so I didn't have a lot of time to fix a personal device.

However thank you for your help and I truly do appreciate the information you have provided. It looks like it will be another long day at the office but I will report back tonight if that resolves my issue. Thank you again much appreciated.

 

[cyberjock]

Ok.

You can find out the actual total disk space of your disks by doing smartctl -a /dev/XXX | grep Capacity

If your new disk isn't at least the same size as your smallest old disk, you're going to have to buy a bigger disk.

This happens to a few unlucky people. Usually people that migrated pools from other OSes and sometimes for people with older drives of a particular size. Over the years the characteristics of the drive will change by a few MB or GB, and then it's game over for people that want backwards compatibility. :(

 

[terroreek]

Ok.

You can find out the actual total disk space of your disks by doing smartctl -a /dev/XXX | grep Capacity

If your new disk isn't at least the same size as your smallest old disk, you're going to have to buy a bigger disk.

This happens to a few unlucky people. Usually people that migrated pools from other OSes and sometimes for people with older drives of a particular size. Over the years the characteristics of the drive will change by a few MB or GB, and then it's game over for people that want backwards compatibility. :(

Ok thank you sir, and honestly really do appreciate the help. I will report back, on what I find. I have a feeling that I might have to go get another drive. Sorry for being a pain.

 

[cyberjock]

If I were you I'd consider redoing that pool from FreeNAS 0.7. Lots of things have changed and the new FreeNAS project has expectations that can bone you if your system doesn't meet them. I can almost guarantee your pool doesn't meet them, which may be part of your problem.

 

[terroreek]

Hi cyberjock,

Ok so setting the swap to 0 and then replacing the drive did the trick. It re-slivered everything without any issues. I would agree with you, I think that will be my new task. I have been going through some of the recommendations and it looks like my setup needs to be beefed up. I don't have nearly enough ram for the setup. I am going to look at maybe new hardware, or maybe doing a different setup, maybe debian with a btrfs raid 10 setup.

 

[cyberjock]

/shivers at the idea of btrfs.. good luck with that! Hope you don't lose your data if you go with btrfs....

 

[terroreek]

It's a just a thought, I know btrfs gets a bad rap. I have been using it on my linux boxes for years, just not in raid setups, with no data loss. I prefer the snaphshot logic in btrfs, to me it's amaze balls to use for your root file system, handy when you are on debian sid, it's easy to roll back. The only reason I am considering it, is because doesn't have the memory requirements ZFS does, the 8GB requirement didn't exist back on freenas 0.7 :). It's a thought, I'd prefer to stay on Freenas and ZFS. I'll start looking for some hardware, if I can't find what I am looking for its my last option.