Permanent errors have been detected <metadata>:<0x1e>?

[icsy7867]

AMD FX-8120 (I had this already, I know it's overkill. Currently underclocked to ~2.4ghz, slightly undervoltaged and running AMD Cool n' Quiet)
ASUS M5A97 Motherboard
16GB DDR3 ECC Memory
4x 1.5TB WD Drives RaidZ1
1x 2TB WD Drive Striped

On my last scrub this error popped up on a single ZFS drive. This drive does not have overly important data, but it will take some time to restore if I need to wipe the pool. I was hoping there was another way or that you guys had some suggestions.

A couple of days ago I upgrade to ECC Ram. I ran a 1-pass memtest which passed. I know you need to do more than 1 pass, and I will do that overnight when I get a chance.

Running

zpool status -v it tells me:

Code:

Password:
  pool: FTPData
state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: scrub repaired 0 in 3h22m with 1 errors on Sun Jan 31 03:22:22 2016
config:

        NAME                                          STATE     READ WRITE CKSUM
        FTPData                                        ONLINE       0     0     0
          gptid/a84af760-b4a9-11e5-820a-c86000da62cb  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x1e>

Running
smartctl -a /dev/ada0-5 all return passes

And all my data is obtainable, usable and mountable since it is metadata that seems to have failed.

Lastly I am getting a separate issue on my 4 1.5TB drives. But I do not think it is too bad (Please correct me if I am wrong).
But I get:

Code:

ahcichx Timeout on slot X port 0 

only at boot and it never shows up again. Maybe when the disks are being initialized, or if they go to sleep this error pops up the one time?

Also been reading through:
https://forums.freenas.org/index.php?threads/permanent-errors-in-zfs-pool.14453/
Performance wise, though, this thing has been and still is running like a champ!

 

[Mirfster]

A couple of days ago I upgrade to ECC Ram

Does this mean that up until now you have been running with non-ECC memory?

Also, what vdev(s) is "FTPData" pool using?

 

[gpsguy]

From what he said and the results of zpool status, it appears to be his single 2TB drive.

Historically, the only way to fix these metadata issues have been to backup any data that you can, delete the pool, and start over. Fortunately, you only have a single drive.

Also, what vdev(s) is "FTPData" pool using?

 

[icsy7867]

Yeah. I was trying to use hardware that I already had. But after the reading on the importance of ECC memory I went ahead and got some.

I have enough room on my RaidZ1 to back up everything on the 2TB drive. I will just start a copy command tonight to get everything copied over and start trying to rebuild it.

I am a little surprised, though, that for this one file error, it is this difficult to track down! But I will rebuild this sucker and hopefully all will be well after that. Thanks for your support guys! If anyone else has some ideas I am definitely open to suggestions. I did a full SMART test, which passed and once that completed I scrubbed the 2TB drive again and the error persists.

Thanks for your support guys. I really appreciate your time!

 

[gpsguy]

The problem is that we don't know what's contained in "<metadata>:<0x1e>". Often, when zpool status -v uncovers and error, it will list a file (or files). Those you can delete and move on. But, with these, we don't know where the problem lies.

 

[icsy7867]

Thanks for the post.

I copied everything using Rsync from the 2TB drive to my raidz1 pool. Destroyed the volume and readded it. After naming it the same thing and rsync'ing the files back over to the 2TB drive (Keeping the same folder tree) everything seems to be working fine. I have 3 jails that use that drive and they all seem to be accessing it correctly after reapplying the correct permissions.

I did a full scrub and everything is checking out now!

 

[Sakuru]

It's highly recommended to use redundancy to prevent issues like this. If you get another 2 TB drive (it doesn't have to be the same model, but that's preferred) you can set up a mirror.

 

[icsy7867]

I have thought about this! Everything on that 2TB drive is... not critically important. If the drive just died it would not be the end of the world. With 5 HDD's and an 8 core processor (underclocked, undervoltaged and AMD Cool n quiet is enabled) I am trying to be as power efficient, heat efficient, and green as possible.