How can I get rid of this data corruption message?

[simos.sigma]

Hello community!!!

Yesterday, using the "TeraCopy" program, I was backing up all my files from the pool "tank" of my TrueNAS Core server to an external HDD that was connected to my Windows machine.

When the copy operation finished, I got an error message in "TeraCopy" that talked about a damaged file that could not be copied, and I got a similar error on the TrueNAS server.

CRITICAL
Pool tank state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.
24/08/2023 16:11:33 (Europe/Athens)

My problem is that even though I have replaced the damaged file it keeps showing me the message errors: Permanent errors have been detected in the following files: tank/media/movies:<0x835>.

Code:

root@truenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

  pool: plugins
 state: ONLINE
config:

        NAME                                          STATE     READ WRITE CKSUM
        plugins                                       ONLINE       0     0     0
          gptid/089c40fd-35e0-11ee-8a43-00196691bf31  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          ONLINE       0     0     0
          gptid/3103dcd8-35e0-11ee-8a43-00196691bf31  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        tank/media/movies:<0x835>

Any idea what should I do in that case?

Thank you for your time!!!

 

[sretalla]

tank/media/movies:<0x835>

That's dataset metadata... you may be able to clear it with a destroy of that dataset, but it's not looking great for your pool if this is happening...

The next option after that is a whole pool restore.

You might want to think about your pool/hardware setup as to how corruption might have arrived while you're at it as that should be impossible in a well designed pool with good hardware.

 

[winnielinnie]

You can use to "zpool clear" command to try to get rid of the error message. But since nothing was corrected / repaired, it's moot.

Code:

zpool clear tank

A subsequent scrub will reveal the corruption again.

You're not using any redundancy, which itself is dangerous already.

 

[simos.sigma]

While I was searching for a solution, here on the forum, I read about "scrub" and I ran it using TrueNAS Core shell. It didn't appear to be doing anything at the time, but as I found out later, it was running in the background. I gave up for a while because I had to deal with something else and after about two hours TrueNAS gave me this message and the error was gone.

INFO
Scrub of pool 'tank' finished.
25/08/2023 18:17:06 (Europe/Athens)

Αnd when I checked my pools using the zpool status -v the error was indeed gone from my pool "tank".

Code:

root@truenas[~]# zpool status -v
  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          ada0p2    ONLINE       0     0     0

errors: No known data errors

  pool: plugins
 state: ONLINE
config:

        NAME                                          STATE     READ WRITE CKSUM
        plugins                                       ONLINE       0     0     0
          gptid/089c40fd-35e0-11ee-8a43-00196691bf31  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 01:49:05 with 0 errors on Fri Aug 25 18:17:06 2023
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank                                          ONLINE       0     0     0
          gptid/3103dcd8-35e0-11ee-8a43-00196691bf31  ONLINE       0     0     0

errors: No known data errors

So, regarding the error message, I imagine, "scrub" was the solution in my case(?).

And something else about "scrub". I notice that into "Tasks -> Scrub Tasks" there are two scrub tasks scheduled to run every Sunday. One for "plugins" pool and one for "tank" pool. Should I leave them enabled or it's better to do "scrub" only when I face issues with pools?

You might want to think about your pool/hardware setup as to how corruption might have arrived while you're at it as that should be impossible in a well designed pool with good hardware.

Are you talking about the SATA cables for example? The HDD? Is there anyway now, or the next time where this can happen, to locate where this corruption comes from?

You're not using any redundancy, which itself is dangerous already.

What do you mean by "redundancy"?

 

[danb35]

Should I leave them enabled or it's better to do "scrub" only when I face issues with pools?

Yes, you should leave them enabled. The scrub is ZFS' main way to detect (and repair, if possible) errors in your data.

What do you mean by "redundancy"?

Another drive or drives from which the faulty data could be retrieved or reconstructed, which is how ZFS is ordinarily intended to operate. With only one disk, the best it can do (which it has done) is tell you there's a problem--it doesn't ordinarily have any way to fix it.

 

[simos.sigma]

Yes, you should leave them enabled. The scrub is ZFS' main way to detect (and repair, if possible) errors in your data.

Okay, thank you!!!

With only one disk, the best it can do (which it has done) is tell you there's a problem--it doesn't ordinarily have any way to fix it.

I understand what you're saying and you're right, but unfortunately, I don't have the comfort for a second disk at the moment!

But if in the future, I want to add a second disk which will do the "job" you say, can I somehow just add it, or do I have to go through the whole pool process from the beginning?

 

[Arwen]

No, you don't have to go through any major work.

Attaching a mirror is standard. You don't get any additional space, but gain redundancy. Be careful, if you use the wrong command, you increase your storage, but not redundancy.