corrupted rrdcached files

[SneakiCow]

Hi. The wrong disk got pulled in a hot swap and caused a crash. Now everything in this directory

Code:

/var/db/collectd/rrd/truenas.local/zfs_arc_v2/

seems to corrupted, coming up on terminal with either Input/output error or Integrity check failed. I have tried to remove the files and stop the service but neither worked.

Also unhappily, one the mirrors in the pool (3x mirrored pairs) is degraded with checksum errors on both disks. I have been trying to swap in new disks (known good disks) which seem to replicate the checksum error problem. There was one bad disk that was removed, but the rest and the current disk all pass SMART tests.

So I'm not sure if the rrdcached plugin is the source of the problems or the crashed disk. Also not sure where to go try and fix the problem.

thanks

 

[Samuel Tai]

These files are part of the system dataset, which in your case appeared to be distributed across your striped mirror pool. Unfortunately, you've stated one stripe has both members of the mirror showing checksum errors. Your pool is toast, and will need to be rebuilt from backup.

 

[SneakiCow]

Hey thanks.

I either just pasted over the cracks or actually fixed the degraded pool by replacing the mirror stripe and running "zpool clear *pool*"
but unsurprisingly the rrdcached files are still throwing errors on the terminal; constantly. Also this has stuffed the reporting on the dashboard. So to understand the only way/easiest way to fix/rebuild these files is to remake the pool or restore from a snapshot?

LG tim

 

[Samuel Tai]

No, you didn't fix it. You only got the zpool to ignore the checksum failures. Your underlying data is still munged. A snapshot restore won't work for that reason. You'll need to replace those 2 disks, recreate your pool, and restore from backup.

 

[SneakiCow]

Sweet Thanks