Permanent Errors on not mounted Pool

[n-buck]

Hello everyone

One of my disks failed, so I replaced it.
Unfortunatley during resilvering, two errors appeard.

Show : zpool status

sudo zpool status -v Nas_Storage
pool: Nas_Storage
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: resilvered 3.33T in 0 days 07:45:57 with 2 errors on Mon Nov 30 21:45:08 2020
config:

NAME STATE READ WRITE CKSUM
Nas_Storage DEGRADED 0 0 2
raidz1-0 DEGRADED 0 0 4
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
xbd4 ONLINE 0 0 0
xbd5 ONLINE 0 0 0
xbd6 ONLINE 0 0 0
xbd7 ONLINE 0 0 0
replacing-6 DEGRADED 0 0 0
10427938720908268574 OFFLINE 0 0 0 was /dev/xbd8
xbd8 ONLINE 0 0 0
xbd9 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126:/localhost/aggregation-cpu-average/cpu-interrupt.rrd
Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126:/localhost/aggregation-cpu-min/cpu-interrupt.rrd

The issue I have is, that I do not want to restore the entire pool. I just want to delete these files.
How can I achieve this? I cannot just delete the file, since it does not exist.

I am using FreeNAS-11.2-U8.
The Pool I'm talking about has 8 Disks, one of them is redundant.

 

[sretalla]

Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126

Those files are in a snapshot, so you should be able to easily delete that (assuming you're happy to lose the rest of its content)

 

[n-buck]

Thanks for the hint.
I deleted all snapshots after 2020 01 26. UnfortunatelyI still have this error!

After deleting them, I did

Code:

zpool clear Nas_Storage

 

[sretalla]

And what did your zpool status -v tell you?

 

[n-buck]

I have the same output, there:

Show : zpool staatus

zpool status -v Nas_Storage
pool: Nas_Storage
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraaded state.
action: Action, wait for the resilver to complete.
scan: resilver in progress since Thu Dec 3 02:20:33 2020 470G scanned at 724M/s, 4.90G issued at 136M/s, 26.5T total 589M resilvered, 0.02% done, 2 days 08:49:09 to go
config:

NAME STATE READ WRITE CKSUM
Nas_Storage DEGRADED 0 0 2
raidz1-0 DEGRADED 0 0 4
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
xbd4 ONLINE 0 0 0
xbd5 ONLINE 0 0 0
xbd6 ONLINE 0 0 0
xbd7 ONLINE 0 0 0
replacing-6 DEGRADED 0 0 0
10427938720908268574 OFFLINE 0 0 0 was /dev/xbd8
xbd8 ONLINE 0 0 0
xbd9 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126:/localhost/aggregation-cpu-average/cpu-interrupt.rrd
Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126:/localhost/aggregation-cpu-min/cpu-interrupt.rrd

Please note, that the resilvering just started, since I powerd the Nas down after the last attempt (Just started to verify 100% that the same files are the problem).

What I did before:
Delete all Snapshots of Nas_Storage newer or on 20200126.
Wait for resilvering to finish.
Clean the Pool (zpool clear Nas_Storage).
Wait for resilvering to finish.
zpool status -v gave me the same output as with the snapshots!

 

[sretalla]

So I guess the snapshots aren't deleted then...

Can you try zfs list -t snapshot | grep manual-20200126

 

[n-buck]

Oh indeed, they are still around (They are not in the UI, thouhg).

Show : zfs list -t snapshot

nico@freenas:~ % zfs list -t snapshot | grep manual-20200126
Nas_Storage/.system@manual-20200126 108K - 162K -
Nas_Storage/.system/configs-1e9984bcf13340bcb68bc263ecb0a902@manual-20200126 121K - 14.7M -
Nas_Storage/.system/configs-c2484c3f88124e51b79845e4fb993a70@manual-20200126 0 - 23.7M -
Nas_Storage/.system/cores@manual-20200126 680K - 21.4M -
Nas_Storage/.system/rrd-1e9984bcf13340bcb68bc263ecb0a902@manual-20200126 26.8M - 27.7M -
Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126 0 - 27.5M -
Nas_Storage/.system/samba4@manual-20200126 700K - 909K -
Nas_Storage/.system/syslog-1e9984bcf13340bcb68bc263ecb0a902@manual-20200126 1.37M - 1.72M -
Nas_Storage/.system/syslog-c2484c3f88124e51b79845e4fb993a70@manual-20200126 0 - 61.0M -
Nas_Storage/.system/webui@manual-20200126

So, how do I get rid of them? :)

 

[sretalla]

zfs destroy Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70@manual-20200126

That should do it (or at least tell you why not).

 

[n-buck]

Ok, I was able to destroy it. Resilvering is in progress again.
I'll give feedback when it done.
Thank you!

 

[n-buck]

Unfortunately this did not work...
I have destroyed all Snapshots for Nas_Storage:

Show : list snapshots

nico@freenas:~ % zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
PluginPool/iocage/jails/nextcloud@ioc_update_11.2-RELEASE-p15 300K - 336K -
PluginPool/iocage/jails/nextcloud/root@ioc_update_11.2-RELEASE-p15 785M - 1.76G -
PluginPool/iocage/releases/11.2-RELEASE/root@nextcloud 2.48M - 1.05G -
freenas-boot/ROOT/11.2-U8@2019-10-07-17:28:57 1.93M - 758M -
freenas-boot/ROOT/11.2-U8@2020-01-18-04:31:24 1.97M - 758M -
freenas-boot/ROOT/11.2-U8@2020-02-26-09:55:11 760M - 761M -

But I still have some errors.
The date disapered but it seems to be the same file...

Show : zpool status

nico@freenas:~ % sudo zpool status -v Nas_Storage
pool: Nas_Storage
state: DEGRADED
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: resilvered 3.28T in 0 days 07:36:04 with 2 errors on Fri Dec 4 07:39:22 2020
config:

NAME STATE READ WRITE CKSUM
Nas_Storage DEGRADED 0 0 2
raidz1-0 DEGRADED 0 0 4
ada1 ONLINE 0 0 0
ada2 ONLINE 0 0 0
xbd4 ONLINE 0 0 0
xbd5 ONLINE 0 0 0
xbd6 ONLINE 0 0 0
xbd7 ONLINE 0 0 0
replacing-6 DEGRADED 0 0 0
10427938720908268574 OFFLINE 0 0 0 was /dev/xbd8
xbd8 ONLINE 0 0 0
xbd9 ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70:/localhost/aggregation-cpu-average/cpu-interrupt.rrd
Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70:/localhost/aggregation-cpu-min/cpu-interrupt.rrd

 

[sretalla]

You might need to do something like this:

service collectd onestop
rm /mnt/Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70/localhost/aggregation-cpu-average/cpu-interrupt.rrd
rm /mnt/Nas_Storage/.system/rrd-c2484c3f88124e51b79845e4fb993a70/localhost/aggregation-cpu-min/cpu-interrupt.rrd
service collectd onestart

See if that can get you over the line.

 

[n-buck]

I cannot delete these files, since they do not exist. (even if collectd is stopped).
I don't have an /mnt/Nas_Storage/.system/] folder at all.

Edit:
Can I insert the old disk (had some read-errors, but I think most of it should be ok) and try to scrub the pool, or is this doomed to fail since I started the disk-replacement already?