Pool tank state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaf

[ASap]

I've seen a couple of this topic, but the resolution is still unclear.

So, earlier this morning, I got an alert email from my TrueNAS server saying:

TrueNAS @ truenas

New alerts:
Pool tank state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.
Current alerts:
Pool tank state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

All hardware and disk are new (specs in my signature spoiler)

It's 4 x IronWolf Pro 4TB in RaidZ.

Code:

root@truenas[/home/admin]# zpool status
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
	The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
	the pool may no longer be accessible by software that does not support
	the features. See zpool-features(7) for details.
config:

	NAME         STATE     READ WRITE CKSUM
	boot-pool    ONLINE       0     0     0
	  nvme1n1p3  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:

	NAME                                      STATE     READ WRITE CKSUM
	tank                                      ONLINE       0     0     0
	  raidz1-0                                ONLINE       0     0     0
	    3e5220fe-8ccb-116a-826a-03b929bd1508  ONLINE       0     0     0
	    dd68c5bb-11d7-1621-afaa-f657d7123f8e  ONLINE       0     0     1
	    a67b45e4-ff4a-131c-82a6-39e0149158c1  ONLINE       0     0     1
	    a6cf187f-cc39-19a0-b793-61bf337e300c  ONLINE       0     0     0
	logs	
	  ba9483e5-1165-4d5e-b7a3-2ef7be454869    ONLINE       0     0     0

errors: No known data errors
root@truenas[/home/admin]#

From the Web Interface seems there's no way of clearing this error.
This is my first time having this error after two days of uptime. Is it safe to ignore? Or is something gone wrong? I don't want it to get more sinister in the future.

I've run the Conveyancing S.M.A.R.T. test, and all drives returned successfully.

 

[NugentS]

zpool clear - as it states in the text you posted
Run the smartctl -t long test as well.

You have two devices reporting cksum errors, usually, but not always cable related

 

[ChrisRJ]

If it is more than cables, you should check up on your backups. If you don't have a current one, now is the time for it.

 

[ASap]

Thanks for the responses, I've run an extended S.M.A.R.T test on all drives, and all returned successfully.

I scrubbed the pool, and it didn't find any errors.

And I figured the checksum error happened when I failed to copy one of the @eaDir-directories created by my previous Synology NAS.

Because when I tried to copy that again, it caused my 3rd drive to show checksum failed, and that folder was inaccessible even from the CLI. After some research on the Internet, it turns out that's just an extended attribute for Synology NAS and not related to the actual data. So I went ahead and deleted it.

So here's what I did:

Code:

root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
config:

    NAME         STATE     READ WRITE CKSUM
    boot-pool    ONLINE       0     0     0
      nvme1n1p3  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
    attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
    using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 00:44:05 with 0 errors on Sat Dec 31 22:10:17 2022
config:

    NAME                                      STATE     READ WRITE CKSUM
    tank                                      ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        3e5220fe-8ccb-116a-826a-03b929bd1508  ONLINE       0     0     0
        dd68c5bb-11d7-1621-afaa-f657d7123f8e  ONLINE       0     0     1
        a67b45e4-ff4a-131c-82a6-39e0149158c1  ONLINE       0     0     1
        a6cf187f-cc39-19a0-b793-61bf337e300c  ONLINE       0     0     1
    logs  
      ba9483e5-4165-1d5e-b7a3-2ef7be454869    ONLINE       0     0     0

errors: No known data errors
root@truenas[~]# zpool clear tank
root@truenas[~]# zpool status
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
config:

    NAME         STATE     READ WRITE CKSUM
    boot-pool    ONLINE       0     0     0
      nvme1n1p3  ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 00:44:05 with 0 errors on Sat Dec 31 22:10:17 2022
config:

    NAME                                      STATE     READ WRITE CKSUM
    tank                                      ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        3e5220fe-8ccb-116a-826a-03b929bd1508  ONLINE       0     0     0
        dd68c5bb-11d7-1621-afaa-f657d7123f8e  ONLINE       0     0     0
        a67b45e4-ff4a-131c-82a6-39e0149158c1  ONLINE       0     0     0
        a6cf187f-cc39-19a0-b793-61bf337e300c  ONLINE       0     0     0
    logs  
      ba9483e5-1165-4d5e-b7a3-2ef7be454869    ONLINE       0     0     0

errors: No known data errors
root@truenas[~]#

Hey, Happy New Year 2023, everyone

 

[NugentS]

I notice you have a SLOG drive (log vdev) - what are you using for that?

 

[ChrisRJ]

Also, the SLOG device is not mentioned in your system specs (from signature). So you might want to add it there

 

[ASap]

I thought the SLOG would speed up any Sync writes to the disk.
I was thinking of removing it since it's not bringing many benefits.

 

[NugentS]

"I thought the SLOG would speed up any Sync writes to the disk."
It might - are you actually doing any? SMB is async. Most sync writes are from ESX or NFS

 

[ASap]

"I thought the SLOG would speed up any Sync writes to the disk."
It might - are you actually doing any? SMB is async. Most sync writes are from ESX or NFS

Just SMB predominantly.

 

[ChrisRJ]

Just SMB predominantly.

Check out this.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

"I thought the SLOG would speed up any Sync writes to the disk."
It might - are you actually doing any? SMB is async. Most sync writes are from ESX or NFS

SMB from Mac is synchronous according to people I trust on this topic.

 

[ASap]

Has anyone here tried NFS on Mac?
I can see both SMB and NFS protocols are supported in macOS.