First ZFS error, please advice.

[GrimmReaperNL]

So, two days ago I received the following email:

New alerts:

Pool TrueNAS state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

Current alerts:

Pool TrueNAS state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

Dashboard shows 'Pool: UNHEALTHY' and 'Disks with Errors: 1'

The disks looking icon (pool status) in the top right does nothing for me.

'Storage > Topology -> Manage disks' shows 1 disk with 1 read error:


A short SMART test for this drive results in:

########## SMART status report for sdh drive (Toshiba MG08ACA... Enterprise : 7130A0BWFVJG) ########## SMART overall-health self-assessment test result: PASSED ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0 2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0 3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8177 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 46 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 080 080 000 Old_age Always - 8010 10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 46 23 Helium_Condition_Lower 0x0023 100 100 075 Pre-fail Always - 0 24 Helium_Condition_Upper 0x0023 100 100 075 Pre-fail Always - 0 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 45 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 52 194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 31 (Min/Max 16/42) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 220 Disk_Shift 0x0002 100 100 000 Old_age Always - 117702698 222 Loaded_Hours 0x0032 080 080 000 Old_age Always - 8002 223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0 224 Load_Friction 0x0022 100 100 000 Old_age Always - 0 226 Load-in_Time 0x0026 100 100 000 Old_age Always - 594 240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0 No Errors Logged Num Test_Description (Most recent Short & Extended Tests - Listed by test number) # 1 Short offline Completed without error 00% 8010 - # 3 Extended offline Completed without error 00% 7976 - SCT Error Recovery Control: Read: Disabled Write: Disabled

Which tells me there's nothing wrong with the drive? correct?

Would I be good to just clear the error? Anything I should check/repair first?
Thanks for your help.

 

[NugentS]

do a long test on the disk with an error please - but I don't see anything obvious there as an issue

 

[GrimmReaperNL]

do a long test on the disk with an error please - but I don't see anything obvious there as an issue

Long test takes just under 24 hours. Will get back to you.
Thanks for the quick reply.

 

[joeschmuck]

The only thing I see which is odd is Test "# 2" is missing. Not something I'm use to seeing.

You had a drive issue, that is one problem. I suspect the Pool is also reporting a problem, might be related and might not.
Run the command zpool status and post the results.
You can run this command while the SMART test is running.

 

[Redcoat]

Note that it did fail its last long test:

 

[joeschmuck]

Note that it did fail its last long test:

Then it should be listed in the drive test log and saying it failed. Maybe it was test #2 that is missing for some odd reason?

 

[GrimmReaperNL]

Long test is still running unfortunately.

The only thing I see which is odd is Test "# 2" is missing. Not something I'm use to seeing.

You had a drive issue, that is one problem. I suspect the Pool is also reporting a problem, might be related and might not.
Run the command zpool status and post the results.
You can run this command while the SMART test is running.

Where is test #2 missing? I'm not sure what you're seeing.
Here is zpool status

root@TrueNAS[~]# zpool status pool: TrueNAS state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: scrub repaired 0B in 15:50:43 with 0 errors on Thu Feb 9 12:25:26 2023 config: NAME STATE READ WRITE CKSUM TrueNAS ONLINE 0 0 0 raidz3-0 ONLINE 0 0 0 0a38cf8b-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 09737924-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 d9fd11cf-9761-11ed-abfe-3cecef8c44fa ONLINE 0 0 0 0a014274-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 0b3f1b68-a583-11ec-9714-3cecef8c44fa ONLINE 1 0 0 0ae27c9a-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 56f3655d-2ed5-11ed-ab7a-3cecef8c44fa ONLINE 0 0 0 0a775747-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 0dcc0444-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 0a474b61-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 0cf68994-a583-11ec-9714-3cecef8c44fa ONLINE 0 0 0 errors: No known data errors pool: boot-pool state: ONLINE scan: scrub repaired 0B in 00:00:46 with 0 errors on Thu Feb 9 03:45:47 2023 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 sdi2 ONLINE 0 0 0 errors: No known data errors root@TrueNAS[~]#

From your script email:

Drives for this pool are listed below: 0a38cf8b-a583-11ec-9714-3cecef8c44fa -> sdj2 09737924-a583-11ec-9714-3cecef8c44fa -> sdb2 d9fd11cf-9761-11ed-abfe-3cecef8c44fa -> sde2 0a014274-a583-11ec-9714-3cecef8c44fa -> sdf2 0b3f1b68-a583-11ec-9714-3cecef8c44fa -> sdh2 0ae27c9a-a583-11ec-9714-3cecef8c44fa -> sdd2 56f3655d-2ed5-11ed-ab7a-3cecef8c44fa -> sdl2 0a775747-a583-11ec-9714-3cecef8c44fa -> sdk2 0dcc0444-a583-11ec-9714-3cecef8c44fa -> sdc2 0a474b61-a583-11ec-9714-3cecef8c44fa -> sda2 0cf68994-a583-11ec-9714-3cecef8c44fa -> sdg2

Note that it did fail its last long test:

View attachment 63417

Test was 'interrupted (host reset)'. 2 long test before that ran without issue.

SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 8038 - # 2 Short offline Completed without error 00% 8032 - # 3 Extended offline Aborted by host 90% 8032 - # 4 Short offline Completed without error 00% 8010 - # 5 Short offline Completed without error 00% 7991 - # 6 Extended offline Completed without error 00% 7976 - # 7 Short offline Completed without error 00% 7919 - # 8 Short offline Completed without error 00% 7895 - # 9 Short offline Completed without error 00% 7871 - #10 Short offline Completed without error 00% 7847 - #11 Short offline Completed without error 00% 7823 - #12 Extended offline Completed without error 00% 7807 - #13 Short offline Completed without error 00% 7748 - #14 Short offline Completed without error 00% 7724 - #15 Short offline Completed without error 00% 7700 - #16 Extended offline Interrupted (host reset) 30% 7698 - #17 Short offline Completed without error 00% 7652 - #18 Short offline Completed without error 00% 7628 - #19 Short offline Completed without error 00% 7619 - #20 Short offline Completed without error 00% 7604 - #21 Short offline Completed without error 00% 7580 -

Thanks for reading everybody :)

 

[Redcoat]

Where is test #2 missing? I'm not sure what you're seeing.

Look at the bottom of the SMART test attachment to your first post ...

 

[GrimmReaperNL]

Look at the bottom of the SMART test attachment to your first post ...

View attachment 63441

Ah, so it is.
I pulled it from @joeschmuck's smart email script. Text says 'most recent' short and extended test. Test 1 and 2 are both shorts, with 1 being the most recent. Test 3 is an extended.
I'm guess that's why #2 isn't listed.

 

[joeschmuck]

I pulled it from @joeschmuck's smart email script.

Yes, you have it exactly correct, the script only lists the last Short and last Extended tests, which means that test #2 must have been a Short test, but it won't list if it failed or not. Either way, the drive looks good from the data.

 

[NickF]

You can potentially run the command

zpool YOURPOOLNAME clear

Which will remove the error from complaining, but not resolve the underlying issue that caused it. If it does not return, you can probably sleep at night but obviously you need to watch it. I've seen this happen when a drive is failing but SMART isn't catching any issues. I've also seen this happen with a bad RAM stick, and I have also seen it after unexpected power loss.

I would do a scrub first before running the zpool clear command so ZFS can fix any other transient errors if there are any.

 

[GrimmReaperNL]

do a long test on the disk with an error please - but I don't see anything obvious there as an issue

Damn, long test took close to two days. Here's the results:

SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0 2 Throughput_Performance 0x0005 100 100 050 Pre-fail Offline - 0 3 Spin_Up_Time 0x0027 100 100 001 Pre-fail Always - 8177 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 46 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 050 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 100 100 050 Pre-fail Offline - 0 9 Power_On_Hours 0x0032 080 080 000 Old_age Always - 8106 10 Spin_Retry_Count 0x0033 100 100 030 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 46 23 Helium_Condition_Lower 0x0023 100 100 075 Pre-fail Always - 0 24 Helium_Condition_Upper 0x0023 100 100 075 Pre-fail Always - 0 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 45 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 52 194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 31 (Min/Max 16/42) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 220 Disk_Shift 0x0002 100 100 000 Old_age Always - 52166721 222 Loaded_Hours 0x0032 080 080 000 Old_age Always - 8098 223 Load_Retry_Count 0x0032 100 100 000 Old_age Always - 0 224 Load_Friction 0x0022 100 100 000 Old_age Always - 0 226 Load-in_Time 0x0026 100 100 000 Old_age Always - 594 240 Head_Flying_Hours 0x0001 100 100 001 Pre-fail Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 8086 - # 2 Extended offline Completed without error 00% 8086 - # 3 Short offline Completed without error 00% 8038 - # 4 Short offline Completed without error 00% 8032 - # 5 Extended offline Aborted by host 90% 8032 - # 6 Short offline Completed without error 00% 8010 - # 7 Short offline Completed without error 00% 7991 - # 8 Extended offline Completed without error 00% 7976 - # 9 Short offline Completed without error 00% 7919 - #10 Short offline Completed without error 00% 7895 - #11 Short offline Completed without error 00% 7871 - #12 Short offline Completed without error 00% 7847 - #13 Short offline Completed without error 00% 7823 - #14 Extended offline Completed without error 00% 7807 - #15 Short offline Completed without error 00% 7748 - #16 Short offline Completed without error 00% 7724 - #17 Short offline Completed without error 00% 7700 - #18 Extended offline Interrupted (host reset) 30% 7698 - #19 Short offline Completed without error 00% 7652 - #20 Short offline Completed without error 00% 7628 - #21 Short offline Completed without error 00% 7619 -

Looking at it, there don't appear to be any continuing issues. So I guess I'll do the zpool clear <pool> <dev id> and see if/when the error comes back.

Thanks for weighing in everybody.

 

[joeschmuck]

Running zpool TrueNAS clear should clear the issue. Should it come back again, on the same drive, let us know and we will need to troubleshoot it further. I've never needed to include the <dev id>. I'd try without first.