SOLVED HDD is degraded with no SMART Errors

[Ashvath Sridharan]

Hello TrueNAS Community!

Over the few weeks, I've been setting up my new server and migrating things like configs and stuff from my old server. Ever since everything was back to normal, I started recieving errors from one of my HDD, thinking that it won't be a big issue, I countinued to use my server as is, until my whole server stopped responding and my pool failed saying that the pool was exported, after reading some threads about this, my answer was to just restart my NAS and it did work but then the degraded pool warning still showed up. After waiting for another scrub to finish, it still was showing the warning but when I found the HDD in TrueNAS GUI that was degraded, it said it was degraded but under SMART testing there were no errors, I do have an extra drive which is of same capacity so I was thinking to just replace the HDD with my other known working one but I'm not sure on what to do as I setup all my drives in ZFS Stripe. Please help me a soon as possible, I am aiming to get rid of the warning without losing my data but if there is no other choice then I am willing to lose all my data as it isn't mission critical.

Looking for Responses!
Ashvath Sridharan

 

[joeschmuck]

I setup all my drives in ZFS Stripe

Very bad to do this as you now know. If you need to replace a hard drive, you need to copy all your data off the server if you want to retain your data.

As for the error messages, you need to be precise in exactly what the messages were. I suspect they were ZFS error messages, not drive error messages. There is a difference.

Do the following and use code brackets (the icon </> in the drop down menu, see screen capture) to maintain the text formatting.
At the CLI: zpool status -v and paste that here.

The next step depends on what the message says.

 

[Ashvath Sridharan]

Very bad to do this as you now know. If you need to replace a hard drive, you need to copy all your data off the server if you want to retain your data.

So if I use the replace option under my degraded drive, I will lose my data?

 

[joeschmuck]

Post the requested output using the code brackets, the formatting is very important but if you made one single large stripe, absolutely all your data will go away with any single drive failure or replacement. This is ZFS, not Windoze. We will help if we can. I'm hoping you do not have an actual drive failure.

 

[Ashvath Sridharan]

I tried using TrueNAS CLI Shell in the NAS itself connected to a monitor and a keyboard but it said [/ICODE]Namespace zpool not found[/ICODE] , but when I typed the command in the shell via the GUI it gave me

pool: Sridharan state: DEGRADED status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: [URL]https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P[/URL] scan: resilvered 448K in 00:00:02 with 0 errors on Sat Feb 24 18:02:41 2024 config: NAME STATE READ WRITE CKSUM Sridharan DEGRADED 0 0 0 0f4e4e32-b78d-40b3-993b-8c5f00143378 ONLINE 0 0 0 63b5874b-95bb-42f5-ac43-954ac6a452af DEGRADED 0 0 0 too many errors 2be132b1-1d0c-45ad-b657-645fcb721254 ONLINE 0 0 0 bc6ed997-cf61-457a-a567-946e2a081357 ONLINE 0 0 0 logs 9e6a5811-34bb-4f87-ac0c-afb4e9a1d3eb ONLINE 0 0 0 cache sdc1 ONLINE 0 0 0 errors: No known data errors pool: boot-pool state: ONLINE scan: scrub repaired 0B in 00:00:41 with 0 errors on Thu Feb 15 03:45:42 2024 config: NAME STATE READ WRITE CKSUM boot-pool ONLINE 0 0 0 sdg3 ONLINE 0 0 0 errors: No known data errors


Sorry if I am doing this wrong, I am quite new to shell commands in TrueNAS, please correct me if any mistakes or errors.

 

[joeschmuck]

At the command line: zpool clear Sridharan and then you can do the status command again, should be all clear.

This is a ZFS error, not hard drive error. I'm not saying the hard drive couldn't have caused this however it could be several things.

 

[Ashvath Sridharan]

I did look around in the TrueNAS reddit community and this is what I found Changing a disk on a striped ZFS pool. Can you just verify if this is true?

 

[Ashvath Sridharan]

Oh my god! It did work! Thank you so so so much Joe. You really are a life saver! Just to know, what could be the main reasons to why this happened? Is it recommended that I replace the drive? Is there anymore steps to do to prevent this in the future?

 

[joeschmuck]

Look at you, smarter than I am. I completely forgot about that trick but yes it is very true, it can be done. Keep track of your drives by serial number.

 

[joeschmuck]

Check your drives out for errors before replacing any. Use the command smartctl -a /dev/sda for the sda drive, look for errors (see the link in my signature to Decode Your SMART Data) and this will give you a general idea. I am slowly working un an updated document to include NVMe and will post that once it is complete. Make sure you check the data for all your drives.

If you are not doing SMART test, you should start doing them, however with a striped pool, it adds risk if the drive is already stressed.

 

[Ashvath Sridharan]

Ok thanks a lot for the information. But I have a small doubt, I am planning to add a caching drive, I do have a lot of USB drives, do you think I can put like a 32GB USB 3.0 Drive onto my NAS and use it for caching? Will that make any difference?

 

[joeschmuck]

Let me ask you something, how are the drives connected to the computer? USB interface could cause the error you had.

A 32GB cache drive is pretty small, USB makes it ultra slow, much slower than your hard drives so you would actually be hurting performance substantially. You are running 8GB RAM, the minimum, I would not use a log, cache, or anything other than the bare minimum. It sounds like you have grand plans for this system however it reads like a Frankenstein machine. But if it does what you need it to, then all is good.

 

[Ashvath Sridharan]

Ok so a few days in, and I have the same warning again. Should I run the smartctl command or should I just replace the hard drive? And in the above message you've sent, do you think I need a caching drive? Half of my RAM is being used with ZFS cache, so I'm guessing it's a viable option to do so (please correct me if I'm wrong) I am not planning to upgrade the RAM as the prices are a bit too high for my liking but I do have some HDDs laying around so I am ready to replace my "if corrupted" hard drive.

 

[joeschmuck]

do you think I need a caching drive?

No, not at all.

Ok so a few days in, and I have the same warning again.

That is a shame. Post the output of zpool status -v to see if there is any difference, clues. Previously it resilvered, it would be nice to know if that happened here again.

Also do a few things:
1) Post the output of smartctl -x /dev/sda assuming sda is still the suspect drive.
2) If you have not run a SMART Long test in the past 72 hours, run smartctl -t long /dev/sda and let it complete. Then do step 1 again and hope the test passed.

Half of my RAM is being used with ZFS cache, so I'm guessing it's a viable option to do so (please correct me if I'm wrong) I am not planning to upgrade the RAM as the prices are a bit too high for my liking but I do have some HDDs laying around so I am ready to replace my "if corrupted" hard drive.

Something that is difficult to pin down is TrueNAS (CORE at least) will look like it's working just fine and then burp all over you and we all believe it is due to low RAM. 8GB it technically the minimum but only for NAS operations, simple stuff. Nothing more. So that is possible. How do you know if you are short on RAM, in the GUI look at the SWAP Utilization size. If it is not zero then you ran out of RAM. Nothing can fix this except more RAM.

The other theory I have I can't really do much with until you post the SMART data for the suspect drive. Depending on the drive it might be designed to work that way. You have a lot of different drive types and they function based on the intended use. If this is your surveillance drive, that is more than likely the issue. I will explain if this is the case, but lets see what you have. I'd be happy if it was the drive but I'm not convinced the drive is the problem. Need a bit more data for now.

 

[Ashvath Sridharan]

No, not at all.


That is a shame. Post the output of zpool status -v to see if there is any difference, clues. Previously it resilvered, it would be nice to know if that happened here again.

I don't think I configured a resilvering task but I do know that I configured a scrubbing task to happen every once a week and I haven't done a SMART test ever since I created this server (Never configured any tasks for that) so I am guessing I should do step 2 and then step 1? And for the RAM I was initially using TrueNAS Core until I had a lot of problems with it therefore I moved to TrueNAS Scale and now my swap size is almost occupying my boot drive and my RAM usage is still a bit high leaving around 0.6GB of RAM left. I will try to post the results by today.

 

[Ashvath Sridharan]

For zpool status
<pool: Sridharan
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
scan: resilvered 360K in 00:00:02 with 0 errors on Wed Mar 6 16:11:28 2024
config:

NAME STATE READ WRITE CKSUM
Sridharan DEGRADED 0 0 0
0f4e4e32-b78d-40b3-993b-8c5f00143378 ONLINE 0 0 0
63b5874b-95bb-42f5-ac43-954ac6a452af DEGRADED 2 0 0 too many errors
2be132b1-1d0c-45ad-b657-645fcb721254 ONLINE 0 0 0
bc6ed997-cf61-457a-a567-946e2a081357 ONLINE 0 0 0
f2e2f1bd-b79e-4eeb-81d2-05c807c49488 ONLINE 0 0 0
logs
9e6a5811-34bb-4f87-ac0c-afb4e9a1d3eb ONLINE 0 0 0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:16 with 0 errors on Sat Mar 2 03:46:17 2024
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sde3 ONLINE 0 0 0

errors: No known data errors>

 

[joeschmuck]

So another failure. Thankfully no data was impacted and you can try to clear the error message using 'zpool clear poolname', but this is not healthy.

I've asked before for you to run a SMART Long test and post the testing results for the suspect drive, are you going to do what was requested? I mean, can can just replace the drive if you want, it isn't the way I'd handle it, I would want to know if the drive is actually failing. Maybe you have a SATA data cable issue or power issue. I'm willing to help but you need to be willing to do some testing and report the results.

 

[Ashvath Sridharan]

Drive sda passed the SMART Test 2 times and it says the lifetime is about 18234

 

[Ashvath Sridharan]

Results after doing SMART Test as you said for smartctl -x command,

Code:

<=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.14 (AF)
Device Model:     ST500DM002-1BD142
Serial Number:    W2AQJX1L
LU WWN Device Id: 5 000c50 06017f438
Firmware Version: KC45
User Capacity:    500,107,862,016 bytes [500 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database 7.3/5528
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Mar  8 00:08:32 2024 +04
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM level is:     208 (intermediate), recommended: 208
APM feature is:   Unavailable
Rd look-ahead is: Enabled
Write cache is:   Enabled
DSN feature is:   Unavailable
ATA Security is:  Disabled, frozen [SEC2]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  600) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (  83) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x303f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   115   076   006    -    96126864
  3 Spin_Up_Time            PO----   100   098   000    -    0
  4 Start_Stop_Count        -O--CK   097   097   020    -    3517
  5 Reallocated_Sector_Ct   PO--CK   100   100   036    -    0
  7 Seek_Error_Rate         POSR--   088   060   030    -    747562309
  9 Power_On_Hours          -O--CK   080   080   000    -    18239
 10 Spin_Retry_Count        PO--C-   100   100   097    -    0
 12 Power_Cycle_Count       -O--CK   097   097   020    -    3495
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   001   001   000    -    707
188 Command_Timeout         -O--CK   100   100   000    -    8 8 8
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   063   054   045    -    37 (Min/Max 30/42)
194 Temperature_Celsius     -O---K   037   046   000    -    37 (0 18 0 0 0)
195 Hardware_ECC_Recovered  -O-RC-   039   025   000    -    96126864
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    4
240 Head_Flying_Hours       ------   100   253   000    -    18224h+16m+36.242s
241 Total_LBAs_Written      ------   100   253   000    -    1648223258
242 Total_LBAs_Read         ------   100   253   000    -    3841627309
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      5  Comprehensive SMART error log
0x03       GPL     R/O      5  Ext. Comprehensive SMART error log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  NCQ Command Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xa1       GPL,SL  VS      20  Device vendor specific log
0xa2       GPL     VS    2248  Device vendor specific log
0xa8       GPL,SL  VS     129  Device vendor specific log
0xa9       GPL,SL  VS       1  Device vendor specific log
0xab       GPL     VS       1  Device vendor specific log
0xb0       GPL     VS    2928  Device vendor specific log
0xbd       GPL     VS     252  Device vendor specific log
0xbe-0xbf  GPL     VS   65535  Device vendor specific log
0xc0       GPL,SL  VS       1  Device vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (5 sectors)
Device Error Count: 699 (device log contains only the most recent 20 errors)
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 699 [18] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:53.431  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 16 40 00     00:56:53.390  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:53.292  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:49.701  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 14 40 00     00:56:49.660  READ FPDMA QUEUED

Error 698 [17] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:49.701  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 14 40 00     00:56:49.660  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:49.562  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:46.013  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 12 40 00     00:56:45.971  READ FPDMA QUEUED

Error 697 [16] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:46.013  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 12 40 00     00:56:45.971  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:45.865  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:42.200  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 10 40 00     00:56:42.158  READ FPDMA QUEUED

Error 696 [15] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:42.200  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 10 40 00     00:56:42.158  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:42.061  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:38.445  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 0e 40 00     00:56:38.403  READ FPDMA QUEUED

Error 695 [14] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:38.445  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 0e 40 00     00:56:38.403  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:38.306  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:34.674  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 0c 40 00     00:56:34.631  READ FPDMA QUEUED

Error 694 [13] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:34.674  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 0c 40 00     00:56:34.631  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:34.534  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:30.927  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 08 40 00     00:56:30.885  READ FPDMA QUEUED

Error 693 [12] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:30.927  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 08 40 00     00:56:30.885  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:30.787  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:27.164  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 06 40 00     00:56:27.122  READ FPDMA QUEUED

Error 692 [11] occurred at disk power-on lifetime: 2367 hours (98 days + 15 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 51 00 00 00 00 00 6e 0e 18 00 00  Error: UNC at LBA = 0x006e0e18 = 7212568

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:27.164  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 06 40 00     00:56:27.122  READ FPDMA QUEUED
  2f 00 00 00 01 00 00 00 00 00 10 e0 00     00:56:27.025  READ LOG EXT
  60 00 00 01 00 00 00 00 6e 0e 00 40 00     00:56:23.451  READ FPDMA QUEUED
  60 00 00 00 02 00 00 00 6e 0e 04 40 00     00:56:23.409  READ FPDMA QUEUED

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     18234         -
# 2  Extended offline    Completed without error       00%     18214         -
# 3  Short offline       Completed without error       00%      5372         -
# 4  Short offline       Interrupted (host reset)      20%      3825         -
# 5  Short offline       Completed without error       00%      2371         -
# 6  Short offline       Completed without error       00%      2371         -
# 7  Short offline       Completed without error       00%         0         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       522 (0x020a)
Device State:                        Active (0)
Current Temperature:                    37 Celsius
Power Cycle Min/Max Temperature:     30/42 Celsius
Lifetime    Min/Max Temperature:     18/46 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        59 minutes
Min/Max recommended Temperature:     14/55 Celsius
Min/Max Temperature Limit:           10/60 Celsius
Temperature History Size (Index):    128 (10)

Index    Estimated Time   Temperature Celsius
  11    2024-03-02 19:01    38  *******************
  12    2024-03-02 20:00    39  ********************
  13    2024-03-02 20:59    38  *******************
  14    2024-03-02 21:58    38  *******************
  15    2024-03-02 22:57    38  *******************
  16    2024-03-02 23:56    36  *****************
  17    2024-03-03 00:55    40  *********************
  18    2024-03-03 01:54    39  ********************
  19    2024-03-03 02:53    37  ******************
  20    2024-03-03 03:52    37  ******************
  21    2024-03-03 04:51    38  *******************
  22    2024-03-03 05:50    39  ********************
  23    2024-03-03 06:49    37  ******************
  24    2024-03-03 07:48    38  *******************
  25    2024-03-03 08:47    38  *******************
  26    2024-03-03 09:46    39  ********************
 ...    ..(  4 skipped).    ..  ********************
  31    2024-03-03 14:41    39  ********************
  32    2024-03-03 15:40    38  *******************
  33    2024-03-03 16:39    39  ********************
  34    2024-03-03 17:38    41  **********************
  35    2024-03-03 18:37    40  *********************
  36    2024-03-03 19:36    38  *******************
 ...    ..(  3 skipped).    ..  *******************
  40    2024-03-03 23:32    38  *******************
  41    2024-03-04 00:31    37  ******************
  42    2024-03-04 01:30    36  *****************
  43    2024-03-04 02:29    37  ******************
  44    2024-03-04 03:28    36  *****************
 ...    ..(  2 skipped).    ..  *****************
  47    2024-03-04 06:25    36  *****************
  48    2024-03-04 07:24    37  ******************
  49    2024-03-04 08:23    37  ******************
  50    2024-03-04 09:22    38  *******************
 ...    ..(  7 skipped).    ..  *******************
  58    2024-03-04 17:14    38  *******************
  59    2024-03-04 18:13    39  ********************
 ...    ..(  2 skipped).    ..  ********************
  62    2024-03-04 21:10    39  ********************
  63    2024-03-04 22:09    37  ******************
  64    2024-03-04 23:08    37  ******************
  65    2024-03-05 00:07    36  *****************
 ...    ..(  5 skipped).    ..  *****************
  71    2024-03-05 06:01    36  *****************
  72    2024-03-05 07:00    37  ******************
  73    2024-03-05 07:59    38  *******************
 ...    ..( 14 skipped).    ..  *******************
  88    2024-03-05 22:44    38  *******************
  89    2024-03-05 23:43    37  ******************
  90    2024-03-06 00:42    37  ******************
  91    2024-03-06 01:41    36  *****************
 ...    ..(  3 skipped).    ..  *****************
  95    2024-03-06 05:37    36  *****************
  96    2024-03-06 06:36    37  ******************
  97    2024-03-06 07:35    38  *******************
  98    2024-03-06 08:34    38  *******************
  99    2024-03-06 09:33    39  ********************
 ...    ..(  8 skipped).    ..  ********************
 108    2024-03-06 18:24    39  ********************
 109    2024-03-06 19:23     ?  -
 110    2024-03-06 20:22    30  ***********
 111    2024-03-06 21:21    39  ********************
 112    2024-03-06 22:20    38  *******************
 113    2024-03-06 23:19    39  ********************
 114    2024-03-07 00:18    36  *****************
 115    2024-03-07 01:17    36  *****************
 116    2024-03-07 02:16    37  ******************
 117    2024-03-07 03:15    36  *****************
 118    2024-03-07 04:14    36  *****************
 119    2024-03-07 05:13    36  *****************
 120    2024-03-07 06:12    37  ******************
 121    2024-03-07 07:11    38  *******************
 ...    ..( 10 skipped).    ..  *******************
   4    2024-03-07 18:00    38  *******************
   5    2024-03-07 18:59    42  ***********************
   6    2024-03-07 19:58    38  *******************
   7    2024-03-07 20:57    39  ********************
   8    2024-03-07 21:56    38  *******************
   9    2024-03-07 22:55    37  ******************
  10    2024-03-07 23:54    37  ******************

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP/SMART Log 0x04) not supported

Pending Defects log (GP Log 0x0c) not supported

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x000a  2            3  Device-to-host register FISes sent due to a COMRESET
0x0001  2            0  Command failed due to ICRC error
0x0003  2            0  R_ERR response for device-to-host data FIS
0x0004  2            0  R_ERR response for host-to-device data FIS
0x0006  2            0  R_ERR response for device-to-host non-data FIS
0x0007  2            0  R_ERR response for host-to-device non-data FIS>

Edit by Joe: Added code tags to make readable.

 

[Ashvath Sridharan]

Results after SMART Test for command zpool status,

pool: Sridharan
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
scan: resilvered 360K in 00:00:02 with 0 errors on Wed Mar 6 16:11:28 2024
config:

NAME STATE READ WRITE CKSUM
Sridharan ONLINE 0 0 0
0f4e4e32-b78d-40b3-993b-8c5f00143378 ONLINE 0 0 0
63b5874b-95bb-42f5-ac43-954ac6a452af ONLINE 0 0 0
2be132b1-1d0c-45ad-b657-645fcb721254 ONLINE 0 0 0
bc6ed997-cf61-457a-a567-946e2a081357 ONLINE 0 0 0
f2e2f1bd-b79e-4eeb-81d2-05c807c49488 ONLINE 0 0 0
logs
9e6a5811-34bb-4f87-ac0c-afb4e9a1d3eb ONLINE 0 0 0

errors: No known data errors

pool: boot-pool
state: ONLINE
scan: scrub repaired 0B in 00:01:16 with 0 errors on Sat Mar 2 03:46:17 2024
config:

NAME STATE READ WRITE CKSUM
boot-pool ONLINE 0 0 0
sde3 ONLINE 0 0 0

errors: No known data errors