Brand new Seagate Exos X16 12 TB failing after 25 days, recording temperatures over 90 degrees Celsius

[resill]

Hi,

I'm quickly losing faith in Seagate's Exos X16 12 TB drives, after 2 drives failing on me within 1-4 months. The first drive failed on me after ~100 days (https://www.truenas.com/community/threads/zpool-status-read-errors.98886/), and now the second drive after ~25 days (this thread).

Have a look at the SMART report:

Code:

# smartctl -a /dev/ada3 
smartctl 7.2 2020-12-30 r5155 [FreeBSD 12.2-RELEASE-p12 amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Exos X16
Device Model:     ST12000NM001G-2MV103
Serial Number:    ZTN0NKEV
LU WWN Device Id: 5 000c50 0e32b79b5
Firmware Version: SN03
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Mar 12 09:42:16 2022 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
Offline data collection status:  (0x82)    Offline data collection activity
                    was completed without error.
                    Auto Offline Data Collection: Enabled.
Self-test execution status:      ( 121)    The previous self-test completed having
                    the read element of the test failed.
Total time to complete Offline
data collection:         (  567) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (   1) minutes.
Extended self-test routine
recommended polling time:      (1080) minutes.
Conveyance self-test routine
recommended polling time:      (   2) minutes.
SCT capabilities:            (0x70bd)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.There are a

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   055   055   044    Pre-fail  Always       -       235168146
  3 Spin_Up_Time            0x0003   098   098   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       3
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   078   060   045    Pre-fail  Always       -       65196338
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       619
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       3
 18 Head_Health             0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   001   001   000    Old_age   Always       -       575
188 Command_Timeout         0x0032   100   095   000    Old_age   Always       -       12885164037
190 Airflow_Temperature_Cel 0x0022   070   008   040    Old_age   Always   In_the_past 30 (Min/Max 28/92 #35)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       2
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       182
194 Temperature_Celsius     0x0022   030   092   000    Old_age   Always       -       30 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       176
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       176
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Pressure_Limit          0x0023   100   098   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       606h+10m+35.419s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       18937165820
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       11571499714

SMART Error Log Version: 1
ATA Error Count: 574 (device log contains only the most recent five errors)
    CR = Command Register [HEX]
    FR = Features Register [HEX]
    SC = Sector Count Register [HEX]
    SN = Sector Number Register [HEX]
    CL = Cylinder Low Register [HEX]
    CH = Cylinder High Register [HEX]
    DH = Device/Head Register [HEX]
    DC = Device Command Register [HEX]
    ER = Error register [HEX]
    ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 574 occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:18:33.575  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:30.492  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:27.343  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:24.260  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:21.215  READ DMA EXT

Error 573 occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:18:30.492  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:27.343  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:24.260  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:21.215  READ DMA EXT
  60 00 04 ff ff ff 4f 00      00:18:21.214  READ FPDMA QUEUED

Error 572 occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:18:27.343  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:24.260  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:21.215  READ DMA EXT
  60 00 04 ff ff ff 4f 00      00:18:21.214  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:18:21.183  READ LOG EXT

Error 571 occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:18:24.260  READ DMA EXT
  25 00 00 ff ff ff 4f 00      00:18:21.215  READ DMA EXT
  60 00 04 ff ff ff 4f 00      00:18:21.214  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:18:21.183  READ LOG EXT
  60 00 04 ff ff ff 4f 00      00:18:18.123  READ FPDMA QUEUED

Error 570 occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  25 00 00 ff ff ff 4f 00      00:18:21.215  READ DMA EXT
  60 00 04 ff ff ff 4f 00      00:18:21.214  READ FPDMA QUEUED
  2f 00 01 10 00 00 00 00      00:18:21.183  READ LOG EXT
  60 00 04 ff ff ff 4f 00      00:18:18.123  READ FPDMA QUEUED
  60 00 00 ff ff ff 4f 00      00:18:18.122  READ FPDMA QUEUED

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%       606         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Similar to the previous drive, which @Pitfrr pointed out (over here: https://www.truenas.com/community/threads/zpool-status-read-errors.98886/), this drive also shows a lot of read failures, at the exact same address:

Code:

40 51 00 ff ff ff 0f  Error: UNC at LBA = 0x0fffffff = 268435455

However, a lot more read errors, in a much shorter time span than last time:

Code:

Error 574 occurred at disk power-on lifetime: 606 hours (25 days + 6 hours)

And then, like last time, the extended test didn't finish (it's completed but at 90%) and ended with a read failure:

Code:

# 1  Extended offline    Completed: read failure       90%       606         -

However, unlike last time, the above was not all that scared me. Let me zoom in for you:

Code:

190 Airflow_Temperature_Cel 0x0022   070   008   040    Old_age   Always   In_the_past 30 (Min/Max 28/92 #35)

Am I correct to assume that '008' in the 'WORST' column means that, at some point, a temperature of 92 degrees Celsius was recorded? For people more familiar with Fahrenheit, that's nearly water boiling temperature. Yikes . Is Seagate shipping drives with integrated multi-core CPUs now? With the rising energy bills in Europe, I almost wish it was that hot in my house this winter.

Apart from that insane temperature reading, this is getting very symptomatic and worrying to me. Note that I just again ordered a new Seagate Exos X16 12 TB. Yes, again. For 2 reasons:

• To see if this is just really bad luck, or
• To see if this could be the reason why they are currently relatively cheap here (~€ 270 per drive, compared to a WD Red Plus 12 TB for ~€ 330 in The Netherlands).

If the 3rd drive fails again after a few weeks / months, I'll report back. I hope this is all just bad luck, but maybe anyone here has had similar problems with this particular model or other drive models?

 

[NugentS]

Exos 12TB here - no issues after nearly 2 years
I hope they are all under warranty

If its any help - the below are the stats from one of my 12's. Minmax is 18/29

SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 083 064 044 Pre-fail Always - 26236427 3 Spin_Up_Time 0x0003 090 090 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 84 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 088 060 045 Pre-fail Always - 581323538 9 Power_On_Hours 0x0032 083 083 000 Old_age Always - 15489 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 83 18 Head_Health 0x000b 100 100 050 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 2 190 Airflow_Temperature_Cel 0x0022 077 049 040 Old_age Always - 23 (Min/Max 18/29) 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 437 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1813 194 Temperature_Celsius 0x0022 023 048 000 Old_age Always - 23 (0 4 0 0 0) 195 Hardware_ECC_Recovered 0x001a 007 001 000 Old_age Always - 26236427 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Pressure_Limit 0x0023 100 100 001 Pre-fail Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 15389h+11m+01.431s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 10478766286 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 71930792995

 

[Etorix]

The report does state "Max 92", which is definitively not healthy.
What about the other drives? How are they cooled? Is there any reason why this specific drive would have been cooked to its premature death?

 

[NugentS]

92 - one would have to wrap the drive in an electric blanket and add hot water bottles to get to that temp. Even in a low airflow for some wierd reason, 92 degrees?

 

[IOSonic]

Could it be a false reading? Did the drive feel like it was 92 degrees? You would know it if you touched it. Use discretion about plugging it in again; if it is 92 degrees, it's probably not good for nearby components or those that may be plugged into it.

Check out these hard drive reports backblaze does:

Hard Drive Test Data

Hard Drive test data from the Backblaze data center. Backblaze is affordable, easy-to-use cloud storage.

www.backblaze.com

Seagate drives are cheaper and faster, but in my experience, they fail more often.

 

[ChrisRJ]

I have lost 3 Seagate Exos X16 16 TB in 2021, so my confidence is also not at record high. On the other hand, at least the RMA handling through Seagate was exemplary.

As to the temperature: I would assume there is proper cooling and none of the other drives shows comparable maximum temperatures, right?

 

[EnKo]

In case not done yet, set an alert limit for disk temperature and an e-mail-address for reporting! Otherwise you just find a problem in case of total failure. I recently setup a 4x4 disk rack with EXOS X16 and found the temperature level is very high compared to smaller devices. Even I set the fans on maximum there is still a temperature problem if the disks run for several hours on high load.

 

[resill]

Hi all,


These are the readings from my other 3 drives in the same vdev, in the same chasis, same fans cooling the drives:

Code:

190 Airflow_Temperature_Cel 0x0022   067   049   040    Old_age   Always       -       33 (Min/Max 31/37)
190 Airflow_Temperature_Cel 0x0022   066   045   040    Old_age   Always       -       34 (Min/Max 32/37)
194 Temperature_Celsius     0x0022   100   100   000    Old_age   Always       -       31 (Min/Max 18/39)

The top 2 are other X16 12 TBs, and the 3rd is a Toshiba N300 12 TB.

Much better and expected readings, I would say. This is why I'm puzzled as well to what could've caused the huge temperature spike. Whenever it happened, I wasn't there at the time to verify the temperature myself.

Have there been documented cases of false readings, or has anyone experienced false readings before?

How can I set email alerting for temperature? That sounds like a good idea in general.

 

[artlessknave]

are you sure the fans are cooling each drive evenly? could one drive be at the edge and not getting airflow?
for example, many of the front load type rackmount chassis MUST have spacers in empty slots, or all the airflow goes through the empty trays instead of around drives.

I dont see any info about the chassis you are using.

 

[resill]

are you sure the fans are cooling each drive evenly? could one drive be at the edge and not getting airflow?
for example, many of the front load type rackmount chassis MUST have spacers in empty slots, or all the airflow goes through the empty trays instead of around drives.

I dont see any info about the chassis you are using.

Sure, I also added my build in my signature now. My 4 disks are installed in a "be quiet! Silent Base 601" tower case (https://www.bequiet.com/en/case/1503), and I got 2x "be quiet! Pure Wings 2 140 mm" (https://www.bequiet.com/en/casefans/448) directly cooling all 4 disks, all getting direct airflow. But even if a disk weren't getting direct airflow, it shouldn't reaching near water boiling temperatures, right?

 

[Etorix]

Probably not without nearby drives also recording abnormal temperatures…
So this looks like a wrong reading and/or an internal issue with this drive. Have you contacted Seagate for support or RMA?

 

[resill]

Probably not without nearby drives also recording abnormal temperatures…
So this looks like a wrong reading and/or an internal issue with this drive. Have you contacted Seagate for support or RMA?

Should I contact Seagate directly for this matter? I usually contact the retailer where I bought the drive from first. Do you get a more verbose log from the manufacturer if you contact them directly? That would be interesting, and I'd be very curious.

 

[Etorix]

I have no personal experience with Seagate, but when one of my WD drives began to spat errors, WD immediately accepted the SMART report as a reason for RMA and sent me a replacement without going through the retailer.

 

[artlessknave]

"be quiet! Silent Base 601"

hmm, if the drive setup works the way I think it does, the cooling should be even enough that one drive shouldn't be so much higher. weird.
the only other thing I can think of is, could there be somthing near just that drive that gets really hot, and is like touching the drive or the mounting and transfering a ton of heat? I'm guessing probably "no", but having a few exos die like that seems like it really shouldnt happen.

 

[Etorix]

The first drive went belly up after growing a few hundred bad sectors in a short time; normal temperature though.
It's two drives failing in two different ways, which does not suggest an issue with the case.
No other drive recorded an abnormal temperature, so either this 92°C is a spurious reading/failed sensor (a case for RMA on its own), or something briefly created a lot of heat inside the drive itself (in which case, it's amazing the drive still works).

It's either bad luck, or something more sinister with 12 TB Seagate drives (also in the X12 series).

Farewell Seagate Exos X12 12TB Enterprise Hard Drives

Kicking-off our 2022 Hardware Failures thread is a pair of Seagate Exos X12 12TB hard drives (2019 vintage) that did not make it long in 2022

www.servethehome.com

Backblaze Also Points to Seagate Exos X12 Reliability Challenges

Our readers recently pointed out that Backblaze's hard drive reliability report for 2021 on Seagate Exos X12 drives aligns with a recent story

www.servethehome.com

 

[EnKo]

You can set temperature limits at Storage / Disks and then on the arrow of each disk to follow Edit Disk. You can set a critical and an informational limit, where just critical seems to has a function and informational seems not to do anything.